-
Notifications
You must be signed in to change notification settings - Fork 28
feat(framework): add QA and communication evaluations #512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
alexsoyes
wants to merge
12
commits into
next
Choose a base branch
from
feat/qa-video-evidence
base: next
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
6eb0b01
feat(aidd-dev): add standalone QA journey evidence
alexsoyes aee73be
feat(aidd-dev): generalize test-scope journeys
alexsoyes 6bba3f9
refactor(aidd-dev): trim redundant QA action inputs
alexsoyes 6070b03
feat(aidd-dev): streamline browser QA evidence
alexsoyes f7a4952
feat(aidd-context): add browser QA memory fields
alexsoyes b130716
feat(aidd-dev): scope QA to browser journeys
alexsoyes 504b412
refactor(browser-qa): alex's streamline documentation
alexsoyes f1f9430
refactor(plan): clarify phase template and test scope instructions
alexsoyes a772272
feat(aidd-vcs): link browser QA videos in pull requests
alexsoyes 3127fb7
fix(aidd-dev): make browser QA videos reviewable
alexsoyes 43f14f9
fix(aidd-dev): harden browser QA video capture
alexsoyes 2366ffc
refactor(aidd-dev): streamline browser QA setup
alexsoyes File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| --- | ||
| name: 11-browser-qa | ||
|
blafourcade marked this conversation as resolved.
|
||
| description: Run post-review browser QA and produce short named videos for a locked happy path and sourced browser edge cases. Use when the user wants concise reviewer evidence for a web journey. Not for API, CLI, automated tests, diff review, or application fixes. | ||
| argument-hint: prerequisites | load-scope | prepare-run | run-scenarios | ||
| --- | ||
|
|
||
| # Browser QA | ||
|
|
||
| ```mermaid | ||
| flowchart LR | ||
| prerequisites["prerequisites"] --> scope["load-scope"] --> prepare["prepare-run"] --> run["run-scenarios"] | ||
| ``` | ||
|
|
||
| ## Actions | ||
|
|
||
| Read only the next action's file before running it. | ||
|
|
||
| | # | Action | Does | | ||
| | --- | --------------- | ---------------------------------------------------------- | | ||
| | 00 | `prerequisites` | Verify the browser runner and media dependencies | | ||
| | 01 | `load-scope` | Lock one happy path and a bounded set of sourced edge cases | | ||
| | 02 | `prepare-run` | Resolve the shortest deterministic path to executable runs | | ||
| | 03 | `run-scenarios` | Record, normalize, verify, reset, and report every scenario | | ||
|
|
||
| ## Transversal rules | ||
|
|
||
| - Run against a reviewed change and never patch the application. | ||
| - Never spawn agents. Batch independent reads and tool checks, but keep state-changing browser work sequential. | ||
| - Do not narrate action transitions, searches, fixtures, selectors, or successful checks. Report only a blocker, a required decision, or the final verdict and paths. | ||
28 changes: 28 additions & 0 deletions
28
plugins/aidd-dev/skills/11-browser-qa/actions/00-prerequisites.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # 00 - Prerequisites | ||
|
|
||
| Verify the runner dependencies before resolving the QA scope. | ||
|
|
||
| ## Input | ||
|
|
||
| None. | ||
|
|
||
| ## Output | ||
|
|
||
| Verified `npx`, Playwright CLI `0.1.17`, `ffmpeg`, and `ffprobe`. | ||
|
|
||
| ## Process | ||
|
|
||
| 1. **Check.** In one pass, resolve `npx`, run `npx --yes @playwright/cli@0.1.17 --version`, and resolve `ffmpeg` and `ffprobe`. | ||
| 2. **Continue.** When every check passes, continue without reporting it. | ||
| 3. **Resolve.** When a dependency is missing, ask one concise question: the user installs the listed dependencies, or authorizes you to install them now. | ||
| 1. If the user installs them, provide only the shortest platform-appropriate commands and stop until they confirm completion. | ||
| 2. If authorized, install only the missing dependencies, then rerun every check. | ||
| 4. **Stop.** Report the shortest decisive error when installation is declined or a recheck fails. | ||
| 5. **Protect.** Never add runner or media dependencies to the application manifest. | ||
|
|
||
| ## Test | ||
|
|
||
| - Passing checks produce no message. | ||
| - A missing dependency produces one choice and no unapproved installation. | ||
| - Either installation path reruns every check before continuing. | ||
| - No application dependency file changes. |
28 changes: 28 additions & 0 deletions
28
plugins/aidd-dev/skills/11-browser-qa/actions/01-load-scope.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # 01 - Load Scope | ||
|
|
||
| Lock the smallest defensible browser QA scope before execution. | ||
|
|
||
| ## Input | ||
|
|
||
| Plan path or implementation artifact. | ||
|
|
||
| ## Output | ||
|
|
||
| - 1 locked browser happy path, | ||
| - a bounded set of sourced browser edge cases, | ||
| - a source label, | ||
| - a resolved evidence folder. | ||
|
|
||
| ## Process | ||
|
|
||
| 1. **Resolve.** the requested feature from its plan. | ||
| 2. **Filter.** Keep only Happy path and Edge case sections where every task's Mermaid actor is `browser` (Ignore non-browser, mixed-channel, and unmarked scenario sections without displaying them). | ||
| 3. **Lock.** Lock 1 browser happy path from the explicit user journey, then the filtered plan Test Scope, then browser-observable acceptance criteria in the implementation artifact. | ||
| 1. Ask one concise question only when those sources conflict or expose multiple browser journeys. | ||
| 4. **Collect.** Include every filtered planned edge case + candidates from explicit browser validation, error, empty-state, permission, boundary, or recovery branches already visible in the implementation artifact. | ||
| 1. Search directly related browser tests only when the filtered plan contains no edge case. | ||
| 5. **Bound.** Deduplicate candidates against planned edges. Keep at least 3 proposed edges, ranked by user impact, browser observability, determinism, and proximity to the requested journey. | ||
| 6. **Decide.** Automatically include a proposed edge only when it is deterministic, browser-observable, in scope, and non-destructive. Require a decision only for an external or destructive action. | ||
| 7. **Validate.** Reject a scenario without a source, trigger, browser-observable outcome, or executable teardown when it changes state. | ||
| 8. **Locate.** Use the existing AIDD feature folder when the source belongs to one. Otherwise use `aidd_docs/tasks/<yyyy_mm>/<yyyy_mm_dd>_<feature-slug>/`. | ||
| 9. **Show.** Emit `Happy path: locked (<source>)` and one compact `Edge case | Source | Decision` table. Do not repeat scenario steps. |
27 changes: 27 additions & 0 deletions
27
plugins/aidd-dev/skills/11-browser-qa/actions/02-prepare-run.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # 02 - Prepare Run | ||
|
|
||
| Resolve the application state and scenario paths before retained recording begins. | ||
|
|
||
| ## Input | ||
|
|
||
| Verified prerequisites and the earlier defined scope. | ||
|
|
||
| ## Output | ||
|
|
||
| A successful prepared run with a reachable application, authenticated sessions, deterministic fixtures, executable scenario steps, and proven teardown. | ||
|
|
||
| ## Process | ||
|
|
||
| 1. **Preflight.** Check the application and fixed `1280×720` viewport. | ||
| 2. **Reuse.** Read `aidd_docs/memory/testing.md` first when it exists. | ||
| 1. Resolve Browser QA entry, auth, fixtures, and reset from its `Browser QA` section, then a directly related browser test, then one targeted browser snapshot. | ||
| 2. Stop searching as soon as the run is executable. | ||
| 3. **Authenticate.** Establish the required role before recording. | ||
| 1. Never include login discovery or secret lookup in evidence. | ||
| 4. **Fixture.** Use deterministic data satisfying each setup. | ||
| 1. Never choose a live record by guesswork. | ||
| 5. **Rehearse** only non-mutating steps and selectors. | ||
| 1. Never execute the final state-changing action merely to rehearse it. | ||
| 6. **Reset.** Resolve an executable teardown for every state-changing scenario. | ||
| 1. If preparation changed state, execute the teardown and verify the baseline now; a future restart is not proof. | ||
| 7. **Return.** Keep only the fixture, initial URL, minimal steps, expected outcome, teardown, and isolated session id per scenario. |
37 changes: 37 additions & 0 deletions
37
plugins/aidd-dev/skills/11-browser-qa/actions/03-run-scenarios.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # 03 - Run Scenarios | ||
|
|
||
| Execute, save, and report one clean browser QA take per scenario. | ||
|
|
||
| ## Input | ||
|
|
||
| The prepared run, source label, and resolved evidence folder. | ||
|
|
||
| ## Output | ||
|
|
||
| `<evidence-folder>/qa.md` + 1 final WebM per scenario. | ||
|
|
||
| ## Process | ||
|
|
||
| 1. **Group.** Run at most two read-only scenarios concurrently in isolated sessions. | ||
| 1. Run every state-changing scenario sequentially. | ||
| 2. **Record.** Apply setup before recording, then follow the runner's recording contract. | ||
| 3. **Verdict.** Compare actual with expected. | ||
| 1. Retain a product failure and mark the run failed. | ||
| 4. **Recover.** Discard a setup or tooling failure, reset, and retry once. | ||
| 1. A second operational failure blocks the scenario. | ||
| 5. **Reset.** Execute teardown after every state-changing take, verify the baseline, then close the session. | ||
| 6. **Normalize.** Normalize at most two independent raw files concurrently. | ||
| 1. Save only `qa/happy-path.webm` and `qa/edge-case-<scenario-slug>.webm` after `ffprobe` and chronological frame inspection pass. | ||
| 7. **Clean.** Delete raw takes and temporary validation frames only after every final file passes codec, dimension, duration, path, cut-point, and frame checks. | ||
| 1. Never retain screenshots or alternate media. | ||
| 8. **Report.** Fill the report asset with the source label. | ||
| 1. Keep one result row per scenario and add Findings only for a failure or blocker. | ||
| 9. **Return.** Output the verdict and evidence paths, then ask `Open happy-path.webm in the browser for review?`; open the final file there when confirmed. | ||
|
|
||
| ```md | ||
| @../references/run-scope-playwright-cli.md | ||
|
alexsoyes marked this conversation as resolved.
|
||
| ``` | ||
|
|
||
| ```md | ||
| @../assets/qa-report-template.md | ||
|
alexsoyes marked this conversation as resolved.
|
||
| ``` | ||
12 changes: 12 additions & 0 deletions
12
plugins/aidd-dev/skills/11-browser-qa/assets/qa-report-template.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| # Browser QA: {{feature}} | ||
|
|
||
| - **Verdict**: {{pass | fail | skipped}} | ||
| - **Source**: {{plan path | implementation path | user request}} | ||
| - **Run**: {{yyyy_mm_dd}} | ||
|
|
||
| ## Scenarios | ||
|
|
||
| | Scenario | Result | Verdict | Evidence | Duration | | ||
| | -------- | ------ | ------- | -------- | -------- | | ||
|
|
||
| {{findings-section-when-needed}} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.