fix(recorder): carry a recorded wait step's duration through to replay - #107
fix(recorder): carry a recorded wait step's duration through to replay#107myselfsiddharth wants to merge 3 commits into
Conversation
TrajectoryRecorder.wait(intent, ms) always takes an explicit duration but never wrote it anywhere the trajectory could carry it, so replay's "wait" case found no bound duration and fell back to the networkidle probe — a different condition, on a different clock, than the one actually recorded. ADR-0008 adds wait_ms to the action shape in trajectory.schema.json and cache-row.schema.json. The recorder now emits it, the compiler passes it through unchanged (same treatment as key/url_template/custom_op), and the runner checks it before the existing param-bound lookup and before falling back to networkidle — so a recorded wait replays as that exact sleep, and networkidle stays reachable only for a bare, duration-less wait action that never comes from the recorder anymore. Closes #83.
|
@copilot resolve the merge conflicts in this pull request |
|
@copilot Review this PR after reading the docs |
Resolved in commit |
Reviewed after reading the updated docs ( |
OM152002
left a comment
There was a problem hiding this comment.
Ran it rather than read it. The fix is correct; one defect at a boundary the schema itself declares valid.
Verified: "latent today" holds (zero wait steps across all five in-tree trajectory/bundle files); validate:contracts passes on all six existing artifacts unregenerated; wait_ms: 150 slept 152 ms with settled undefined. Both guards bite — reverting the recorder one-liner fails the e2e test, and reverting the precedence to firstParam(...) ?? action.wait_ms fails at 10004 ms, proving the --param would have won. CI green locally: 178 unit, 14 integration.
wait_ms: 0 still falls through to networkidle
Both schemas say "minimum": 0, and wait(intent, 0) is a legal call — but the runner gates on the value:
const msRaw = action.wait_ms ?? firstParam(action, params); // 0 ?? x -> 0 (correct)
if (Number.isFinite(ms) && ms > 0) { ... } // 0 fails hereMeasured on this PR's own never-idle fixture:
wait_ms: 150 -> 152ms settled=undefined SLEPT (recorded path)
wait_ms: 0 -> 5004ms settled=false *** NETWORKIDLE PROBE ***
That is the exact semantic drift ADR-0008 closes, surviving where the schema says zero is valid. The ?? is right; the downstream > 0 undoes it. Either set exclusiveMinimum: 0 in both schemas, or — better, and truer to the ADR — gate on action.wait_ms !== undefined rather than on the value. A recorded zero is a recorded observation; replay should reproduce it as an instant no-op, not reinterpret it as "no duration given".
A fifth action shape didn't get the field
src/*/types.ts has five copies of the action shape, all carrying key? / url_template? / custom_op?. Four were updated; src/cache/types.ts:50-57 was missed. Runtime survives (finalize() spreads ...candidate.compiled_action), but cache-row.schema.json now declares a field the cache package's own type does not — so a cache-side consumer can't read row.compiled_action.wait_ms without a type error. Relevant to #64, which rewrites rows through that type.
Non-blocking
"replays as that sleep" asserts elapsed < 3000 + settled === undefined. The settled check is the real discriminator, but expect(elapsed).toBeGreaterThanOrEqual(150) would pin that it slept the recorded duration, not just that it avoided the probe.
Option C is the right call and well-argued — keeping networkidle because it's a different primitive rather than a degraded one, with loadProgram() accepting hand-written programs as the concrete reason. Rejecting option A because a --param could silently override a recorded value, and then testing that precedence, is the part most PRs skip.
OM152002
left a comment
There was a problem hiding this comment.
Can you fix the PR with suggested changes in the comment? And if it's deliberate, lmk....I'll approve and merge
…kidle Review on #107 found the fix stopped one value short of the boundary the schemas themselves declare valid. Both contracts say `"minimum": 0` and `recorder.wait(intent, 0)` is a legal call, so `wait_ms: 0` is a value the recorder can genuinely produce. The runner resolved it correctly (`0 ?? x -> 0`) and then discarded it one line later on `ms > 0`, falling through to the bounded networkidle probe — measured at 5004ms with `settled: false` on the never-idle fixture, against 8ms now. That is the same condition-swap ADR-0008 exists to close, surviving at the one input the schema still admits. The runner now gates on presence, not magnitude: a recorded duration is an observation and replays as itself, including zero as an instant no-op. A negative or non-finite `wait_ms` is not an observation the recorder can produce, so it falls through as if nothing was recorded rather than reaching Playwright as a negative timeout. Also from review: - `src/cache/types.ts` was the fifth copy of the action shape and the one that missed the field. Runtime survived on `finalize()`'s spread, but `cache-row.schema.json` declared a field the cache package's own type did not, so a cache-side consumer reading `row.compiled_action.wait_ms` got TS2339 — verified, and relevant to #64 which rewrites rows through it. - "replays as that sleep" asserted only `elapsed < 3000` + `settled` undefined, which a `wait_ms` that was read and then ignored would still pass. Both sleep tests now pin `elapsed >= 150`. Tests: the zero case is pinned in both packages (compiler must not let a falsy-but-recorded 0 collapse into "absent"; runner must not reinterpret it), plus a negative-value fallback case. Reverting the runner gate fails the new zero test at 5008ms. `minimum: 0` stays rather than becoming `exclusiveMinimum: 0` — zero is meaningful, not a value to forbid. ADR-0008 and docs/gate/runner.md record the presence-not-magnitude rule. npm run ci green (181 unit, 14 integration); npm run test:canary green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016CAt2F7SR83iuqvnM4GzAf
|
Both defects were real, neither was deliberate. Fixed in
|
| before | after | |
|---|---|---|
wait_ms: 0 |
5004 ms, settled=false |
8 ms, settled=undefined |
wait_ms: 150 |
152 ms | 155 ms |
Left minimum: 0 alone rather than taking exclusiveMinimum: 0 — zero is now meaningful, so forbidding it would be the wrong half of the fix.
One thing your comment didn't ask for but the same gate implied: a negative or non-finite wait_ms can only reach the runner from a hand-authored program, and it must not become waitForTimeout(-1). It now falls through as if nothing was recorded. Pinned by a test.
The new zero test bites — reverting the gate fails it at 5008 ms on expected false to be undefined.
Fifth action shape
Confirmed and fixed. Also confirmed it was load-bearing rather than cosmetic, by the path you named:
probe.ts(3,30): error TS2339: Property 'wait_ms' does not exist on type 'CompiledAction'.
That's row.compiled_action.wait_ms against src/cache/types.ts before the change, clean after — so #64 would have hit it.
Non-blocking
Taken. Both sleep tests now assert elapsed >= 150 alongside the settled check, so a wait_ms that is read and then ignored no longer passes on the ceiling alone.
Also
The zero case is pinned in both packages, not just the runner — buildCompiledAction already used !== undefined so it was correct, but nothing stopped a future edit from letting a falsy-but-recorded 0 collapse into "absent", which would break the runner's precedence chain from upstream. ADR-0008's Decision section and docs/gate/runner.md now state the presence-not-magnitude rule and why minimum: 0 stays.
npm run ci green (181 unit, 14 integration), npm run test:canary green.
Generated by Claude Code
Closes #83.
Summary
TrajectoryRecorder.wait(intent, ms)(src/recorder/session.ts) always takes an explicit duration but never wrote it anywhere the trajectory could carry it:At replay,
executeAction's"wait"case (src/runner/actions.ts) found no bound duration viafirstParam(action, params)and fell through to the boundednetworkidleprobe. So a recorded "wait 500ms" replayed as "wait for network idle" — a different condition, on a different clock, that can pass or fail independently of what was actually recorded.Latent today (no trajectory/bundle in the tree has a
waitstep — verified acrossexperiments/gate-v1/trajectories/*.json,contracts/examples/trajectory.example.json,artifacts/compiled/*.bundle.json), but real the moment a recorded task includes a deliberate wait.Decision — ADR-0008
Per CONTRIBUTING ("prefer extending a schema via ADR over ad-hoc JSON fields"), this is a schema change, so it gets one. Considered and rejected synthesizing a
param_ref+ literal binding for the recorded duration (conflates a compile-time constant with the runtime-supplied-value mechanismfill/select/uploadalready use, and would let a same-named--paramsilently override what was recorded). Chose instead a literalwait_msfield, the same treatment already given to other compile-time-constant action fields (keyfor press,url_templatefor navigate).Changes
wait_ms(non-negative integer) added to the action shape incontracts/trajectory.schema.jsonandcontracts/cache-row.schema.json. Additive/optional — no existing artifact is invalidated, and none needed regenerating (nothing in-tree has awaitstep).wait()now emitsaction: { type: "wait", wait_ms: ms }. Its only public entry point requiresms, so a recorder-produced trajectory can no longer emit a bare, duration-less wait.buildCompiledAction()passeswait_msthrough unchanged, same askey/url_template/custom_op."wait"case now resolves a duration asaction.wait_ms→firstParam(action, params)(the existing, separately-tested runtime-bound mechanism) → the unchangednetworkidlefallback. The literal is checked first so it can't be silently overridden by a same-named--parambinding.docs/gate/runner.md's "Bounded waits" section documents the fix;docs/README.md's ADR table anddocs/gate/runner.md's frontmatter dates updated.Test plan
tests/unit/compiler.test.ts:wait_mssurvivescompileTrajectoryunchanged; stays absent when the recorded step didn't carry one.tests/unit/runner-bounded-wait.test.ts: await_ms-carrying action replays as a sleep (nosettledfield, meaningnetworkidlenever ran); a recordedwait_mstakes precedence over a same-action--parambinding that would otherwise make it wait 10s.tests/unit/recorder.test.tswith a realrecorder.wait("...", 50)call against a live fixture page, asserting the emitted trajectory step carrieswait_ms: 50and the whole trajectory still validates against the (updated) schema.param_refs-based mechanism, untouched).npm run ci(secret-scan, contracts, lint, lint:docs — 46 docs, typecheck, unit — 175 tests, integration — 5 tests) — all greennpm run test:canary— green (privacy boundary, merge-blocking)🤖 Generated with Claude Code