Skip to content

Refuse STAR evidence before the round opens - #229

Merged
jserv merged 2 commits into
mainfrom
evidence
Oct 3, 2026
Merged

jserv merged 2 commits into
mainfrom
evidence

Conversation

@jserv

@jserv jserv commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

A report could read behavioral (skipped) beside STAR scores and a summary of the STAR answer. The interviewer can write its own [SYSTEM EVENT] ... Begin STAR behavioral reserve, ask a behavioral question at minute 7, and record STAR evidence, while only the platform's round_transition sets the round status the report reads. This takes the two fixes the issue thread asked for: record_framework_evidence refuses every STAR phase until the platform opens the round (platform-owned skips at the five-minute warning and the end still close it), and the report can no longer pair a skipped round with STAR content.

On the report side, the brief now says whether the behavioral round opened, never opened, or does not exist in this interview, and tells the reviewer an out-of-turn exchange is not evidence for any field. Validation refuses a STAR score or STAR plan item for an unopened round, so the existing repair loop rewrites them from the coding round; trimming them after validation instead would drop the matching feedback improvements and could leave a list below the two items the schema requires. The live prompt now says only the platform sends a stage direction. A coding-only interview no longer gets STAR skip rows, which with the evidence ledger full also evicted a coding observation. Bundle 27 carries live prompt 19 and report prompt 16; the rubric and report schema are unchanged.

Verified with ./scripts/test.sh, which passes locally. The browser lanes skipped for lack of Chromium and Playwright on this machine, as did cargo-audit and shellcheck. New tests cover the validator refusing and accepting by round state, the repair loop rewriting and then running out on STAR content, the frozen assessment and real prompt builder carrying the round state, a STAR answer after the five-minute warning, and a coding-only interview getting no skips.

Left out: the header minutes are the planned budgets rather than time spent, and whether a coding round that finishes early may open the behavioral reserve before the coding budget elapses. Both were set aside in the issue as separate questions.

Closes #211


Summary by cubic

Stops STAR evidence from being recorded before the platform's round transition opens the behavioral round, so a report can no longer show STAR scores and a summary of the STAR answer beside behavioral (skipped).

  • record_framework_evidence now refuses every STAR phase until the round opens and tells the interviewer to return to coding; platform-owned skips at the five-minute warning and the end still close an unopened round.
  • The report brief states whether the behavioral round opened, never opened, or wasn't configured, and tells the reviewer that an out-of-turn behavioral exchange is not evidence for any field.
  • Report validation refuses a STAR score or STAR plan item for an unopened round, so the repair loop rewrites them from the coding round rather than trimming them, which would drop the matching feedback improvements and could leave a list under the two items the schema requires.
  • The live prompt says only the platform sends a [SYSTEM EVENT], and a coding-only interview no longer gets STAR skip rows, which with a full ledger also evicted a coding observation.
  • The transcript line marking where the round opened is crate-private, so it is not a prompt-format detail a dependent can rely on.
  • Bundle 27 carries live prompt 19 and report prompt 16; rubric and report schema are unchanged.

Written for commit 68ff727. Summary will update on new commits.

Review in cubic

cubic-dev-ai[bot]

This comment was marked as resolved.

@jserv
jserv requested review from ColtenOuO and alanhc October 3, 2026 09:16
Comment thread src/agent/prompts.rs Outdated
Comment thread src/agent/report.rs Outdated
A report could read "behavioral (skipped)" beside STAR scores and a
summary of the STAR answer, because the interviewer can write its own
round-start event, ask a behavioral question at minute 7, and record
STAR evidence, while only the platform's transition sets the round
status. The evidence tool now refuses every STAR phase until that
transition and tells the interviewer to return to coding, and the live
prompt says only the platform sends a stage direction. The report brief
says whether the round opened, never opened, or does not exist; an
opened round is marked in the report transcript, so an answer given out
of turn before the mark is not assessed. For an unopened round,
validation refuses a STAR plan item, so the repair loop rewrites it
from the coding round rather than the report being trimmed below the
two improvements each list must hold, and the server clears the STAR
scores of the report it accepts, which costs no repair. A coding-only
interview no longer gets STAR skip rows, which with the ledger full
also evicted a coding observation. Bundle 27 carries live prompt 19 and
report prompt 16.

Closes #211
cubic-dev-ai[bot]

This comment was marked as resolved.

The behavioral round mark and the helper that inserts it were
re-exported as public agent API, though only the report builder and
its unit tests use them. Both are now crate-private, the mark visible
to tests only, so a prompt-format detail is not a surface a dependent
can come to rely on.
@jserv
jserv merged commit 3e603ef into main Oct 3, 2026
7 checks passed
@jserv
jserv deleted the evidence branch October 3, 2026 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report marks the behavioral round skipped when STAR evidence was recorded

2 participants