Feat/response prompt rewrite - #263
Conversation
Formatting only, no behaviour change, split out so the prompt rewrite that follows reads as the change it is. Both files predate the format check added in #250, which runs on the files a PR touches — so editing them at all means formatting them.
The reply rules landed as code in #241; the prompt still required the shape they fail a draft for — "Always include code examples", "Structure responses with **bold headers**", a closing "Was this helpful?". So the model was being asked for the reply the linter in #256 would collapse, and enforcing that linter first would have withheld answers the agent had been instructed to write. The personality and formatting blocks now carry the rules from the Agent's Output Doc: verdict in the first sentence, one approach rather than a menu, length follows the evidence, markdown only where there is content to organise, at most one code sample and only when retrieval supplied it, one API version per reply, a docs page or repo file behind every substantive claim, no hedged names, no praise openers, no self-commentary, and two sentences under 60 words when nothing settles the question. GROUNDING_RULES is untouched — #242 already corrected it — and both section headers keep their names so its override clause still resolves. The zero-retrieval fallback no longer invites an answer "from general CopilotKit knowledge if possible". With nothing retrieved there is nothing to be right from and nothing to cite, so the only correct reply there is the handoff. 16 tests, red first. Five pair the prompt against eval/rules.ts: each asserts checkReply fails a reply for the phrase, then asserts the prompt bans that same phrase, so a rule that stops firing cannot leave a prompt line silently unenforced. The handoff cap reads HANDOFF_WORD_CAP rather than restating 60. 1214 tests pass, tsc clean, no new lint. Four mutations, each killing its tests: restored "Was this helpful?"; dropped the word cap; restored the general-knowledge fallback; softened "never a menu of three".
jerelvelarde
left a comment
jerelvelarde
left a comment
There was a problem hiding this comment.
This is the piece the last three PRs were building toward, and pairing it to eval/rules.ts rather than shipping it as prose is what makes it reviewable. Approving.
The stale-CI worry does not apply — I checked rather than assumed
The green run is bound to 246803e and the branch is well behind main, with #256 having rewritten eval/rules.ts (+195/-20) — the exact module the new tests import. So I merged current main into this branch locally:
- merge is clean, no conflicts
- full package suite: 1276 passing, 66 files
So the assertions survive #256. Worth an update-branch before merging so CI says so too, but the content is fine.
The pairing tests are the best thing here, and they are not vacuous
I checked that specifically, because ruleFor(...)?.passed with optional chaining is the shape that silently passes when a rule id goes missing. It does not:
| Mutation | Result |
|---|---|
rename no-banned-phrases in rules.ts |
2 failed |
restore "Was this helpful?" to the prompt |
1 failed |
A renamed rule makes ruleFor return undefined, and undefined is not false, so the test fails rather than passing quietly. That is the right way round.
Reading HANDOFF_WORD_CAP off the module instead of restating 60 is the same instinct — move the cap and either the prompt moves or the test breaks. The two halves cannot drift.
On the rewrite itself
The diagnosis is right and it is the half that was missing. #241 could measure padding and #256 could refuse it, but the prompt was still ordering it — "Always include code examples", "Structure responses with bold headers", "Was this helpful?". Formatting was mandatory and having something to say was not, so the layout got filled with whatever was available: praise, a restatement, three approaches, a paragraph about the agent's own limits. Enforcing against a shape the prompt demanded would have been the worst of both.
Two lines I want to call out as load-bearing:
- "Length follows the evidence... if there is not enough content to organise, do not organise it." That is the actual fix. Everything else follows from it.
- The zero-retrieval fallback. Replacing "answer from general CopilotKit knowledge if possible" with an explicit handoff instruction closes root cause 1 at the one point where it was written into the prompt in so many words. With no sources there is nothing to be right from — the old text invited exactly the invented answer the grounding rules forbid two paragraphs later.
And the note that the non-mechanical lines (verdict first, one approach, length follows evidence) cannot get a rule is the honest part. Nothing checkable separates a well-judged three-paragraph answer from a padded one. Saying so, and pointing at the harness as the way to find out, is better than pretending the linter covers it.
Two small things, neither blocking
The PR body is empty. The code comments carry the reasoning and reference "the Agent's Output Doc", "root cause 1/2", "Case C", "Case D", #241 and CopilotKit#6927 — none of it linked from the PR. This is the change someone will come back to in six months when a reply reads oddly; a three-line body pointing at the doc and #241 would save that dig.
The live eval mode is still the open question, and this is the change that most needs it. #241 shipped the scoring half and explicitly not the caller that feeds real threads through it, so right now we have a prompt rewrite whose effect is unmeasurable. That is not a reason to hold this — the old prompt was measurably wrong — but it does mean "did it help?" stays unanswered until the harness runs live.
2 participants
No description provided.