Two gaps from evaluating the apify-mcp-server #1147 run (refactor!: separate legacy MCP wiring from ActorsMcpServer — draft PR apify/apify-mcp-server#1155). Run: 13 dispatches, ~1.76M subagent tokens, ~97 min agent-compute, converged in 2 inner rounds + 1 polish, 0 blocking findings. Costs below come from that run's _progress.md ledger.
Sibling of #20 (from the #1140 run). Independent — close piecemeal.
1. fulfillment re-runs the full oracle instead of trusting test-results.txt
fulfillment cost 135k and re-executed all five oracle commands live, even though iter-3/test-results.txt was already the orchestrator's authoritative capture and the skill says to trust the oracle over self-reports. Re-running green is not its job — its unique, non-reproducible value is mapping each success criterion to evidence (here, 35 criteria). The role text permits "the non-mutating check a criterion names," which was read as license to re-run the whole suite.
Fix: in the fulfillment role scope (SKILL.md role table), state that iter-N/test-results.txt is authoritative — cite it, do not re-run the suite; "a check a criterion names" means one named command, not the oracle. Spend budget on criteria→evidence mapping.
2. Blind downstream stages carry no cross-repo context and state wrong facts
Generalizes #20 item 3 (followups never reads the issue tracker) from followups to success_criteria. On this run, success_criteria asserted "apify-mcp-server-internal is not in this session" — false: the orchestrator had already read that repo and pinned its exact consumption surface. The stage is blind to the solution by design and also cannot see orchestrator plumbing (_progress.md), so a cross-repo fact it needed to write correct criteria was in neither of its inputs.
Fix: let verify record cross-repo/consumption facts in _request_fact_check.md (they are verified facts, not solution), so success_criteria — which already reads the fact-check — inherits them instead of guessing.
Sub-point (criteria-vs-design conflict, same blindness root): criterion 25's grep listed task_execution.ts in its "no McpError import" check. Because the criteria author is blind to the design, that grep partly contradicted decision D8 ("remove McpError only from tool_call_engine.ts and tool_dispatch.ts"), and the implementer changed task_execution.ts's unreachable default arm to satisfy it. Verified behavior-equivalent, and the implementer flagged it in claim.md for reconciliation — which is the right outcome, but worth a one-line SKILL.md note: a criterion that contradicts an approved decision is an orchestrator/human reconciliation, never a silent implementer scope expansion.
Two gaps from evaluating the apify-mcp-server #1147 run (
refactor!: separate legacy MCP wiring fromActorsMcpServer— draft PR apify/apify-mcp-server#1155). Run: 13 dispatches, ~1.76M subagent tokens, ~97 min agent-compute, converged in 2 inner rounds + 1 polish, 0 blocking findings. Costs below come from that run's_progress.mdledger.Sibling of #20 (from the #1140 run). Independent — close piecemeal.
1.
fulfillmentre-runs the full oracle instead of trustingtest-results.txtfulfillmentcost 135k and re-executed all five oracle commands live, even thoughiter-3/test-results.txtwas already the orchestrator's authoritative capture and the skill says to trust the oracle over self-reports. Re-running green is not its job — its unique, non-reproducible value is mapping each success criterion to evidence (here, 35 criteria). The role text permits "the non-mutating check a criterion names," which was read as license to re-run the whole suite.Fix: in the
fulfillmentrole scope (SKILL.mdrole table), state thatiter-N/test-results.txtis authoritative — cite it, do not re-run the suite; "a check a criterion names" means one named command, not the oracle. Spend budget on criteria→evidence mapping.2. Blind downstream stages carry no cross-repo context and state wrong facts
Generalizes #20 item 3 (
followupsnever reads the issue tracker) fromfollowupstosuccess_criteria. On this run,success_criteriaasserted "apify-mcp-server-internal is not in this session" — false: the orchestrator had already read that repo and pinned its exact consumption surface. The stage is blind to the solution by design and also cannot see orchestrator plumbing (_progress.md), so a cross-repo fact it needed to write correct criteria was in neither of its inputs.Fix: let
verifyrecord cross-repo/consumption facts in_request_fact_check.md(they are verified facts, not solution), sosuccess_criteria— which already reads the fact-check — inherits them instead of guessing.Sub-point (criteria-vs-design conflict, same blindness root): criterion 25's grep listed
task_execution.tsin its "noMcpErrorimport" check. Because the criteria author is blind to the design, that grep partly contradicted decision D8 ("removeMcpErroronly fromtool_call_engine.tsandtool_dispatch.ts"), and the implementer changedtask_execution.ts's unreachable default arm to satisfy it. Verified behavior-equivalent, and the implementer flagged it inclaim.mdfor reconciliation — which is the right outcome, but worth a one-lineSKILL.mdnote: a criterion that contradicts an approved decision is an orchestrator/human reconciliation, never a silent implementer scope expansion.