Skip to content

chore: Follow-ups from evaluating the apify-mcp-server #1147 run #21

Description

@jirispilka

Two gaps from evaluating the apify-mcp-server #1147 run (refactor!: separate legacy MCP wiring from ActorsMcpServer — draft PR apify/apify-mcp-server#1155). Run: 13 dispatches, ~1.76M subagent tokens, ~97 min agent-compute, converged in 2 inner rounds + 1 polish, 0 blocking findings. Costs below come from that run's _progress.md ledger.

Sibling of #20 (from the #1140 run). Independent — close piecemeal.

1. fulfillment re-runs the full oracle instead of trusting test-results.txt

fulfillment cost 135k and re-executed all five oracle commands live, even though iter-3/test-results.txt was already the orchestrator's authoritative capture and the skill says to trust the oracle over self-reports. Re-running green is not its job — its unique, non-reproducible value is mapping each success criterion to evidence (here, 35 criteria). The role text permits "the non-mutating check a criterion names," which was read as license to re-run the whole suite.

Fix: in the fulfillment role scope (SKILL.md role table), state that iter-N/test-results.txt is authoritative — cite it, do not re-run the suite; "a check a criterion names" means one named command, not the oracle. Spend budget on criteria→evidence mapping.

2. Blind downstream stages carry no cross-repo context and state wrong facts

Generalizes #20 item 3 (followups never reads the issue tracker) from followups to success_criteria. On this run, success_criteria asserted "apify-mcp-server-internal is not in this session" — false: the orchestrator had already read that repo and pinned its exact consumption surface. The stage is blind to the solution by design and also cannot see orchestrator plumbing (_progress.md), so a cross-repo fact it needed to write correct criteria was in neither of its inputs.

Fix: let verify record cross-repo/consumption facts in _request_fact_check.md (they are verified facts, not solution), so success_criteria — which already reads the fact-check — inherits them instead of guessing.

Sub-point (criteria-vs-design conflict, same blindness root): criterion 25's grep listed task_execution.ts in its "no McpError import" check. Because the criteria author is blind to the design, that grep partly contradicted decision D8 ("remove McpError only from tool_call_engine.ts and tool_dispatch.ts"), and the implementer changed task_execution.ts's unreachable default arm to satisfy it. Verified behavior-equivalent, and the implementer flagged it in claim.md for reconciliation — which is the right outcome, but worth a one-line SKILL.md note: a criterion that contradicts an approved decision is an orchestrator/human reconciliation, never a silent implementer scope expansion.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions