PROJECT: Maintenance agent — org-wide PR triage, audit log, staged auto-merge

[mmcky]

Summary

QuantEcon carries 266 open PRs across 209 repos (2026-08-03). A large share are mechanical — dependabot bumps, sync-generated batches, disposition debt — consuming maintainer attention far out of proportion to the judgment they require. Meanwhile the org already has the ingredients for doing this work responsibly with an agent: strong per-repo CI (quantecon-book-theme runs a 103-screenshot visual regression suite), the audit plugin family in QuantEcon/skills, the status-translations collect→JSON→Pages pattern, and prior bot-identity experience (quantecon-news-reporter[bot]).

This issue proposes a maintenance agent: a scheduled agent that triages every open PR org-wide, publishes its reasoning to an auditable GitHub-native decision log, and — only after its recommendations demonstrably track maintainer decisions — graduates to auto-merging a narrow, policy-defined class of PRs. It is a sibling of the existing automation family, and a data producer for both #328 (operational telemetry) and #321 (config reporting).

The problem, measured (snapshot 2026-08-03)

Metric	Value
Open PRs org-wide	266
… dependabot	50 (35 open ≥30 days)
… mmcky (largely agent-assisted batches awaiting review)	96
… pre-commit-ci	0 — its trusted autofix loop self-manages
Open ≥1 year (disposition debt, not review debt)	88
quantecon-book-theme	12 open: 9 dependabot + 3 mmcky

Illustrative case: book-theme #418 (webpack-dev-server 5→6, a major bump) — fully green incl. 103 visual regression tests, mergeable, untouched since Jul 27. The bottleneck is not information; it is a trusted process for saying "the checks that matter passed, the blast radius is understood, merge."

Design in brief

Judgment lives in the agent; authority lives in deterministic code. Five components:

1. Collector (deterministic, nightly) — GraphQL sweep of all open PRs → dated snapshot committed to a new status-maintenance repo (same shape as status-translations: cron → collector → committed JSON → Pages dashboard).
2. Triage agent (Claude, scheduled) — reads snapshot + policy; per PR writes a decision record (JSONL): verdict, lane, evidence (changelog delta × how the repo actually uses the package, CI nuance, advisories), risks, recommended action, policy + prompt version. Records are back-filled with the eventual human outcome, so agreement is measured with zero extra effort.
3. Policy file (human-owned, versioned) — maintenance-policy.yml: lanes, repo tiers, rate caps, cooldowns, promotion thresholds. The agent may propose changes via PR; it can never edit policy itself.
4. Actor (deterministic gate) — when enabled, re-validates in plain code every mechanical predicate of an "auto-merge eligible" claim (author, semver class, lockfile-only diff, required checks green — not skipped, cooldown, caps, no unresolved human comments) before calling the merge API. A hallucinated eligibility claim cannot merge anything.
5. Reporter — Pages dashboard (triage state, agreement rate, every decision linkable) + daily/weekly brief.

Trust lanes (change class × repo tier gate every action):

• Lane A — bot-authored patch/minor, lockfile-only, all required checks green → end state: auto-merge (rate-capped, cooldown).
• Lane B — majors, grouped bumps, advisory-driven updates, build-tooling changes → agent writes the verification report; human one-click merges. Stays human.
• Lane C — judgment content (features, docs, lectures, translations) → triage/label/route only. Always human.
• Lane D — disposition debt (the 88 ≥1-year PRs) → batch close/supersede/revive proposals, mirroring the Decisions: dispositions for the 24 open lecture-proposal issues #344 motion for lecture-proposal issues.

Staged autonomy ladder: advise → measure agreement on the dashboard → delegate per-lane when a lane sustains ≥98% agreement over ≥30 decisions / 4 weeks (starting proposal) → any human revert/override freezes the lane pending review. Auto-merge is an earned end state, not a mode switch.

Safeguards for the auto-merge bar: supply-chain cooldown (released ≥5 days, no advisory since; advisory-driven bumps instead get expedited human review), per-repo/org daily merge caps, org-level kill switch variable, pre-written revert path + post-merge CI watch that auto-opens a revert PR and freezes the lane if main goes red.

Rollout

• Phase 0 — prototype (no GitHub writes): scheduled Claude session produces a daily triage brief; lane definitions stabilize.
• Phase 1 — plumbing: status-maintenance repo + nightly collector Action + Pages dashboard; register quantecon-maintenance App identity.
• Phase 2 — advisor in-org: triage as a scheduled Actions workflow (an audit-family skill from QuantEcon/skills); labels + verification comments in pilot repos (book-theme, actions, 1–2 sphinx tool repos); agreement measured continuously.
• Phase 3 — delegated merges: Lane A only, pilot repos only, behind the deterministic gate. Weekly "what I merged and why" brief.
• Phase 4 — widen as Migrate the five native lecture CI pipelines to quantecon/actions/build-lectures #348's CI standardization makes policy portable across lecture repos.

Relationship to #328 and #321

• The nightly PR inventory and decision/outcome records are time-series control-plane health data for Server & build telemetry — operational dashboard for runner health, build performance & cost #328 (backlog, merge latency, agreement/revert rates); stored GitHub-native now, exportable to CloudWatch/Grafana via OTLP once Server & build telemetry — operational dashboard for runner health, build performance & cost #328 picks its backend. Server & build telemetry — operational dashboard for runner health, build performance & cost #328's RunsOn CloudWatch first step proceeds independently.
• Config-drift observations the agent makes en route (e.g. dependabot groups lacking applies-to: security-updates — the cause of book-theme's eight ungrouped npm PRs) feed Centralized environment & config reporting for lecture series #321's weekly config-diff reporting.
• status-maintenance is a second instance of the status-* pattern Centralized environment & config reporting for lecture series #321 proposes, sharing collector/dashboard code with status-translations.

Near-term wins (independent of the agent)

1. book-theme: add an applies-to: security-updates group to dependabot.yml.
2. book-theme: clear the 9-PR dependabot batch as the agent's first supervised run (patch/minor transitive bumps are Lane A shape; #418 + advisory-driven ones are Lane B) — producing the first decision records.
3. Disposition sweep of the 88 ≥1-year PRs via the audit skill family (the Decisions: dispositions for the 24 open lecture-proposal issues #344 motion, applied to PRs).
4. Register the bot identity early so agent actions separate from personal PRs in every future metric.

Open questions

1. Merge method — squash everywhere or per-repo convention?
2. Lane D auto-close: ever enabled, or always human batch-approval?
3. Home for maintenance-policy.yml — meta or skills?
4. Promotion threshold (98% / 4 weeks / 30 decisions is a starting proposal).
5. Does the brief fold into the existing weekly report?
6. Translation-repo PRs: out of scope for v1, or triage-only summaries to translation leads?

---

Distilled from the v0.1 design document (2026-08-03, drafted with Claude in the QuantEcon Agents project — full doc has the architecture diagram, decision-record schema, data appendix with sources, and suggested first artifacts; attached below / available on request).