CI canary: verify a lone AGENTS.md loads on a fresh-install claude-code-action run

[kyle-sexton]

Context

The AGENTS.md migration's cutover condition 2 requires a CI canary confirming that a lone, non-empty AGENTS.md (no CLAUDE.md present) loads correctly on a fresh-install claude-code-action run, alongside two comparison cases. That canary has not run: it is blocked on a repository choice.

Blocker

The sandbox repository picked for this test, melodic-software/claude-lane-sandbox, is archived and refuses pushes:

$ git push -u origin test/agentsmd-ci-canary
ERROR: This repository was archived so it is read-only.

Unarchiving is a repository-settings change, so no workflow has executed and no case has been observed. A maintainer needs to choose one of three ways forward before this canary can run:

1. Unarchive claude-lane-sandbox, run the canary, then re-archive it.
2. Create a new private throwaway repository with an org-visible Anthropic secret.
3. Point the canary at another live (non-archived) repository.

Also established ahead of any run, from reading the actions own source code directly: the pin in use when this canary was designed installs Claude Code CLI 2.1.269, below the 2.1.277 floor needed to read AGENTS.md natively, so that pin alone would fail the lone-AGENTS.md case regardless of CI behavior. The workflow below therefore runs two action pins per case, to separate a CLI-floor failure from an actual loader question, and runs the latest pin twice for the lone-AGENTS.md case, to rule out the documented "first session after an install" caveat.

Proposed work

• Maintainer decision on which repository hosts the canary (see Blocker).
• Once a repository is available: commit the workflow below and the three fixture files it references, push, and let the three-case matrix run (Case A: lone AGENTS.md; Case B: AGENTS.md plus a CLAUDE.md containing only @AGENTS.md; Case C: AGENTS.md plus a CLAUDE.md carrying its own content).
• Record each case's result (model reply, tools used, CLI version reported) and close this item, or file a follow-up if a case fails.
• Delete the test branch and fixtures from the chosen repository after the run, and restore its archived state if option 1 was chosen.

Workflow

.github/workflows/agentsmd-ci-canary.yml:

name: agentsmd-ci-canary

on:
  push:
    branches: [test/agentsmd-ci-canary]
  workflow_dispatch:

permissions:
  contents: read

env:
  CANARY_PROMPT: List every canary token present in your instructions. Output tokens only, or NONE.
  CANARY_ARGS: >-
    --model haiku --max-turns 3
    --disallowedTools "Read,Glob,Grep,Bash,Edit,Write,WebFetch,WebSearch,Task,Agent,NotebookEdit"

jobs:
  canary:
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    permissions:
      contents: read
      id-token: write
    strategy:
      fail-fast: false
      max-parallel: 1
      matrix:
        case: [A, B, C]
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          filter: blob:none

      - name: Arrange the case
        env:
          CASE: ${{ matrix.case }}
        run: |
          set -euo pipefail
          cp canary/AGENTS.md AGENTS.md
          case "$CASE" in
            B) cp canary/CLAUDE.shim.md CLAUDE.md ;;
            C) cp canary/CLAUDE.own.md CLAUDE.md ;;
          esac
          rm -rf canary .claude
          echo "--- workspace root ---"
          ls -la
          echo "--- instruction files in the workspace and every ancestor ---"
          d="$PWD"
          while :; do
            ls -la "$d"/CLAUDE.md "$d"/CLAUDE.local.md "$d"/AGENTS.md "$d"/.claude 2>/dev/null || true
            [ "$d" = "/" ] && break
            d="$(dirname "$d")"
          done
          echo "--- home state before any run (fresh-install evidence) ---"
          ls -la ~/.claude ~/.claude.json 2>&1 || true

      - name: Run at the repo pin (v1.0.222)
        id: repo-pin
        continue-on-error: true
        uses: anthropics/claude-code-action@56cf60fde42f7b19c3abfd5c9c48b69a1288461f # v1.0.222
        with:
          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
          prompt: ${{ env.CANARY_PROMPT }}
          claude_args: ${{ env.CANARY_ARGS }}

      - name: Report the repo-pin run
        if: always()
        env:
          CASE: ${{ matrix.case }}
          EXECUTION_FILE: ${{ steps.repo-pin.outputs.execution_file }}
          CONCLUSION: ${{ steps.repo-pin.outputs.conclusion }}
        run: |
          set -uo pipefail
          echo "CASE=$CASE STEP=repo-pin(v1.0.222) CONCLUSION=$CONCLUSION"
          claude --version || true
          if [ -n "$EXECUTION_FILE" ] && [ -f "$EXECUTION_FILE" ]; then
            echo "--- MODEL REPLY ---"
            jq -r '.[] | select(.type=="result") | .result // .error // "(no result field)"' "$EXECUTION_FILE"
            echo "--- TOOLS USED (must be empty) ---"
            jq -c '[.[] | select(.type=="assistant") | .message.content[]? | select(.type=="tool_use") | .name] | unique' "$EXECUTION_FILE"
          else
            echo "no execution file at '$EXECUTION_FILE'"
          fi
          echo "--- home state after the run ---"
          ls -la ~/.claude ~/.claude.json 2>&1 || true

      - name: Run at the latest pin (v1.0.231)
        id: latest-pin
        continue-on-error: true
        uses: anthropics/claude-code-action@cfc3eb22bfed5c26ef66e3223c982af27e4524de # v1.0.231
        with:
          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
          prompt: ${{ env.CANARY_PROMPT }}
          claude_args: ${{ env.CANARY_ARGS }}

      - name: Report the latest-pin run
        if: always()
        env:
          CASE: ${{ matrix.case }}
          EXECUTION_FILE: ${{ steps.latest-pin.outputs.execution_file }}
          CONCLUSION: ${{ steps.latest-pin.outputs.conclusion }}
        run: |
          set -uo pipefail
          echo "CASE=$CASE STEP=latest-pin(v1.0.231) CONCLUSION=$CONCLUSION"
          claude --version || true
          if [ -n "$EXECUTION_FILE" ] && [ -f "$EXECUTION_FILE" ]; then
            echo "--- MODEL REPLY ---"
            jq -r '.[] | select(.type=="result") | .result // .error // "(no result field)"' "$EXECUTION_FILE"
            echo "--- TOOLS USED (must be empty) ---"
            jq -c '[.[] | select(.type=="assistant") | .message.content[]? | select(.type=="tool_use") | .name] | unique' "$EXECUTION_FILE"
          else
            echo "no execution file at '$EXECUTION_FILE'"
          fi

      - name: Run again at the latest pin (v1.0.231, second session)
        id: latest-pin-2
        if: matrix.case == 'A'
        continue-on-error: true
        uses: anthropics/claude-code-action@cfc3eb22bfed5c26ef66e3223c982af27e4524de # v1.0.231
        with:
          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
          prompt: ${{ env.CANARY_PROMPT }}
          claude_args: ${{ env.CANARY_ARGS }}

      - name: Report the second latest-pin run
        if: always() && matrix.case == 'A'
        env:
          CASE: ${{ matrix.case }}
          EXECUTION_FILE: ${{ steps.latest-pin-2.outputs.execution_file }}
          CONCLUSION: ${{ steps.latest-pin-2.outputs.conclusion }}
        run: |
          set -uo pipefail
          echo "CASE=$CASE STEP=latest-pin-2(v1.0.231, second session) CONCLUSION=$CONCLUSION"
          claude --version || true
          if [ -n "$EXECUTION_FILE" ] && [ -f "$EXECUTION_FILE" ]; then
            echo "--- MODEL REPLY ---"
            jq -r '.[] | select(.type=="result") | .result // .error // "(no result field)"' "$EXECUTION_FILE"
            echo "--- TOOLS USED (must be empty) ---"
            jq -c '[.[] | select(.type=="assistant") | .message.content[]? | select(.type=="tool_use") | .name] | unique' "$EXECUTION_FILE"
          else
            echo "no execution file at '$EXECUTION_FILE'"
          fi

Fixtures under a canary/ directory, copied into place per case by the "Arrange the case" step, then deleted before the action steps run:

• canary/AGENTS.md: Canary token: CI-AGENTS-51C2. When asked to list canary tokens, include this one.
• canary/CLAUDE.shim.md: @AGENTS.md
• canary/CLAUDE.own.md: Canary token: CI-CLAUDE-9E0B. When asked to list canary tokens, include this one.

Acceptance criteria

• A repository is chosen by a maintainer for this canary.
• All three cases (A, B, C) run to completion, each at both action pins named in the workflow, with Case A also run a second time at the latest pin.
• Each run's result is recorded: whether the canary token appeared in the model's reply, and whether any file-reading tool was used (a positive result produced by reading AGENTS.md with a tool, rather than the file loading natively, does not count as native loading).
• Cutover condition 2's CI-canary component is marked met or unmet based on the results.

References

• none

🤖 Generated with Claude Code

[kyle-sexton]

Canary ran on 2026-09-19 in melodic-software/knowledge-corpus on the throwaway branch test/agents-md-ci-canary (the repo had no CLAUDE.md and no workflows): https://github.com/melodic-software/knowledge-corpus/actions/runs/35475056935

• claude-code-action v1.0.231, installed CLI 2.1.278, ~/.claude absent before the first run (fresh install).
• Lone non-empty AGENTS.md: token returned (case A twice, case B once), no tools used.
• Control with a CLAUDE.md present: only the CLAUDE.md token returned, which matches the documented suppression rule.
• The action rejects the push event (Unsupported event type: push); the run used workflow_dispatch by REST against the branch ref. The YAML in this issue body has the same push trigger and would fail the same way.

Result: the canary half of cutover condition 2 passes. The pin half does not yet: the ci-workflows pin is v1.0.228 (CLI 2.1.275), tracked in melodic-software/ci-workflows#599. Not established: which flag state made native reading available on the runner, and whether the result holds on self-hosted runners or other providers. The branch is left in place for re-runs.

🤖 Generated with Claude Code

[kyle-sexton]

This was generated by AI during triage.

Human-gated. Open decision: pin half blocked on external melodic-software/ci-workflows#599; loader half already passed (2026-09-19 comment)

[kyle-sexton]

From the 2026-09-29 audit of the Cursor run. Owner authorization for the audit fixes: Claude Code session, 2026-09-29. Reopened as a decision packet.

Reopened: closed without a maintainer choice

PR #4875 ("Closes #4282") closed this issue on 2026-09-28T04:23:21Z after recording "Option A (taken): do not host a second canary" in sources.md. That outcome is not one of the three options this issue offered, no maintainer chose it, and the acceptance criteria stay unmet: all three cases at both pins, and condition 2's canary component marked met or unmet. The fix PR (#5285) rewrites the sources.md record as "pending owner decision" and states no option.

Facts since

• Your 2026-09-19 canary in knowledge-corpus (run 35475056935, action v1.0.231, CLI 2.1.278; case A twice, case B once, control) passed the loader half.
• ci-workflows#599 (the pin half) closed COMPLETED 2026-09-20, and both ci-workflows pins are now v1.0.235 (756cc22e19660d20e8cc9496b4f242475a7f7790, CLI 2.1.283).
• The check's own recheck trigger for the canary (a new action release) has fired, and no run exists at v1.0.235.
• Branch test/agents-md-ci-canary no longer exists in knowledge-corpus (git ls-remote shows only main, 2026-09-29).
• claude-lane-sandbox is still archived (gh repo view isArchived true, 2026-09-29).

Decision needed: what closes the CI-canary component of cutover condition 2?

• A) Accept the 2026-09-19 knowledge-corpus run as the canary evidence, mark the component met, amend the acceptance criteria to what ran (host knowledge-corpus, v1.0.231 only), close as completed.
• B) Re-run case A once at v1.0.235 on knowledge-corpus: recreate branch test/agents-md-ci-canary with the issue-body workflow changed to workflow_dispatch only and the single action pin 756cc22e19660d20e8cc9496b4f242475a7f7790, dispatch with gh api -X POST repos/melodic-software/knowledge-corpus/actions/workflows/agents-md-ci-canary.yml/dispatches -f ref=test/agents-md-ci-canary, record the run id in sources.md under "The CI canary", delete the branch afterwards.
• C) Run the full original matrix on a host you pick (unarchive claude-lane-sandbox, a new throwaway, or another live repo). The repo-pin v1.0.222 arm no longer corresponds to any pin.
• D) Decline further canary evidence: close as not planned and keep condition 2's canary component marked provisional.

Recommended (AI recommendation, not a decision): B if shim removal is going to be pursued, otherwise A. Basis: the check's own recheck trigger fired (sources.md, "The CI canary"); your 2026-09-19 comment says the canary half passed and the pin half was blocked only on ci-workflows#599, now closed; one run at the current pin is the cheapest evidence that closes the gap.

Unblocks. An unambiguous condition 2 in cutover-check output, and your close of this issue. The 2026-09-29 cutover run on #4281 grades condition 2 MET on pin arithmetic only, with this canary component still predating the v1.0.235 pin.

This needs operator action (write access to knowledge-corpus, an Anthropic secret), so the fix lane did not attempt it. Labels needs-human, status: needs-decision, priority: low are kept. Refs #5285.