From a910a721d134b4e6d03e1089b939311c548f72c6 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Sat, 18 Jul 2026 13:39:23 +0000 Subject: [PATCH 01/66] docs(install): drop obsolete manual Claude Code config step Auto-compact and verbose output are now auto-provisioned via .claude/settings.json; remove the "Configure Claude Code (required)" step from install.sh/.ps1 and reframe adjacent docs (07_faq_technical, 03_best_practices, 02_understanding_daaf) from manual instruction to "on by default." Also fixes a pre-existing broken anchor in 03_best_practices.md. User-doc link audit: 286 internal links checked, no other broken references. Co-Authored-By: Claude Opus 4.8 (1M context) --- scripts/host/install.ps1 | 6 ------ scripts/host/install.sh | 6 ------ user_reference/02_understanding_daaf.md | 2 +- user_reference/03_best_practices.md | 2 +- user_reference/07_faq_technical.md | 8 ++++---- 5 files changed, 6 insertions(+), 18 deletions(-) diff --git a/scripts/host/install.ps1 b/scripts/host/install.ps1 index a8d27906..db8fe90a 100644 --- a/scripts/host/install.ps1 +++ b/scripts/host/install.ps1 @@ -700,12 +700,6 @@ Write-Host " and all DAAF operations in one place." Write-Host "" Write-Host " 2. On first launch, you'll be asked to authenticate with your Anthropic account." Write-Host "" -Write-Host " 3. Configure Claude Code (required):" -Write-Host " - Type /config and set:" -Write-Host " Auto-compact -> False" -Write-Host " Verbose output -> True" -Write-Host " - Press ESC to return to the chat" -Write-Host "" Write-Host "Available scripts (in $InstallDir):" Write-Host " .\daaf.ps1 DAAF Control Panel (recommended)" Write-Host " .\run_daaf.ps1 Launch Claude Code directly" diff --git a/scripts/host/install.sh b/scripts/host/install.sh index aa10034f..6403fb23 100755 --- a/scripts/host/install.sh +++ b/scripts/host/install.sh @@ -676,12 +676,6 @@ echo " and all DAAF operations in one place." echo "" echo " 2. On first launch, you'll be asked to authenticate with your Anthropic account." echo "" -echo " 3. Configure Claude Code (required):" -echo " - Type /config and set:" -echo " Auto-compact -> False" -echo " Verbose output -> True" -echo " - Press ESC to return to the chat" -echo "" echo "Available scripts (in ${INSTALL_DIR}):" echo " bash daaf.sh DAAF Control Panel (recommended)" echo " bash run_daaf.sh Launch Claude Code directly" diff --git a/user_reference/02_understanding_daaf.md b/user_reference/02_understanding_daaf.md index 70778df9..43673778 100644 --- a/user_reference/02_understanding_daaf.md +++ b/user_reference/02_understanding_daaf.md @@ -242,7 +242,7 @@ Four things to keep in mind as you use DAAF: 1. In addition to the context engineering DAAF orchestrates behind the scenes, **what you ask Claude to do and how you ask it to do it** is an immensely important element of getting better quality output from DAAF/Claude. So a lot of what we'll talk about here (and in [**03. Best Practices**](03_best_practices.md)) is how to do this thoughtfully and well to maximize your chances of getting something useful from DAAF. 2. The system is **designed to intelligently select and inject the right context to Claude before your query/question/chat**, based on what you provide in your query/question/chat. But this is **NOT** foolproof, and simply cannot account for every possibility. Feel free to go off the beaten path at will, but just be aware that it's going to necessarily be less supported and structured from there; you may ultimately find it's not working very well for what you want, because I wasn't able to design for that style of work. Trying to write your query in a different way can help, or you can help us improve DAAF by [opening an issue](https://github.com/DAAF-Contribution-Community/daaf/issues) and telling us about it! 3. Because thoughtfully shaping the context is our way of shaping Claude's thinking, DAAF is built for **frontier models** and works them hard, taking advantage of their full context windows wherever possible. That does make DAAF genuinely compute-intensive -- pushing capable models to their limits is inherently so. But cost is a **dial you control**, not a fixed barrier: [empirical benchmarking](https://daaf.openaugments.org/bench/) shows that several models drive DAAF's workflows excellently at very different price points -- including cheaper Anthropic tiers and high-performing open-weight alternatives (like GLM 5.2) via OpenRouter. The quality-cost tradeoff is yours to tune through your choice of model and provider route, and the [DAAFBench results](https://daaf.openaugments.org/bench/) are the best starting reference for making that call. -4. While DAAF's reference files, skills, and workflow instructions are all carefully designed to be loaded at specific moments, Claude may occasionally fail to load them, skip a step, or deviate from its instructions in subtle ways. When this happens, you get an agent working without the specialized knowledge it was supposed to have, which means it falls back on its general training -- its fuzzy long-term memory -- and that's when hallucinations, fabricated variable names, and plausible-sounding-but-wrong details creep in. This is why **Verbose output** in Claude Code's `/config` settings is particularly useful: it lets you see what DAAF's agents are actually thinking behind the scenes, including what motivates which files they're reading and which skills they're loading (or what shortcuts they're deciding on, if they are). +4. While DAAF's reference files, skills, and workflow instructions are all carefully designed to be loaded at specific moments, Claude may occasionally fail to load them, skip a step, or deviate from its instructions in subtle ways. When this happens, you get an agent working without the specialized knowledge it was supposed to have, which means it falls back on its general training -- its fuzzy long-term memory -- and that's when hallucinations, fabricated variable names, and plausible-sounding-but-wrong details creep in. This is why **Verbose output** -- which DAAF turns on by default (you can confirm it in Claude Code's `/config` settings) -- is particularly useful: it lets you see what DAAF's agents are actually thinking behind the scenes, including what motivates which files they're reading and which skills they're loading (or what shortcuts they're deciding on, if they are). --- diff --git a/user_reference/03_best_practices.md b/user_reference/03_best_practices.md index 87ea0696..b2c778b0 100644 --- a/user_reference/03_best_practices.md +++ b/user_reference/03_best_practices.md @@ -347,7 +347,7 @@ There's one oversight responsibility that's easy to overlook because it's about LLMs are non-deterministic, and DAAF's reference loading is orchestrated by an LLM. This means that occasionally -- not often, but not never -- an agent will proceed without loading a skill it was instructed to load, or the orchestrator will skip a reference file it was supposed to read. When this happens, the agent falls back on its general training, which produces output that looks correct but is built on plausible inference rather than curated knowledge. -**Verbose output is your primary monitoring tool.** When you set Verbose output to True in `/config` (which you should -- it's a [required configuration setting](01_installation_and_quickstart.md#configure-claude-code-required)), you can see the internal thought process informing the file reads that DAAF's agents make. Here's what to watch for: +**Verbose output is your primary monitoring tool.** DAAF ships with Verbose output turned on by default (you can confirm it anytime in `/config`), so you can see the internal thought process informing the file reads that DAAF's agents make. Here's what to watch for: **Signs that something may not have loaded:** - An agent explicitly mentions wanting to load something but then never doing it diff --git a/user_reference/07_faq_technical.md b/user_reference/07_faq_technical.md index 3258a357..f5a65a91 100644 --- a/user_reference/07_faq_technical.md +++ b/user_reference/07_faq_technical.md @@ -125,9 +125,9 @@ DAAF always confirms the mode with you before proceeding, so you can adjust if i These are Claude Code slash commands you can type anytime during a session: -- `/config` — Opens the Claude Code settings menu. Key settings to check: - - **Auto-compact:** Set to `false` (DAAF manages its own context) - - **Verbose output:** Set to `true` (lets you see what agents are thinking) +- `/config` — Opens the Claude Code settings menu. DAAF pre-configures the key settings for you (no action needed — you can verify them here if you like): + - **Auto-compact:** Off by default (DAAF manages its own context) + - **Verbose output:** On by default (lets you see what agents are thinking) - `/model` — Switch the active model (use arrow keys to select) - `/clear` — Clear conversation history and start fresh (used when resuming from STATE.md) - `/exit` — Exit Claude Code (first step in ending a session) @@ -966,7 +966,7 @@ This is one of the most common -- and most important -- symptoms to recognize. I DAAF has extensive curated knowledge about its supported data sources, stored in skill files. When these skills load correctly, agents have access to exact variable names, precise endpoint paths, correct coded values, and known caveats. When a skill *doesn't* load -- which can happen due to the non-deterministic nature of LLMs -- the agent falls back on its general training data and fills in the gaps with plausible-sounding but potentially incorrect details. **What to do:** -1. Make sure **Verbose output** is set to **True** in `/config`. This is your primary tool for monitoring how agents are deciding to load or not to load certain reference files. +1. Use **Verbose output** (on by default in DAAF — you can confirm it in `/config`) to monitor how agents are deciding to load or not to load certain reference files. This is your primary tool for spotting a skipped skill or reference. 2. Ask DAAF to verify: "Can you double-check that variable name against the actual skill documentation?" or "Did the agent load the CCD data source skill before writing that script?" 3. If the issue persists, try restarting the session with `/clear` -- a fresh context often resolves loading issues. For Full Pipeline mode, DAAF's session recovery system will pick up where you left off. 4. Report persistent loading failures by [opening an issue](https://github.com/DAAF-Contribution-Community/daaf/issues) -- patterns of failure help us improve DAAF's loading reliability. From 658aa2ea62deba167a6c677595d9c1292ad4817f Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Sat, 18 Jul 2026 17:23:00 +0000 Subject: [PATCH 02/66] fix(host): fail hard on corroborated short backups; warnings-aware banner; graceful caller aborts backup_daaf.{sh,ps1}: add a fatal branch when docker cp reports a nonzero exit AND fewer files land than the volume scan counted (two corroborating signals = truncated backup) -- name the partial folder, tell the user to delete it, exit 1 before the banner. Latch HAD_WARNINGS across the 4 non-fatal WARNING sites so the completion banner reads "Backup completed WITH WARNINGS -- verify before relying on it" instead of an unqualified "Backup complete!" (exit stays 0). Claude-state copy stays WARNING-only. update_daaf.{sh,ps1}, migrate_daaf.sh: make every backup caller fail gracefully rather than abruptly (set -e) or silently. Capture the backup exit (set -e-safe on .sh, LASTEXITCODE on .ps1, with a fail-closed pre-init in update_daaf.ps1) and abort with a clear message; a declined optional backup still proceeds. Tests: bats + Pester coverage for the fatal branch, both banner variants, and the updater and migrate abort/continue gates (closing a pre-existing coverage gap on the migrate_daaf.ps1 gate). Docs: quickstart backup promise revised. Session records in research/2026-07-18_FrameworkDev_BackupShortCopyHardening/ (workspace is gitignored by design; retained in the working tree). Co-Authored-By: Claude Fable 5 --- scripts/host/backup_daaf.ps1 | 48 +++++- scripts/host/backup_daaf.sh | 48 +++++- scripts/host/migrate_daaf.sh | 15 +- scripts/host/update_daaf.ps1 | 17 ++ scripts/host/update_daaf.sh | 17 +- tests/bash/backup_daaf.bats | 145 +++++++++++++++++- tests/bash/migrate_daaf.bats | 111 ++++++++++++++ tests/bash/update_daaf.bats | 77 ++++++++++ tests/powershell/backup_daaf.Tests.ps1 | 27 ++++ tests/powershell/migrate_daaf.Tests.ps1 | 13 ++ tests/powershell/update_daaf.Tests.ps1 | 11 ++ .../01_installation_and_quickstart.md | 2 +- 12 files changed, 509 insertions(+), 22 deletions(-) diff --git a/scripts/host/backup_daaf.ps1 b/scripts/host/backup_daaf.ps1 index 590e75c3..d55d2f13 100644 --- a/scripts/host/backup_daaf.ps1 +++ b/scripts/host/backup_daaf.ps1 @@ -422,6 +422,14 @@ try { $CopyProcess.WaitForExit() $CopyExitCode = if ($null -ne $CopyProcess.ExitCode) { $CopyProcess.ExitCode } else { 0 } +# Latch for the completion banner: any of the non-fatal WARNING paths below +# (file-count mismatch, size mismatch, Claude-state copy failure, permission-manifest +# failure) sets this so the final banner can say the backup completed WITH WARNINGS +# rather than claiming an unqualified success. The exit status stays 0 on those paths +# -- the banner wording is the signal, not the exit code (the corroborated short-copy +# case below is the one that is fatal, and it exits before the banner). +$HadWarnings = $false + # --- Verify --- # The staged tree the copy streamed = volume regular files + 1 symlink manifest # - symlinks. Symlinks were never counted by the source scan (TotalFiles), and the @@ -452,16 +460,35 @@ if ($FileCount -eq 0) { Wait-AndExit 1 } -if ($CopyExitCode -ne 0) { +# --- Corroborated short-copy check (fatal) --- +# A nonzero docker cp exit AND a copied count short of the volume scan are two +# independent signals that agree the backup is truncated -- the nonzero exit +# corroborates the shortfall, so no tolerance is applied here (unlike the 1% count +# and size tolerances below, which absorb benign filesystem/metadata drift on an +# otherwise-clean copy). A truncated backup that passed as usable could cost a user +# their data on restore, so fail hard: name the partial folder, tell them to delete +# it, and exit before the completion banner. (A nonzero exit with a FULL count falls +# through to the Note below -- warnings without an actual shortfall.) +if ($CopyExitCode -ne 0 -and $FileCount -lt $TotalFiles) { + Write-Host "" + Write-Host "ERROR: Backup failed (exit code $CopyExitCode); only $FileCount of $TotalFiles expected files were copied." -ForegroundColor Red + Write-Host " This backup is incomplete and must not be relied on. Delete this" + Write-Host " partial backup folder and re-run once the copy issue is resolved." + Write-Host "Location: $HostPath\" + Wait-AndExit 1 +} elseif ($CopyExitCode -ne 0) { Write-Host "Note: File copy reported warnings (exit code $CopyExitCode); $FileCount of $TotalFiles expected files were transferred." -ForegroundColor Yellow } # --- File-count verification --- # Do NOT trust docker cp's exit code as the sole failure signal: the Windows -# symlink-abort truncation surfaced with a ZERO exit on the user's run. Compare the -# copied data-file count (manifest excluded) against the source scan count and warn -# loudly on a shortfall beyond a 1% tolerance -- the count and size checks are the -# authoritative completeness signals, not $CopyExitCode. +# symlink-abort truncation surfaced with a ZERO exit on the user's run. A nonzero +# exit that ALSO comes up short on the count is already fatal (the corroborated +# short-copy branch above). This block catches the harder case -- a clean ZERO exit +# that still copied too few files -- comparing the copied data-file count (manifest +# excluded) against the source scan count and warning loudly on a shortfall beyond a +# 1% tolerance. The count and size checks are the authoritative completeness signals, +# not $CopyExitCode. if ($TotalFiles -gt 0 -and $FileCount -gt 0) { $CountTolerance = [math]::Max(1, [long]($TotalFiles / 100)) $CountDiff = [math]::Abs($TotalFiles - $FileCount) @@ -470,6 +497,7 @@ if ($TotalFiles -gt 0 -and $FileCount -gt 0) { Write-Host "WARNING: Backup file-count mismatch." -ForegroundColor Yellow Write-Host " Source: $TotalFiles files, Backup: $FileCount files (difference: $CountDiff)" Write-Host " The backup may be incomplete. Consider re-running." + $HadWarnings = $true } } @@ -489,6 +517,7 @@ if ($SourceSizeKB -gt 0 -and $BackupSizeKB -gt 0) { Write-Host "WARNING: Backup size mismatch." -ForegroundColor Yellow Write-Host " Source: ${SourceSizeKB} KB, Backup: ${BackupSizeKB} KB (difference: ${DiffKB} KB)" Write-Host " The backup may be incomplete. Consider re-running." + $HadWarnings = $true } } @@ -575,6 +604,7 @@ if ($claudeVolumeExists) { Write-Host " (no details could be retrieved from the staging container)" } Write-Host " The data volume backup above is still valid." + $HadWarnings = $true } } else { Write-Host "" @@ -639,16 +669,22 @@ if ($manifestScanOk) { Write-Host "WARNING: Could not record the executable-permission manifest." -ForegroundColor Yellow Write-Host " The backup is still valid; on restore, file permissions may need" Write-Host " manual repair if this backup is restored on a Windows host." + $HadWarnings = $true } } else { Write-Host "WARNING: Could not record the executable-permission manifest." -ForegroundColor Yellow Write-Host " The backup is still valid; on restore, file permissions may need" Write-Host " manual repair if this backup is restored on a Windows host." + $HadWarnings = $true } Write-Host "" Write-Host "==========================================" -Write-Host " Backup complete!" +if ($HadWarnings) { + Write-Host " Backup completed WITH WARNINGS -- verify before relying on it" +} else { + Write-Host " Backup complete!" +} Write-Host "==========================================" Write-Host "" Write-Host "Location: $HostPath\" diff --git a/scripts/host/backup_daaf.sh b/scripts/host/backup_daaf.sh index d598f961..a01cd674 100755 --- a/scripts/host/backup_daaf.sh +++ b/scripts/host/backup_daaf.sh @@ -376,6 +376,14 @@ wait "${COPY_PID}" || COPY_EXIT=$? docker rm -f "${STAGE_CID}" > /dev/null 2>&1 || true trap - INT TERM +# Latch for the completion banner: any of the non-fatal WARNING paths below +# (file-count mismatch, size mismatch, Claude-state copy failure, permission-manifest +# failure) sets this so the final banner can say the backup completed WITH WARNINGS +# rather than claiming an unqualified success. The exit status stays 0 on those paths +# -- the banner wording is the signal, not the exit code (the corroborated short-copy +# case below is the one that is fatal, and it exits before the banner). +HAD_WARNINGS=0 + # --- Verify --- # The staged tree the copy streamed = volume regular files + 1 symlink manifest # - symlinks. Symlinks were never counted by the `find -type f` volume scan @@ -407,16 +415,35 @@ if [ "${FILE_COUNT}" -eq 0 ]; then exit 1 fi -if [ "${COPY_EXIT}" -ne 0 ]; then +# --- Corroborated short-copy check (fatal) --- +# A nonzero docker cp exit AND a copied count short of the volume scan are two +# independent signals that agree the backup is truncated -- the nonzero exit +# corroborates the shortfall, so no tolerance is applied here (unlike the 1% count +# and size tolerances below, which absorb benign filesystem/metadata drift on an +# otherwise-clean copy). A truncated backup that passed as usable could cost a user +# their data on restore, so fail hard: name the partial folder, tell them to delete +# it, and exit before the completion banner. (A nonzero exit with a FULL count falls +# through to the Note below -- warnings without an actual shortfall.) +if [ "${COPY_EXIT}" -ne 0 ] && [ "${FILE_COUNT}" -lt "${TOTAL_FILES}" ]; then + echo "" + echo "ERROR: Backup failed (exit code ${COPY_EXIT}); only ${FILE_COUNT} of ${TOTAL_FILES} expected files were copied." + echo " This backup is incomplete and must not be relied on. Delete this" + echo " partial backup folder and re-run once the copy issue is resolved." + echo "Location: $(pwd)/${BACKUP_NAME}/" + exit 1 +elif [ "${COPY_EXIT}" -ne 0 ]; then echo "Note: File copy reported warnings (exit code ${COPY_EXIT}); ${FILE_COUNT} of ${TOTAL_FILES} expected files were transferred." fi # --- File-count verification --- # Do NOT trust docker cp's exit code as the sole failure signal: the Windows -# symlink-abort truncation surfaced with a ZERO exit on the user's run. Compare the -# copied data-file count (manifest excluded) against the source scan count and warn -# loudly on a shortfall beyond a 1% tolerance -- the count and size checks are the -# authoritative completeness signals, not COPY_EXIT. +# symlink-abort truncation surfaced with a ZERO exit on the user's run. A nonzero +# exit that ALSO comes up short on the count is already fatal (the corroborated +# short-copy branch above). This block catches the harder case -- a clean ZERO exit +# that still copied too few files -- comparing the copied data-file count (manifest +# excluded) against the source scan count and warning loudly on a shortfall beyond a +# 1% tolerance. The count and size checks are the authoritative completeness signals, +# not COPY_EXIT. if [ "${TOTAL_FILES}" -gt 0 ] && [ "${FILE_COUNT}" -gt 0 ]; then COUNT_TOLERANCE=$(( TOTAL_FILES / 100 )) if [ "${COUNT_TOLERANCE}" -lt 1 ]; then COUNT_TOLERANCE=1; fi @@ -427,6 +454,7 @@ if [ "${TOTAL_FILES}" -gt 0 ] && [ "${FILE_COUNT}" -gt 0 ]; then echo "WARNING: Backup file-count mismatch." >&2 echo " Source: ${TOTAL_FILES} files, Backup: ${FILE_COUNT} files (difference: ${COUNT_DIFF})" >&2 echo " The backup may be incomplete. Consider re-running." >&2 + HAD_WARNINGS=1 fi fi @@ -450,6 +478,7 @@ if [ "${SOURCE_SIZE_KB}" -gt 0 ] && [ "${BACKUP_SIZE_KB}" -gt 0 ]; then echo "WARNING: Backup size mismatch." >&2 echo " Source: ${SOURCE_SIZE_KB} KB, Backup: ${BACKUP_SIZE_KB} KB (difference: ${DIFF_KB} KB)" >&2 echo " The backup may be incomplete. Consider re-running." >&2 + HAD_WARNINGS=1 fi fi @@ -510,6 +539,7 @@ if docker volume inspect "${CLAUDE_VOLUME_NAME}" &> /dev/null; then echo " (no details could be retrieved from the staging container)" >&2 fi echo " The data volume backup above is still valid." >&2 + HAD_WARNINGS=1 fi docker rm -f "${CLAUDE_STAGE_CID}" > /dev/null 2>&1 || true trap - INT TERM @@ -567,16 +597,22 @@ if EXEC_PATHS=$(docker run --rm -v "${VOLUME_NAME}:/source:ro" busybox sh -c 'fi echo "WARNING: Could not record the executable-permission manifest." >&2 echo " The backup is still valid; on restore, file permissions may need" >&2 echo " manual repair if this backup is restored on a Windows host." >&2 + HAD_WARNINGS=1 fi else echo "WARNING: Could not record the executable-permission manifest." >&2 echo " The backup is still valid; on restore, file permissions may need" >&2 echo " manual repair if this backup is restored on a Windows host." >&2 + HAD_WARNINGS=1 fi echo "" echo "==========================================" -echo " Backup complete!" +if [ "${HAD_WARNINGS}" -eq 1 ]; then + echo " Backup completed WITH WARNINGS -- verify before relying on it" +else + echo " Backup complete!" +fi echo "==========================================" echo "" echo "Location: $(pwd)/${BACKUP_NAME}/" diff --git a/scripts/host/migrate_daaf.sh b/scripts/host/migrate_daaf.sh index 7aed835a..c3ff46cb 100755 --- a/scripts/host/migrate_daaf.sh +++ b/scripts/host/migrate_daaf.sh @@ -373,8 +373,21 @@ if [ "${DAAF_DRY_RUN:-}" = "1" ]; then else ORIGINAL_DIR="$(pwd)" cd "${HOST_DIR}" - DAAF_NESTED=1 bash backup_daaf.sh + # Capture the backup exit explicitly. Under this script's `set -euo pipefail` an + # unguarded failed backup would abort the migration abruptly with no explanation; + # `|| BACKUP_EXIT=$?` defuses set -e so we can abort gracefully with a clear + # message instead. This backup is mandatory (not prompted), so any failure stops + # the migration -- mirroring migrate_daaf.ps1's existing gated abort. + BACKUP_EXIT=0 + DAAF_NESTED=1 bash backup_daaf.sh || BACKUP_EXIT=$? cd "${ORIGINAL_DIR}" + if [ "${BACKUP_EXIT}" -ne 0 ]; then + echo "" + echo "ERROR: Backup failed (exit code ${BACKUP_EXIT})." + echo "The migration will not proceed without a successful backup." + echo "Please resolve the backup issue and re-run: bash migrate_daaf.sh" + exit 1 + fi BACKUP_COMPLETED=true fi diff --git a/scripts/host/update_daaf.ps1 b/scripts/host/update_daaf.ps1 index d1d70193..c240c062 100644 --- a/scripts/host/update_daaf.ps1 +++ b/scripts/host/update_daaf.ps1 @@ -1404,12 +1404,29 @@ if (Test-Path "backup_daaf.ps1") { # above): a bare Remove-Item would clobber a value inherited from a # parent process. $savedNested = $env:DAAF_NESTED + # Defensive pre-init: if the try below unwinds before $backupExit is + # assigned, the nonzero default makes the gate fail closed (abort) rather + # than compare against $null. + $backupExit = 1 try { $env:DAAF_NESTED = "1" & .\backup_daaf.ps1 + $backupExit = $LASTEXITCODE } finally { if ($null -ne $savedNested) { $env:DAAF_NESTED = $savedNested } else { Remove-Item Env:\DAAF_NESTED -ErrorAction SilentlyContinue } } + # A user who opted into a backup and had it fail must not have the update + # proceed on a missing restore point (a DECLINED backup -- the else path below + # -- still lets the update continue). The .sh twin runs under `set -e`, so it + # guards the call with `|| BACKUP_EXIT=$?`; here we read $LASTEXITCODE, which + # PS 5.1 does not gate. Mirrors migrate_daaf.ps1's gated abort. + if ($backupExit -ne 0) { + Write-Host "" + Write-Host "ERROR: Backup failed (exit code $backupExit)." + Write-Host "The update will not proceed without a successful backup." + Write-Host "Please resolve the backup issue and re-run: .\update_daaf.ps1" + Wait-AndExit 1 + } Write-Host "" } } else { diff --git a/scripts/host/update_daaf.sh b/scripts/host/update_daaf.sh index 5ac09450..13531779 100755 --- a/scripts/host/update_daaf.sh +++ b/scripts/host/update_daaf.sh @@ -1312,7 +1312,22 @@ if [ -f "backup_daaf.sh" ]; then CHOICE=$(prompt_choice " Run backup now? [y/n]: " "y n") if [ "${CHOICE}" = "y" ]; then echo "" - DAAF_NESTED=1 bash backup_daaf.sh + # Capture the backup exit explicitly. Under this script's `set -euo pipefail` + # an unguarded failed backup would abort the update abruptly with no + # explanation; `|| BACKUP_EXIT=$?` defuses set -e so we can abort gracefully + # with a clear message instead (mirrors migrate_daaf.ps1's gated abort). A + # user who opted into a backup and had it fail must not have the update + # proceed on a missing restore point; a DECLINED backup (the else path below) + # still lets the update continue. + BACKUP_EXIT=0 + DAAF_NESTED=1 bash backup_daaf.sh || BACKUP_EXIT=$? + if [ "${BACKUP_EXIT}" -ne 0 ]; then + echo "" + echo "ERROR: Backup failed (exit code ${BACKUP_EXIT})." + echo "The update will not proceed without a successful backup." + echo "Please resolve the backup issue and re-run: bash update_daaf.sh" + exit 1 + fi echo "" fi else diff --git a/tests/bash/backup_daaf.bats b/tests/bash/backup_daaf.bats index a9f3e1e9..43984333 100755 --- a/tests/bash/backup_daaf.bats +++ b/tests/bash/backup_daaf.bats @@ -772,19 +772,22 @@ teardown() { assert_output --partial "ERROR" } -@test "backup: copy partially succeeds with non-zero exit but files copied shows warning" { +@test "backup: copy reports non-zero exit but full file count shows Note not error" { export DAAF_NESTED=1 - # Custom docker mock: staging succeeds; the `docker cp` copy returns non-zero - # but still creates files. + # Regression anchor for the Note path: a nonzero `docker cp` exit is NOT fatal + # when the full expected file count still landed (no shortfall). The scan reports + # 3 files and the mock copies exactly 3 -- so the corroborated short-copy fatal + # branch (nonzero exit AND count short) does NOT fire; execution falls through to + # the informational Note and completes successfully. docker() { case "$1" in info) return 0 ;; volume) return 0 ;; run) if [ "${2:-}" = "-d" ]; then echo "stagecid0000"; return 0; fi - # Scan output - printf '5\n512\t/source\n500K\t/source\n500\n' + # Scan output -- 3 files, matching the 3 the copy creates below. + printf '3\n512\t/source\n500K\t/source\n500\n' return 0 ;; wait) echo "0"; return 0 ;; @@ -796,7 +799,7 @@ teardown() { mkdir -p "${dest_dir}" touch "${dest_dir}/file1" "${dest_dir}/file2" "${dest_dir}/file3" fi - # Non-zero exit (partial failure) + # Non-zero exit, but the full count still landed return 1 ;; rm) return 0 ;; @@ -806,9 +809,137 @@ teardown() { export -f docker run bash "${REPO_ROOT}/scripts/host/backup_daaf.sh" - # Should succeed (files were copied) but note the warnings + # Should succeed (full count copied) but note the warnings -- NOT the fatal error. assert_output --partial "warnings" assert_output --partial "files were transferred" + refute_output --partial "must not be relied on" +} + +@test "backup: non-zero copy exit with short file count is a fatal error" { + export DAAF_NESTED=1 + + # Corroborated short copy: the `docker cp` copy returns non-zero AND fewer files + # land (2) than the scan counted (5). The two signals agree the backup is + # truncated, so the script must abort fatally, naming the partial folder and + # telling the user to delete it -- not pass it off as usable. + docker() { + case "$1" in + info) return 0 ;; + volume) return 0 ;; + run) + if [ "${2:-}" = "-d" ]; then echo "stagecid0000"; return 0; fi + # Scan reports 5 files; the copy below lands only 2. + printf '5\n512\t/source\n500K\t/source\n500\n' + return 0 + ;; + wait) echo "0"; return 0 ;; + cp) + local dest_dir="" + for arg in "$@"; do dest_dir="${arg}"; done + dest_dir="${dest_dir%/}" + if [ -n "${dest_dir}" ]; then + mkdir -p "${dest_dir}" + touch "${dest_dir}/file1" "${dest_dir}/file2" + fi + # Non-zero exit with a short count -> fatal + return 1 + ;; + rm) return 0 ;; + *) return 0 ;; + esac + } + export -f docker + + run bash "${REPO_ROOT}/scripts/host/backup_daaf.sh" + assert_failure + assert_output --partial "ERROR" + assert_output --partial "only 2 of 5 expected files were copied" + assert_output --partial "must not be relied on" + # Names the partial backup folder so the user can delete it. + assert_output --partial "Location:" + assert_output --partial "_daaf_backup" +} + +@test "backup: zero-exit short file count warns (not fatal) and banner notes warnings" { + export DAAF_NESTED=1 + + # A clean ZERO copy exit that still lands too few files must NOT be fatal (the + # fatal branch requires a corroborating nonzero exit). It is a WARNING: the scan + # reports 100 files, the copy lands 2, the exit is 0. The file-count-mismatch + # WARNING fires and the completion banner reads WITH WARNINGS, but the script + # still exits 0 and never prints the fatal short-copy error. Logical KB is 0 so + # the size check is skipped, isolating the file-count signal. + docker() { + case "$1" in + info) return 0 ;; + volume) return 0 ;; + run) + if [ "${2:-}" = "-d" ]; then echo "stagecid0000"; return 0; fi + # 100 files, logical KB 0 (skips size verification) + printf '100\n512\t/source\n500K\t/source\n0\n' + return 0 + ;; + wait) echo "0"; return 0 ;; + cp) + local dest_dir="" + for arg in "$@"; do dest_dir="${arg}"; done + dest_dir="${dest_dir%/}" + if [ -n "${dest_dir}" ]; then + mkdir -p "${dest_dir}" + touch "${dest_dir}/file1" "${dest_dir}/file2" + fi + return 0 + ;; + rm) return 0 ;; + *) return 0 ;; + esac + } + export -f docker + + run bash "${REPO_ROOT}/scripts/host/backup_daaf.sh" + assert_success + assert_output --partial "file-count mismatch" + assert_output --partial "Backup completed WITH WARNINGS -- verify before relying on it" + refute_output --partial "must not be relied on" +} + +@test "backup: completion banner reads plain 'Backup complete!' on a clean run" { + export DAAF_NESTED=1 + + # A fully clean run -- full file count, no size drift (logical KB 0 skips the + # size check), Claude state and permission manifest both succeed -- must print + # the plain "Backup complete!" banner and NOT the WITH-WARNINGS variant. + docker() { + case "$1" in + info) return 0 ;; + volume) return 0 ;; + run) + if [ "${2:-}" = "-d" ]; then echo "stagecid0000"; return 0; fi + # 2 files, logical KB 0 (skips size verification) + printf '2\n512\t/source\n500K\t/source\n0\n' + return 0 + ;; + wait) echo "0"; return 0 ;; + cp) + local dest_dir="" + for arg in "$@"; do dest_dir="${arg}"; done + dest_dir="${dest_dir%/}" + if [ -n "${dest_dir}" ]; then + mkdir -p "${dest_dir}" + touch "${dest_dir}/file1" "${dest_dir}/file2" + fi + return 0 + ;; + rm) return 0 ;; + *) return 0 ;; + esac + } + export -f docker + + run bash "${REPO_ROOT}/scripts/host/backup_daaf.sh" + assert_success + assert_output --partial "Backup complete!" + refute_output --partial "WITH WARNINGS" } @test "backup: volume scan outputs unexpected format triggers error" { diff --git a/tests/bash/migrate_daaf.bats b/tests/bash/migrate_daaf.bats index 08d03470..1e177bab 100755 --- a/tests/bash/migrate_daaf.bats +++ b/tests/bash/migrate_daaf.bats @@ -735,6 +735,117 @@ SH assert_output --partial "Migration complete" } +# --- Pre-migration backup gate (section 3) --- +# migrate's backup is MANDATORY (not prompted): section 3 captures the exit with +# `BACKUP_EXIT=0; DAAF_NESTED=1 bash backup_daaf.sh || BACKUP_EXIT=$?` and aborts +# on any nonzero code with a clear message. These are real-run-path tests (the +# dry-run path skips the nested backup call entirely, keeping BACKUP_COMPLETED=true). +# The full-run harness's curl mock writes every downloaded host script, INCLUDING +# backup_daaf.sh, so the exit code is controlled by special-casing that file in the +# curl mock rather than by a pre-seeded stub (which the download would clobber). + +@test "migrate: backup failure aborts the migration with a clear message" { + setup_migrate_integrated + run bash -c ' + cd "'"${TEST_DIR}"'" + curl() { + local outfile="" + local args=("$@") + for ((i=0; i<${#args[@]}; i++)); do + if [ "${args[$i]}" = "-o" ]; then + outfile="${args[$((i+1))]}" + break + fi + done + if [ -n "${outfile}" ]; then + mkdir -p "$(dirname "${outfile}")" + echo "#!/usr/bin/env bash" > "${outfile}" + if [ "$(basename "${outfile}")" = "backup_daaf.sh" ]; then + # The mandatory backup fails: emit a marker, then exit nonzero. + echo "echo \"STUB BACKUP RAN\"" >> "${outfile}" + echo "exit 1" >> "${outfile}" + else + echo "exit 0" >> "${outfile}" + fi + fi + return 0 + } + export -f curl + docker() { + local all_args="$*" + case "$all_args" in + "info") return 0 ;; + *"volume inspect"*) return 0 ;; + *"ps -a"*"--filter"*"volume="*"--format"*) echo "daaf-test-1" ;; + *"inspect --format"*"State.Status"*) echo "running" ;; + *) return 0 ;; + esac + } + export -f docker + bash "'"${REPO_ROOT}"'/scripts/host/migrate_daaf.sh" + ' + assert_failure + assert_output --partial "STUB BACKUP RAN" + assert_output --partial "ERROR: Backup failed (exit code 1)" + assert_output --partial "The migration will not proceed without a successful backup." + # The abort happens in section 3, so no section-4 post-backup step is reached. + refute_output --partial "Starting container" + refute_output --partial "Migration complete" +} + +@test "migrate: backup success lets the migration proceed past the backup step" { + setup_migrate_integrated + run bash -c ' + cd "'"${TEST_DIR}"'" + curl() { + local outfile="" + local args=("$@") + for ((i=0; i<${#args[@]}; i++)); do + if [ "${args[$i]}" = "-o" ]; then + outfile="${args[$((i+1))]}" + break + fi + done + if [ -n "${outfile}" ]; then + mkdir -p "$(dirname "${outfile}")" + echo "#!/usr/bin/env bash" > "${outfile}" + if [ "$(basename "${outfile}")" = "backup_daaf.sh" ]; then + # The mandatory backup succeeds: emit a marker, then exit 0. + echo "echo \"STUB BACKUP RAN\"" >> "${outfile}" + echo "exit 0" >> "${outfile}" + else + echo "exit 0" >> "${outfile}" + fi + fi + return 0 + } + export -f curl + docker() { + local all_args="$*" + case "$all_args" in + "info") return 0 ;; + *"volume inspect"*) return 0 ;; + *"ps -a"*"--filter"*"volume="*"--format"*) echo "daaf-test-1" ;; + *"inspect --format"*"State.Status"*) echo "running" ;; + *"exec"*"true"*) return 0 ;; + *"exec"*"test -f"*"CLAUDE.md"*) return 0 ;; + *"exec"*"remote get-url"*"origin"*) echo "https://github.com/DAAF-Contribution-Community/daaf.git" ;; + *"exec"*"fetch"*) return 0 ;; + *"exec"*"branch --set-upstream"*) return 0 ;; + *"exec"*"remote get-url"*"upstream"*) echo "" ; return 1 ;; + *) return 0 ;; + esac + } + export -f docker + bash "'"${REPO_ROOT}"'/scripts/host/migrate_daaf.sh" + ' + assert_success + assert_output --partial "STUB BACKUP RAN" + # The gate must NOT false-fire, and the flow must reach the post-backup step. + refute_output --partial "The migration will not proceed without a successful backup." + assert_output --partial "Starting container" +} + @test "migrate: already migrated (idempotency) skips graft via replace-ref leg" { # Idempotency via LEG 1 (replace-ref): a non-empty `git replace -l` marks the # graft as already done. cat-file returns NO parent here, so the ONLY thing diff --git a/tests/bash/update_daaf.bats b/tests/bash/update_daaf.bats index ab4d8d14..fd74845e 100755 --- a/tests/bash/update_daaf.bats +++ b/tests/bash/update_daaf.bats @@ -1439,6 +1439,83 @@ setup_state_machine() { assert_output --partial "Failed to start" } +# --- Pre-update backup gate --- +# The updater's pre-update backup is prompted/optional. In the non-interactive test +# harness prompt_choice auto-selects the FIRST valid choice ("y"), so these tests +# exercise the opted-in path: a FAILED backup must abort the update with a clear +# message, and a SUCCESSFUL backup must let it continue (the abort must not +# false-fire). A declined backup (CHOICE=n) is unreachable non-interactively, so it +# is not exercised here. + +@test "update: opted-in backup failure aborts the update with a clear message" { + setup_state_machine + # Stub backup script the updater will find in the CWD and run; it fails (exit 1). + cat > "${TEST_DIR}/backup_daaf.sh" <<'STUB' +#!/usr/bin/env bash +echo "STUB BACKUP RAN" +exit 1 +STUB + chmod +x "${TEST_DIR}/backup_daaf.sh" + run bash -c ' + docker() { + local all_args="$*" + case "$all_args" in + "info") return 0 ;; + *"compose ps"*"--format"*) echo "daaf-docker" ;; + *"compose exec"*"true"*) return 0 ;; + *"compose exec"*"test -f"*"/daaf/CLAUDE.md"*) return 0 ;; + *) return 0 ;; + esac + } + export -f docker + bash "'"${REPO_ROOT}"'/scripts/host/update_daaf.sh" + ' + assert_failure + assert_output --partial "STUB BACKUP RAN" + assert_output --partial "Backup failed (exit code 1)" + assert_output --partial "The update will not proceed without a successful backup" +} + +@test "update: opted-in backup success lets the update continue past the backup step" { + setup_state_machine + # Stub backup script that succeeds (exit 0); the update must NOT abort and should + # proceed to its normal outcome (same SHA everywhere -> "Already up to date"). + cat > "${TEST_DIR}/backup_daaf.sh" <<'STUB' +#!/usr/bin/env bash +echo "STUB BACKUP RAN" +exit 0 +STUB + chmod +x "${TEST_DIR}/backup_daaf.sh" + run bash -c ' + docker() { + local all_args="$*" + case "$all_args" in + "info") return 0 ;; + *"compose ps"*"--format"*) echo "daaf-docker" ;; + *"compose exec"*"true"*) return 0 ;; + *"compose exec"*"test -f"*"/daaf/CLAUDE.md"*) return 0 ;; + *"compose exec"*"test -f"*"/daaf/.git/shallow"*) return 1 ;; + *"compose exec"*"remote get-url"*"origin"*) echo "https://github.com/DAAF-Contribution-Community/daaf.git" ;; + *"compose exec"*"fetch"*) return 0 ;; + *"compose exec"*"rev-parse --verify"*"backup/"*) return 1 ;; + *"compose exec"*"rev-parse --verify"*"origin/main"*) return 0 ;; + *"compose exec"*"branch --show-current"*) echo "main" ;; + *"compose exec"*"rev-parse"*"origin/main"*) echo "abc123same" ;; + *"compose exec"*"rev-parse"*"HEAD"*) echo "abc123same" ;; + *"compose exec"*"diff --name-only"*"HEAD"*) echo "" ;; + *"compose exec"*"branch"*) return 0 ;; + *) return 0 ;; + esac + } + export -f docker + bash "'"${REPO_ROOT}"'/scripts/host/update_daaf.sh" + ' + assert_success + assert_output --partial "STUB BACKUP RAN" + assert_output --partial "Already up to date" + refute_output --partial "The update will not proceed without a successful backup" +} + @test "update: DAAF not installed (CLAUDE.md missing) exits with error" { setup_state_machine run bash -c ' diff --git a/tests/powershell/backup_daaf.Tests.ps1 b/tests/powershell/backup_daaf.Tests.ps1 index 40e8daeb..c0972e8d 100644 --- a/tests/powershell/backup_daaf.Tests.ps1 +++ b/tests/powershell/backup_daaf.Tests.ps1 @@ -457,6 +457,33 @@ Describe "backup_daaf.ps1 error paths" { } } + Context "Corroborated short copy is fatal" { + It "fails when a non-zero copy exit AND a short file count agree" { + # Two corroborating signals (nonzero exit + count below the scan) mean a + # genuinely truncated backup, so the script aborts fatally. + $Content | Should -Match '\$CopyExitCode -ne 0 -and \$FileCount -lt \$TotalFiles' + $Content | Should -Match 'only \$FileCount of \$TotalFiles expected files were copied' + } + It "tells the user to delete the partial backup folder and re-run" { + $Content | Should -Match 'This backup is incomplete and must not be relied on' + $Content | Should -Match 'partial backup folder and re-run' + } + } + + Context "Warnings-aware completion banner" { + It "prints the plain banner when no warnings were latched" { + $Content | Should -Match 'Backup complete!' + } + It "prints the WITH WARNINGS banner when the latch is set" { + $Content | Should -Match 'Backup completed WITH WARNINGS -- verify before relying on it' + } + It "latches HadWarnings at the non-fatal WARNING sites" { + # Initialized false, and set true in each non-fatal WARNING path. + $Content | Should -Match '\$HadWarnings = \$false' + $Content | Should -Match '\$HadWarnings = \$true' + } + } + Context "Volume scan unexpected format" { It "handles scan failure with error" { # When docker run for scan fails (LASTEXITCODE != 0), script exits with error diff --git a/tests/powershell/migrate_daaf.Tests.ps1 b/tests/powershell/migrate_daaf.Tests.ps1 index 3382cc33..fa7ce8fe 100644 --- a/tests/powershell/migrate_daaf.Tests.ps1 +++ b/tests/powershell/migrate_daaf.Tests.ps1 @@ -108,6 +108,19 @@ Describe "migrate_daaf.ps1" { $Content | Should -Match 'backup_daaf\.ps1' } + It "captures the backup exit code from LASTEXITCODE" { + $Content | Should -Match '\$backupExit\s*=\s*\$LASTEXITCODE' + } + + It "aborts the migration on a nonzero backup exit code" { + $Content | Should -Match 'if \(\$backupExit -ne 0\)' + $Content | Should -Match 'Backup failed \(exit code \$backupExit\)' + } + + It "explains that the migration will not proceed without a successful backup" { + $Content | Should -Match 'The migration will not proceed without a successful backup' + } + It "handles fork detection" { $Content | Should -Match 'IsFork' } diff --git a/tests/powershell/update_daaf.Tests.ps1 b/tests/powershell/update_daaf.Tests.ps1 index c0d41b18..43affbba 100644 --- a/tests/powershell/update_daaf.Tests.ps1 +++ b/tests/powershell/update_daaf.Tests.ps1 @@ -108,6 +108,17 @@ Describe "update_daaf.ps1" { $Content | Should -Match 'Backup recommendation' } + It "captures the opt-in backup exit code" { + # The pre-update backup is prompted/optional; its exit must be captured + # so a failure can abort gracefully instead of proceeding silently. + $Content | Should -Match '\$backupExit = \$LASTEXITCODE' + } + + It "aborts the update when the opted-in backup fails" { + $Content | Should -Match 'Backup failed \(exit code \$backupExit\)' + $Content | Should -Match 'The update will not proceed without a successful backup' + } + It "creates a git backup branch" { $Content | Should -Match 'backup/pre-update' } diff --git a/user_reference/01_installation_and_quickstart.md b/user_reference/01_installation_and_quickstart.md index 210bb896..fce70e75 100644 --- a/user_reference/01_installation_and_quickstart.md +++ b/user_reference/01_installation_and_quickstart.md @@ -391,7 +391,7 @@ cd daaf-docker The backup script creates a date-versioned folder (e.g., `2026-04-21_daaf_backup/`) in your `daaf-docker` directory. Multiple backups on the same day are automatically suffixed (`2026-04-21a_daaf_backup/`, `2026-04-21b_daaf_backup/`, etc.). Feel free to move or copy these folders to another location on your computer (or an external drive) for safekeeping. -The backup script also checks itself as it goes: it verifies there's enough free disk space before it starts (and stops with a clear message if there isn't), and after copying it compares the backup against the original — checking both the number of files and their total size — so a truncated or incomplete backup gets flagged with a warning rather than passing silently. +The backup script also checks itself as it goes: it verifies there's enough free disk space before it starts (and stops with a clear message if there isn't), and after copying it compares the backup against the original — checking both the number of files and their total size. How a shortfall is handled depends on how serious it is. If the copy step itself reports an error *and* fewer files landed than the volume scan expected, the backup is treated as genuinely incomplete: the script stops with an error and a nonzero exit code, and tells you to delete the partial backup folder and re-run once the problem is resolved (any tool that ran the backup for you — the updater or the migration tool — then halts too, rather than continuing on top of a bad backup). A milder discrepancy — the copy reported success but the file count or total size is still a little off — is flagged as a warning instead: the backup finishes, but its completion banner reads `Backup completed WITH WARNINGS -- verify before relying on it` rather than the plain `Backup complete!`, so a questionable backup never passes silently. The backup covers both DAAF volumes: your research data, plus Claude Code's own state (your login, session history, and plugins) in a hidden `.daaf-claude-config/` subfolder. **Because the backup includes your Claude Code login credentials, treat backup folders as sensitive** — store them somewhere private, and if you share a backup with a colleague for collaboration, delete the `.daaf-claude-config/` subfolder from the copy first. (The backup also contains one or two small hidden manifest files you can ignore — the restore consumes them automatically. `.daaf-permissions` records which files were executable, so the restore can put file permissions back correctly even when the backup was stored on a Windows drive, which does not preserve them. `.daaf-symlinks`, present only when your data contains symbolic links, records those links so the restore can recreate them — this is what lets backups complete cleanly on Windows, where the copy step would otherwise stop at the first symbolic link.) From 624cd72beac3375182449ba70471ad44472399a8 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Sat, 18 Jul 2026 17:31:01 +0000 Subject: [PATCH 03/66] fix(settings): protect logs and Claude JSONL files Co-Authored-By: Claude --- .claude/settings.json | 8 +- .../01_Harden_Log_Write_Protection.md | 309 ++++++++++++++++++ tests/bash/settings_permissions.bats | 134 ++++++++ 3 files changed, 449 insertions(+), 2 deletions(-) create mode 100644 research/2026-07-18_DAAF_To_Do/01_Harden_Log_Write_Protection.md create mode 100644 tests/bash/settings_permissions.bats diff --git a/.claude/settings.json b/.claude/settings.json index 461a1485..3c25fb5f 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -94,8 +94,12 @@ "Edit(environment_settings.txt)", "Edit(.claude/hooks/*)", "Write(.claude/hooks/*)", - "Edit(.claude/logs/*)", - "Write(.claude/logs/*)", + "Edit(/.claude/logs/**)", + "Write(/.claude/logs/**)", + "Edit(//home/appuser/.claude/**/*.jsonl)", + "Write(//home/appuser/.claude/**/*.jsonl)", + "Edit(/research/*/logs/**)", + "Write(/research/*/logs/**)", "Write(//tmp/**)", "Edit(//tmp/**)" ] diff --git a/research/2026-07-18_DAAF_To_Do/01_Harden_Log_Write_Protection.md b/research/2026-07-18_DAAF_To_Do/01_Harden_Log_Write_Protection.md new file mode 100644 index 00000000..9e1fa473 --- /dev/null +++ b/research/2026-07-18_DAAF_To_Do/01_Harden_Log_Write_Protection.md @@ -0,0 +1,309 @@ +# To Do: Harden Shell Write Protection for Logs and Claude Session JSONL Files + +**Status:** Deferred — ready for a future Framework Development session +**Priority:** High +**Origin date:** 2026-07-18 +**Work type:** Modify Existing — safety-critical Bash hook and regression coverage + +## Purpose + +Extend DAAF's shell-command safety boundary so ordinary Bash-mediated mutations cannot directly alter archived DAAF logs, research-project logs, or Claude Code JSONL state. The work must preserve path-only and metadata-only inspection while blocking direct write primitives whenever a protected destination is visible in the submitted command. + +This document is self-contained. A future session should still re-read the live files and verify all assumptions before editing because the hook, Claude Code, and repository may have changed after 2026-07-18. + +## Problem Statement + +DAAF currently uses two distinct enforcement layers: + +1. Claude Code permission rules in `/daaf/.claude/settings.json` govern built-in file-editing tools. +2. `/daaf/.claude/hooks/bash-safety.sh` inspects submitted Bash command text before execution. + +The settings layer now denies built-in edits for these path classes: + +```text +/.claude/logs/** +//home/appuser/.claude/**/*.jsonl +/research/*/logs/** +``` + +Each path has both `Edit(...)` and `Write(...)` entries. `Edit(...)` is the forward-compatible operative form in current Claude Code documentation. Retaining `Write(...)` for the installed Claude Code 2.1.202 is a conservative compatibility **inference** from the documented version boundary, not behavior proven by current documentation or a runtime evaluator. When DAAF upgrades to Claude Code 2.1.210 or later, remove the path-scoped `Write(...)` entries: the current documentation says they are accepted but unmatched at those versions and generate startup warnings. + +Those settings rules do **not** make Bash writes safe. The live hook already detects several direct mutations of project `.claude/logs`, but its protection is command-form-dependent and does not presently define `/home/appuser/.claude/**/*.jsonl` or research-project `logs/` directories as protected target classes. Wrapper commands, interpreters, alternate redirections, and runtime path indirection create additional gaps. + +## Protected Path Classes + +The future hook change must cover: + +1. DAAF project logs: + ```text + /daaf/.claude/logs/** + .claude/logs/** + /.claude/logs/** + ``` + +2. Research-project archived logs, where one direct child of `/daaf/research` is the project folder: + ```text + /daaf/research//logs/** + research//logs/** + /research//logs/** + ``` + +3. Every `.jsonl` beneath the Claude configuration volume: + ```text + /home/appuser/.claude/**/*.jsonl + ~/.claude/**/*.jsonl + $HOME/.claude/**/*.jsonl + ${HOME}/.claude/**/*.jsonl + ``` + The home-variable forms can be blocked only when recognizable in the submitted command. + +### Absolute content boundary + +Never open, display, grep inside, parse, hash, line-count, copy, move, touch, truncate, or otherwise consume or alter the contents of a real `.jsonl` file while developing or testing this change. Path names and filesystem metadata—type, mode, owner, byte size, and timestamps—are sufficient for inventory. Tests must feed synthetic command strings to the hook; they must not execute the represented mutations. + +## Objective + +Modify `bash-safety.sh` so a submitted Bash command is denied when it visibly attempts to mutate a protected target through a supported write primitive, including when that primitive appears behind a common wrapper or interpreter carrier. + +The hook should remain fail-closed on parsing ambiguity affecting these safety-critical paths and should produce an actionable denial message that identifies the protected log/transcript boundary without printing file contents. + +## Non-Goals + +This task does not promise an OS-level write prohibition. In particular, it does not guarantee detection of: + +- destinations assembled entirely from runtime variables; +- encoded or dynamically decoded paths; +- paths inherited through an already-established current working directory; +- writes through previously opened file descriptors; +- interpreter programs that derive the destination without a recognizable protected fragment; +- kernel-, process-, mount-, or container-level mutations outside Claude Code's tool path. + +A later design may investigate sandbox `denyWrite`, mount policy, or filesystem permissions as a separate defense. Do not add those mechanisms silently to this task: they have broader compatibility and deployment implications and require their own scoping checkpoint. + +## Required Mutation Coverage + +For each protected path class, map and test the following families. + +### Redirections and shell file opens + +Block visible protected destinations used with: + +```text +> +>> +>| +<> +2> and other numbered file-descriptor redirects +3>> and other numbered appends +&> +&>> +``` + +Include here-document and here-string commands when their output is redirected to a protected destination. Account for quoting, repeated slashes, and intervening `./` segments. + +### Direct filesystem mutation tools + +Cover at minimum: + +| Family | Commands / forms | +|---|---| +| Stream output | `tee`, including append mode | +| Copy/move/sync | `cp`, `mv`, `rsync`, `install` | +| Removal/linking | `rm`, `ln` | +| Raw/write utilities | `dd`, `truncate`, `touch` | +| In-place editing | `sed -i` and accepted equivalent in-place forms | +| Metadata mutation | `chmod`, `chown`, `chgrp`, `chattr` where available | + +Do not rely solely on the mutating command appearing at the start of a shell segment. Test destinations followed by trailing options as well as conventional destination-final forms. + +### Wrappers and alternate command positions + +Map and block protected mutations routed through: + +```text +find ... -exec +xargs +env +command +time +nice +pipeline-position commands +``` + +The scanner should distinguish a read-only `find` inventory from a mutating `find -exec` action. + +### Shell and interpreter carriers + +When a protected target is visible in the submitted command, cover mutation programs carried by: + +```text +bash -c +sh -c +python / python3 +perl +ruby +node +``` + +The design must explicitly document whether it blocks any interpreter invocation containing a protected literal or attempts a narrower write-capability test. For safety-critical transcript paths, favor a clear fail-closed rule over fragile parsing of arbitrary program text, but include ALLOW controls to measure false positives. + +### Working-directory shortcuts + +Evaluate `cd` and `pushd` into protected directories. DAAF's one-command-per-call hook reduces ordinary multi-command state changes, but shell carriers and persistent shell behavior may still make short relative writes possible. Document what is blocked and what remains beyond lexical inspection. + +## Design Constraint: Lexical Inspection Is Not Semantic Resolution + +`bash-safety.sh` receives command text, not the process's eventual system calls. It can normalize and inspect recognizable path fragments, command tokens, quoting, and redirections; it cannot prove the runtime destination of every write. + +The completed change must state this limitation in: + +- hook comments near the protected-target logic; +- the regression battery's scope statement; +- the relevant Defense-in-Depth documentation if user approval includes that file; +- the final Framework Development checkpoint. + +Do not describe the result as making JSONL files immutable or impossible to modify. The accurate claim is that DAAF blocks the tested direct and wrapper-mediated Bash forms when a protected destination is statically recognizable. + +## Likely Files to Modify + +Read every target in full before editing and confirm the list during Framework Development Checkpoint 1. + +| File | Expected role | +|---|---| +| `/daaf/.claude/hooks/bash-safety.sh` | Live safety hook; human-controlled deployment target only after a staged draft passes validation | +| `/daaf/scripts/test_safety_hooks.sh` | Standing synthetic command-envelope regression battery; add primitive × path-class cases | +| `/daaf/tests/bash/bash_safety.bats` | Focused BATS coverage using the `BASH_SAFETY_SH` draft override | +| `/daaf/CLAUDE.md` | Update the Defense-in-Depth description only with explicit user approval | +| `/daaf/.claude/skills/shell-scripting/references/testing.md` | Update battery documentation if invocation or coverage contracts change | +| A dated Framework Development workspace | Hold `bash-safety.sh.proposed`, session notes, and any losslessly persisted review/research notes | + +The current static settings test is `/daaf/tests/bash/settings_permissions.bats`. It verifies the structured-tool rules but is not a Bash-hook test and must not be repurposed to claim runtime matcher coverage. + +## Staged Draft and Human-Controlled Deployment + +The live hook is safety-protected. Follow this sequence: + +1. Start a new Framework Development session and complete Phase 1 scoping. +2. Run the existing live-hook regression suites to establish a green baseline. +3. Copy the live hook into a dated research workspace as a proposed draft using the repository's approved workflow. +4. Modify only the staged draft during authoring. +5. Test the draft with: + ```bash + bash /daaf/scripts/test_safety_hooks.sh /absolute/path/to/bash-safety.sh.proposed + BASH_SAFETY_SH=/absolute/path/to/bash-safety.sh.proposed bats /daaf/tests/bash/bash_safety.bats + ``` +6. Run parser and ShellCheck validation on the draft. +7. Complete adversarial, consistency, and completeness reviews. +8. Present Framework Development Checkpoint 2. +9. Only after explicit user approval, have the user deploy the byte-identical tested draft to `/daaf/.claude/hooks/bash-safety.sh` through a host editor or a user-typed command. +10. Verify the deployed hook is byte-identical to the tested draft and that Git still records mode `100755`. +11. Re-run the live-hook suites and workspace invariant checker. + +Do not use opaque shell writes to modify the live hook. Do not weaken or deregister any safety hook to make deployment easier. + +## Required Regression Matrix + +All cases must live inside the battery/BATS files and reach the hook through a synthetic PreToolUse JSON envelope. Never put dangerous case strings in the outer invocation command. + +### BLOCK controls + +For representative targets from all three protected path classes, include: + +- direct and nested paths; +- project-relative, settings-root-like, and absolute forms; +- single-quoted and double-quoted targets; +- repeated `/` and embedded `./` segments; +- every supported redirection; +- every direct mutation command listed above; +- trailing-option variants for copy-style commands; +- wrapper forms (`find -exec`, `xargs`, `env`, `command`, `time`, `nice`); +- shell carriers and each supported interpreter; +- `~`, `$HOME`, and `${HOME}` home-path spellings; +- `cd`/`pushd` cases chosen during design. + +Use synthetic examples such as `example.jsonl`. Do not point a case at a real session UUID or archive path when a representative fake path proves the rule. + +### ALLOW controls + +Preserve legitimate metadata and similarly named non-target work: + +- `stat` on a protected path; +- path-only `find ... -printf` without mutating `-exec` actions; +- `findmnt` for mount metadata; +- directories named `logs` outside the three protected classes; +- a non-JSONL file under `/home/appuser/.claude/`; +- a `.jsonl` path outside `/home/appuser/.claude/`; +- safe commands that merely discuss a protected pattern in a shell-inert commit message, consistent with the existing quote-aware carve-out; +- false-positive probes for interpreter commands that do not write, according to the selected fail-closed design. + +### Anti-vacuity requirements + +- Include at least one self-test or deliberate draft mutation proving that the new test selection fails when one protected class is removed. +- Report counts from actual test output, not memory. +- Keep project-log, research-log, and Claude-home JSONL cases separately identifiable in failure output. + +## Validation Commands + +Probe each tool before declaring it unavailable. Expected validation includes: + +```bash +bash -n /absolute/path/to/bash-safety.sh.proposed +shellcheck -x -S warning /absolute/path/to/bash-safety.sh.proposed +bash /daaf/scripts/test_safety_hooks.sh /absolute/path/to/bash-safety.sh.proposed +BASH_SAFETY_SH=/absolute/path/to/bash-safety.sh.proposed bats /daaf/tests/bash/bash_safety.bats +git -C /daaf diff --check +bash /daaf/scripts/check_workspace_invariants.sh +git -C /daaf ls-files -s .claude/hooks/bash-safety.sh scripts/test_safety_hooks.sh +``` + +After human deployment, additionally require an empty `diff` between the proposed and live hooks and rerun the battery against the live hook. + +## Review Plan + +Because this is a Moderate safety-critical hook modification, run three independent read-only reviews in one complete wave: + +1. **Consistency review:** path classes, docs, tests, denial messages, and duplicated descriptions agree. +2. **Security/quality review:** adversarially search for direct forms the live hook blocks but the draft newly allows, new bypasses in the added logic, false positives, parser ambiguity, and performance/pathological-input issues. +3. **Completeness review:** verify `/daaf/agent_reference/FRAMEWORK_INTEGRATION_CHECKLIST.md` §5 items HM1–HM7, executable modes, test wiring, and documentation surfaces. + +Do not synthesize or act on partial review returns; wait for the complete wave. + +## Acceptance Criteria + +The task is complete only when all of the following are observed: + +- [ ] Phase 1 re-verifies the live hook and affected files before edits. +- [ ] Project `.claude/logs`, direct research-project `logs`, and Claude-home `.jsonl` targets have explicit tested coverage. +- [ ] The required direct primitives, redirection variants, wrappers, and interpreter carriers are represented in tests. +- [ ] Read-only metadata ALLOW controls remain green. +- [ ] Existing safety-hook regression cases remain green. +- [ ] The staged draft passes `bash -n` and ShellCheck. +- [ ] The complete safety battery and focused BATS suite pass against the staged draft. +- [ ] All three reviews return and any confirmed issues are resolved or explicitly accepted by the user. +- [ ] Workspace invariants pass. +- [ ] The user approves deployment. +- [ ] The deployed live hook is byte-identical to the tested draft and remains Git mode `100755`. +- [ ] Post-deployment live suites pass. +- [ ] The final report distinguishes tested lexical coverage from unproven OS-level immutability. + +## Dependencies and Scope Boundaries + +- Use Framework Development mode and obtain explicit approval before modifying settings, hooks, or CLAUDE.md. +- Follow `/daaf/agent_reference/FRAMEWORK_INTEGRATION_CHECKLIST.md` §5 HM1–HM7. +- Follow `/daaf/.claude/skills/shell-scripting/references/testing.md` for draft-path battery and BATS conventions. +- Preserve the absolute prohibition on consuming real JSONL contents. +- Do not install runtime packages. +- Do not modify CI workflows without separate authorization. +- Do not commit unless the repository's Git commit preference is enabled and the user separately approves the commit. +- Do not expand this task into sandbox, mount, ownership, retention, archival, or backup-policy changes without a new scope checkpoint. + +## References + +- Claude Code permissions documentation: https://code.claude.com/docs/en/permissions +- Universal DAAF rules: `/daaf/CLAUDE.md` +- Hook integration checklist: `/daaf/agent_reference/FRAMEWORK_INTEGRATION_CHECKLIST.md` §5 HM1–HM7 +- Shell test guidance: `/daaf/.claude/skills/shell-scripting/references/testing.md` +- Current static settings contract: `/daaf/tests/bash/settings_permissions.bats` +- Live hook: `/daaf/.claude/hooks/bash-safety.sh` +- Standing shell battery: `/daaf/scripts/test_safety_hooks.sh` +- Focused Bash safety BATS suite: `/daaf/tests/bash/bash_safety.bats` diff --git a/tests/bash/settings_permissions.bats b/tests/bash/settings_permissions.bats new file mode 100644 index 00000000..44e7088b --- /dev/null +++ b/tests/bash/settings_permissions.bats @@ -0,0 +1,134 @@ +#!/usr/bin/env bats +# ============================================================================ +# Static contract tests for checked-in Claude Code permission rules. +# +# This suite validates the checked-in configuration contract in +# .claude/settings.json. It does NOT dynamically prove Claude Code permission- +# matcher semantics; runtime matcher behavior remains the responsibility of the +# installed Claude Code version and its authoritative documentation. +# ============================================================================ + +load 'test_helper' + +setup() { common_setup; } +teardown() { common_teardown; } + +SETTINGS_FILE="${REPO_ROOT}/.claude/settings.json" +EXPECTED_RULES_JSON='[ + "Edit(/.claude/logs/**)", + "Write(/.claude/logs/**)", + "Edit(//home/appuser/.claude/**/*.jsonl)", + "Write(//home/appuser/.claude/**/*.jsonl)", + "Edit(/research/*/logs/**)", + "Write(/research/*/logs/**)" +]' +EXPECTED_PATHS=( + '/.claude/logs/**' + '//home/appuser/.claude/**/*.jsonl' + '/research/*/logs/**' +) +NEAR_RULES_JSON='[ + "Read(/.claude/logs/**)", + "Read(//home/appuser/.claude/**/*.jsonl)", + "Edit(.claude/logs/**)", + "Write(research/*/logs/**)", + "Edit(/.claude/./logs/**)", + "Write(//home/appuser//.claude/**/*.jsonl)", + "Edit(/research/*/./logs/**)", + "Read(/.claude/././logs/**)", + "Write(//home/appuser/././.claude/**/*.jsonl)", + "Edit(/research/*/././logs/**)" +]' +NON_MEMBER_RULES_JSON='[ + "Read(/archive/.claude/logs/**)", + "Read(/archive/home/appuser/.claude/**/*.jsonl)", + "Read(/archive/research/project/logs/**)", + "Read(//home/appuser/.claude/cache/example.jsonl.bak)", + "Read(/ordinary/path/**)" +]' + +# Select any permission rule targeting one of the three protected path families. +# Structural segment normalization removes every repeated slash and /./ segment +# while preserving whether the original path had zero, one, or two leading +# slashes. Direct anchors exclude unrelated archive/lookalike paths; exact set +# equality below then rejects every selected variant except the six approved +# strings. +PROTECTED_FAMILY_FILTER='def normalized_path: + (if startswith("//") then "//" elif startswith("/") then "/" else "" end) as $anchor + | (if startswith("//") then .[2:] elif startswith("/") then .[1:] else . end) as $rest + | ($rest | split("/") | map(select(. != "" and . != ".")) | join("/")) as $body + | $anchor + $body; + [.permissions.deny[] + | (capture("^(?[^()]+)\\((?.*)\\)$")?) as $rule + | select($rule != null) + | ($rule.path | normalized_path) as $path + | select( + ($path | test("^(//|/)?\\.claude/logs(/|$)")) + or (($path | test("^(//|/)?home/appuser/\\.claude(/|$)")) and ($path | test("\\.jsonl$"))) + or ($path | test("^(//|/)?research/[^/]+/logs(/|$)")) + ) + | .] | sort' + +@test "settings permissions: checked-in settings JSON parses" { + run jq empty "${SETTINGS_FILE}" + assert_success +} + +@test "settings permissions: all six approved deny rules are present" { + run jq -e --argjson expected "${EXPECTED_RULES_JSON}" ' + . as $settings + | all($expected[]; . as $rule | $settings.permissions.deny | index($rule) != null) + ' "${SETTINGS_FILE}" + assert_success +} + +@test "settings permissions: obsolete non-recursive log rules are absent" { + run jq -e ' + [.permissions.deny[] + | select(. == "Edit(.claude/logs/*)" or . == "Write(.claude/logs/*)")] + | length == 0 + ' "${SETTINGS_FILE}" + assert_success +} + +@test "settings permissions: protected log and transcript subset is exactly six rules" { + run jq -e \ + --argjson expected "${EXPECTED_RULES_JSON}" \ + "${PROTECTED_FAMILY_FILTER} == (\$expected | sort)" \ + "${SETTINGS_FILE}" + assert_success +} + +@test "settings permissions: classifier includes near-rules and excludes lookalikes" { + run jq -n -e \ + --argjson expected "${EXPECTED_RULES_JSON}" \ + --argjson extras "${NEAR_RULES_JSON}" \ + --argjson nonmembers "${NON_MEMBER_RULES_JSON}" \ + "{permissions: {deny: (\$expected + \$extras + \$nonmembers)}} + | ${PROTECTED_FAMILY_FILTER} as \$observed + | (all(\$extras[]; . as \$rule | \$observed | index(\$rule) != null)) + and (all(\$nonmembers[]; . as \$rule | \$observed | index(\$rule) == null)) + and (\$observed != (\$expected | sort))" + assert_success +} + +@test "settings permissions: every approved path has exact Edit and Write variants" { + local path + for path in "${EXPECTED_PATHS[@]}"; do + run jq -e --arg path "${path}" ' + ["Edit(" + $path + ")", "Write(" + $path + ")"] as $expected + | [.permissions.deny[] | select(. == $expected[0] or . == $expected[1])] | sort + == ($expected | sort) + ' "${SETTINGS_FILE}" + assert_success + done +} + +@test "settings permissions: existing tmp Edit and Write deny rules remain present" { + run jq -e ' + . as $settings + | ["Edit(//tmp/**)", "Write(//tmp/**)"] + | all(. as $rule | $settings.permissions.deny | index($rule) != null) + ' "${SETTINGS_FILE}" + assert_success +} From a776c151d39a82c885a1632f5e3c6e35a14a772c Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Sat, 18 Jul 2026 23:27:58 +0000 Subject: [PATCH 04/66] fix(host): harden lifecycle scripts against parse, rounding, env, and container-leak defects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified and fixed five defects reported by a downstream port review (sixth report — docs defaults ambiguity — refuted with evidence; cosmetic DEFAULT: tags added anyway): - backup_daaf.ps1: boolean-gate [int]::TryParse staging-status reads at both volumes (fail-closed on non-numeric docker wait output; TryParse writes 0 into the [ref] on parse failure, defeating the =1 default); [math]::Floor at all five division sites (drop banker's rounding — [long](350/100) = 4 vs bash's floor 3) - backup_daaf.sh: reap the staging helper container on launch failure (main + Claude volume), preserving the captured CID instead of blanking it, matching the .ps1 twin's existing pattern - restore_from_backup.sh: same launch-guard parity hardening for both docker create sites (insurance-level; .ps1 twin already reaps in finally) - rebuild_daaf.ps1: save/restore BUILDX_BUILDER around the diagnostic build instead of destroying a user's own exported selection ($env:VAR = $null removes the variable) - all 10 PowerShell settings loaders (4 reported + 6 discovered by repo-wide sweep): reject whitespace-padded keys, matching the bash loaders' column-0 case-glob (symmetric strictness; loader regions md5-identical across all copies) - environment_settings_example.txt: tag SHIM_PORT / SHIM_BACKEND_BASE_URL example values as defaults Tests: Pester + bats regressions pinning each fixed pattern, with pre-fix/fixed discrimination proofs; twin-parity launch-failure reap coverage; padded-key conformance pair on both platforms. All suites green; convention lint 0 failures. Co-Authored-By: Claude Fable 5 --- scripts/host/backup_daaf.ps1 | 37 +++-- scripts/host/backup_daaf.sh | 34 ++++- scripts/host/daaf_lib.ps1 | 6 +- scripts/host/environment_settings_example.txt | 2 + scripts/host/rebuild_daaf.ps1 | 21 ++- scripts/host/restore_from_backup.ps1 | 6 +- scripts/host/restore_from_backup.sh | 29 +++- scripts/host/run_daaf.ps1 | 6 +- scripts/host/run_vscode.ps1 | 6 +- scripts/host/update_daaf.ps1 | 6 +- scripts/host/view_logs.ps1 | 6 +- scripts/host/view_notebooks.ps1 | 6 +- scripts/host/view_quarto.ps1 | 6 +- tests/bash/backup_daaf.bats | 134 ++++++++++++++++++ tests/bash/backup_twin_parity.bats | 85 +++++++++++ tests/bash/daaf_lib.bats | 17 +++ tests/bash/restore_from_backup.bats | 100 +++++++++++++ tests/powershell/backup_daaf.Tests.ps1 | 90 ++++++++++++ tests/powershell/rebuild_daaf.Tests.ps1 | 24 +++- tests/powershell/run_daaf.Tests.ps1 | 10 ++ tests/powershell/run_vscode.Tests.ps1 | 10 ++ tests/powershell/update_daaf.Tests.ps1 | 50 +++++++ tests/powershell/view_logs.Tests.ps1 | 10 ++ tests/powershell/view_notebooks.Tests.ps1 | 10 ++ tests/powershell/view_quarto.Tests.ps1 | 10 ++ 25 files changed, 690 insertions(+), 31 deletions(-) diff --git a/scripts/host/backup_daaf.ps1 b/scripts/host/backup_daaf.ps1 index d55d2f13..bdfaf168 100644 --- a/scripts/host/backup_daaf.ps1 +++ b/scripts/host/backup_daaf.ps1 @@ -83,7 +83,11 @@ function Import-DaafSettingsInline { if ($trimmed -eq "" -or $trimmed.StartsWith("#")) { continue } $eq = $line.IndexOf("=") if ($eq -lt 1) { continue } - $key = $line.Substring(0, $eq).Trim() + # Extract the key WITHOUT trimming: a leading or trailing space means the + # line is not flush at column 0, so it must fall through as unrecognized -- + # matching the bash loaders' column-0 `case` glob so a padded key like + # " DAAF_PROJECT_NAME=..." is rejected identically on both platforms. + $key = $line.Substring(0, $eq) if ($known -notcontains $key) { continue } $val = $line.Substring($eq + 1) if (($val.StartsWith('"') -and $val.EndsWith('"')) -or ($val.StartsWith("'") -and $val.EndsWith("'"))) { @@ -221,9 +225,9 @@ Write-Host "" # staging-failure error below points the user at the Docker Desktop disk image. $BackupDrive = (Get-Item -Path ".").PSDrive.Name $DriveInfo = New-Object System.IO.DriveInfo($BackupDrive) -$AvailableKB = [long]($DriveInfo.AvailableFreeSpace / 1024) +$AvailableKB = [long][math]::Floor($DriveInfo.AvailableFreeSpace / 1024) # Add 10% buffer to account for filesystem overhead -$RequiredKB = [long]($VolumeSizeKB * 110 / 100) +$RequiredKB = [long][math]::Floor($VolumeSizeKB * 110 / 100) if ($AvailableKB -lt $RequiredKB) { $RequiredMB = [math]::Floor($RequiredKB / 1024) $AvailableMB = [math]::Floor($AvailableKB / 1024) @@ -343,8 +347,18 @@ if (-not $stageStartOk) { $savedEAP = $ErrorActionPreference; $ErrorActionPreference = "SilentlyContinue" $StageStatusRaw = (docker wait $StageCid 2>&1 | Select-Object -Last 1) $ErrorActionPreference = $savedEAP +# Gate on TryParse's BOOLEAN return. [int]::TryParse writes 0 into the [ref] on a +# parse FAILURE (not just a parse of "0"), so the old `$null = TryParse(...)` form +# silently overwrote the fail-closed default with 0 whenever `docker wait` emitted a +# non-numeric last line (e.g. a daemon error merged via 2>&1) -- flipping this fatal +# gate from "abort" to "proceed to copy a partial staged tree." Parse into a temp var +# and adopt it ONLY on success; otherwise stay fail-closed at 1 (mirrors the correct +# idiom at restore_from_backup.ps1:279). $StageStatus = 1 -if ($null -ne $StageStatusRaw) { $null = [int]::TryParse("$StageStatusRaw".Trim(), [ref]$StageStatus) } +$StageStatusParsed = 0 +if ($null -ne $StageStatusRaw -and [int]::TryParse("$StageStatusRaw".Trim(), [ref]$StageStatusParsed)) { + $StageStatus = $StageStatusParsed +} if ($StageStatus -ne 0) { # The staging container is DETACHED (`docker run -d`), so the gate's STDOUT -- # including the offender list it prints on an exit-3 -- went to the container @@ -490,7 +504,7 @@ if ($CopyExitCode -ne 0 -and $FileCount -lt $TotalFiles) { # 1% tolerance. The count and size checks are the authoritative completeness signals, # not $CopyExitCode. if ($TotalFiles -gt 0 -and $FileCount -gt 0) { - $CountTolerance = [math]::Max(1, [long]($TotalFiles / 100)) + $CountTolerance = [math]::Max(1, [long][math]::Floor($TotalFiles / 100)) $CountDiff = [math]::Abs($TotalFiles - $FileCount) if ($CountDiff -gt $CountTolerance) { Write-Host "" @@ -507,10 +521,10 @@ $SourceSizeKB = $VolumeLogicalKB # Exclude the ".daaf-symlinks" manifest from the backup byte sum: it exists in the # backup but not the volume, and the source side (VolumeLogicalKB) never counted it # -- so counting it here would skew the comparison. -$BackupSizeKB = [long]((Get-ChildItem -Path $BackupName -Recurse -Force -ErrorAction SilentlyContinue | Where-Object { $_.Name -ne $SymlinksManifest } | Measure-Object -Property Length -Sum).Sum / 1024) +$BackupSizeKB = [long][math]::Floor((Get-ChildItem -Path $BackupName -Recurse -Force -ErrorAction SilentlyContinue | Where-Object { $_.Name -ne $SymlinksManifest } | Measure-Object -Property Length -Sum).Sum / 1024) if ($SourceSizeKB -gt 0 -and $BackupSizeKB -gt 0) { # Allow 1% tolerance for filesystem metadata differences - $ToleranceKB = [math]::Max(1, [long]($SourceSizeKB / 100)) + $ToleranceKB = [math]::Max(1, [long][math]::Floor($SourceSizeKB / 100)) $DiffKB = [math]::Abs($SourceSizeKB - $BackupSizeKB) if ($DiffKB -gt $ToleranceKB) { Write-Host "" @@ -557,8 +571,15 @@ if ($claudeVolumeExists) { $savedEAP = $ErrorActionPreference; $ErrorActionPreference = "SilentlyContinue" $ClaudeStageStatusRaw = (docker wait $ClaudeStageCid 2>&1 | Select-Object -Last 1) $ErrorActionPreference = $savedEAP + # Gate on TryParse's BOOLEAN return (see the data-volume staging block + # above): a parse failure writes 0 into the [ref], which would flip this + # gate from "skip the copy" to "copy a partial staged tree." Parse into a + # temp var; stay fail-closed at 1 on failure. $ClaudeStageStatus = 1 - if ($null -ne $ClaudeStageStatusRaw) { $null = [int]::TryParse("$ClaudeStageStatusRaw".Trim(), [ref]$ClaudeStageStatus) } + $ClaudeStageStatusParsed = 0 + if ($null -ne $ClaudeStageStatusRaw -and [int]::TryParse("$ClaudeStageStatusRaw".Trim(), [ref]$ClaudeStageStatusParsed)) { + $ClaudeStageStatus = $ClaudeStageStatusParsed + } if ($ClaudeStageStatus -eq 0) { $savedEAP = $ErrorActionPreference; $ErrorActionPreference = "SilentlyContinue" $null = docker cp "${ClaudeStageCid}:/staging/." "${ClaudeDestPath}" 2>&1 diff --git a/scripts/host/backup_daaf.sh b/scripts/host/backup_daaf.sh index a01cd674..80251da1 100755 --- a/scripts/host/backup_daaf.sh +++ b/scripts/host/backup_daaf.sh @@ -305,6 +305,12 @@ if ! STAGE_CID="$(docker run -d -v "${VOLUME_NAME}:/source:ro" busybox sh -c "${ echo "" >&2 echo "ERROR: Could not start the staging container for a symlink-safe backup." >&2 echo " No backup files were written. Check Docker Desktop for errors and re-run." >&2 + # Best-effort reap: `docker run -d` can fail AFTER creating the container + # (Created/Exited state) while still printing its CID to stdout, which the + # command substitution above captured -- remove it so a launch failure does not + # leak a helper container (mirrors backup_daaf.ps1's stage-start cleanup). The + # `${STAGE_CID:-}` guard makes this a harmless no-op if no CID was captured. + docker rm -f "${STAGE_CID:-}" > /dev/null 2>&1 || true exit 1 fi trap 'docker rm -f "${STAGE_CID:-}" > /dev/null 2>&1 || true' INT TERM @@ -504,16 +510,32 @@ if docker volume inspect "${CLAUDE_VOLUME_NAME}" &> /dev/null; then # replays it against the claude volume. # Guard the launch so a `docker run -d` failure does NOT abort the whole backup # under `set -e` -- the Claude backup is best-effort (failure must be a WARNING, - # never fatal; the data backup above already succeeded). `|| CLAUDE_STAGE_CID=""` - # swallows the exit status and leaves an empty CID, so the `docker wait` below - # yields status 1 and the `if` falls into the existing WARNING path (mirrors the - # `|| CLAUDE_CID=""` idiom in restore_from_backup.sh's Claude restore block). - CLAUDE_STAGE_CID="$(docker run -d -v "${CLAUDE_VOLUME_NAME}:/source:ro" busybox sh -c "${STAGE_PROGRAM}")" || CLAUDE_STAGE_CID="" + # never fatal; the data backup above already succeeded). Capture the launch + # outcome in a SEPARATE flag rather than blanking the CID: `docker run -d` can + # fail AFTER creating the container (Created/Exited) while still printing its CID, + # and the old `|| CLAUDE_STAGE_CID=""` discarded that CID before any cleanup could + # reap it -- leaking the helper container. Keep the captured CID intact so the + # trap below and the `docker rm -f` at the end of this block can remove it + # (mirrors backup_daaf.ps1's $claudeStageStartOk flag + finally-reap pattern). + # Assigning inside the `if` condition keeps `set -e` from aborting on a launch + # failure while still capturing whatever the substitution produced. + if CLAUDE_STAGE_CID="$(docker run -d -v "${CLAUDE_VOLUME_NAME}:/source:ro" busybox sh -c "${STAGE_PROGRAM}")"; then + CLAUDE_STAGE_START_OK=1 + else + CLAUDE_STAGE_START_OK=0 + fi # Re-register an interrupt trap around this block: the data-copy trap was # cleared above, so without this a Ctrl-C during the Claude staging/cp window # would leak the helper container. Best-effort removal; guarded under set -u. trap 'docker rm -f "${CLAUDE_STAGE_CID:-}" > /dev/null 2>&1 || true' INT TERM - CLAUDE_STAGE_STATUS="$(docker wait "${CLAUDE_STAGE_CID}" 2>/dev/null || echo 1)" + # Only wait on a container that actually launched; a launch failure is treated as + # a nonzero staging status so control flows into the WARNING path below (with the + # CID preserved so the reap at the end of this block can remove it). + if [ "${CLAUDE_STAGE_START_OK}" -eq 1 ]; then + CLAUDE_STAGE_STATUS="$(docker wait "${CLAUDE_STAGE_CID}" 2>/dev/null || echo 1)" + else + CLAUDE_STAGE_STATUS=1 + fi # On a staging-gate trip the offender list went to the DETACHED container's log. # Capture it now, before the `docker rm -f` below removes the container, so the # WARNING branch can relay it. Kept asymmetric with the data volume: this stays a diff --git a/scripts/host/daaf_lib.ps1 b/scripts/host/daaf_lib.ps1 index ff05180a..faef00c2 100644 --- a/scripts/host/daaf_lib.ps1 +++ b/scripts/host/daaf_lib.ps1 @@ -110,7 +110,11 @@ function Import-DaafSettingsFile { $eq = $line.IndexOf("=") if ($eq -lt 1) { continue } - $key = $line.Substring(0, $eq).Trim() + # Extract the key WITHOUT trimming: a leading or trailing space means the + # line is not flush at column 0, so it must fall through as unrecognized -- + # matching the bash loaders' column-0 `case` glob so a padded key like + # " DAAF_PROJECT_NAME=..." is rejected identically on both platforms. + $key = $line.Substring(0, $eq) if ($known -notcontains $key) { continue } $val = $line.Substring($eq + 1) diff --git a/scripts/host/environment_settings_example.txt b/scripts/host/environment_settings_example.txt index f7e1ceae..32c4b359 100644 --- a/scripts/host/environment_settings_example.txt +++ b/scripts/host/environment_settings_example.txt @@ -486,7 +486,9 @@ # ---------------------------------------------------------------------------- # Network / backend overrides (SHIM_BACKEND_API_KEY defaults to OPENAI_API_KEY): # --- To activate, remove the leading '#' on the lines you need: --- +# DEFAULT: 4141 is the default port; the line below shows it, so uncommenting it changes nothing. #SHIM_PORT=4141 +# DEFAULT: https://api.openai.com/v1 is the default backend; the line below shows it, so uncommenting it changes nothing. #SHIM_BACKEND_BASE_URL=https://api.openai.com/v1 #SHIM_BACKEND_API_KEY= # diff --git a/scripts/host/rebuild_daaf.ps1 b/scripts/host/rebuild_daaf.ps1 index 6ae9e148..c0ce5c2a 100644 --- a/scripts/host/rebuild_daaf.ps1 +++ b/scripts/host/rebuild_daaf.ps1 @@ -58,7 +58,11 @@ function Import-DaafSettingsInline { if ($trimmed -eq "" -or $trimmed.StartsWith("#")) { continue } $eq = $line.IndexOf("=") if ($eq -lt 1) { continue } - $key = $line.Substring(0, $eq).Trim() + # Extract the key WITHOUT trimming: a leading or trailing space means the + # line is not flush at column 0, so it must fall through as unrecognized -- + # matching the bash loaders' column-0 `case` glob so a padded key like + # " DAAF_PROJECT_NAME=..." is rejected identically on both platforms. + $key = $line.Substring(0, $eq) if ($known -notcontains $key) { continue } $val = $line.Substring($eq + 1) if (($val.StartsWith('"') -and $val.EndsWith('"')) -or ($val.StartsWith("'") -and $val.EndsWith("'"))) { @@ -290,12 +294,23 @@ Write-Host "" # on `docker compose up --progress`, which is rejected as "unknown flag" on # Docker Compose versions prior to ~v2.27. # BUILDX_BUILDER is set only when the diagnostic builder was selected above, then -# cleared right after the build so it does not leak into later docker calls. +# restored right after the build so it does not leak into later docker calls. +# HAZARD (parallel to the bash twin's comment at rebuild_daaf.sh:251-255, whose +# command-scoped `BUILDX_BUILDER=... docker compose build` prefix never mutates the +# process env): assigning `$env:BUILDX_BUILDER = $null` REMOVES the variable entirely +# rather than setting it empty, so a naive clear would DESTROY a user's own +# pre-exported BUILDX_BUILDER selection. Save the prior value first and restore it +# afterwards; remove the var only when the user had none set. ($env: reads are +# strict-mode-safe even when the variable is unset -- they yield $null, not an error.) +$priorBuildxBuilder = $env:BUILDX_BUILDER # $null when the user had none set if ($UseDiagBuilder) { $env:BUILDX_BUILDER = "daaf-diag-builder" } $savedEAP = $ErrorActionPreference; $ErrorActionPreference = "SilentlyContinue" docker compose build --progress plain $ErrorActionPreference = $savedEAP -if ($UseDiagBuilder) { $env:BUILDX_BUILDER = $null } +if ($UseDiagBuilder) { + if ($null -ne $priorBuildxBuilder) { $env:BUILDX_BUILDER = $priorBuildxBuilder } + else { $env:BUILDX_BUILDER = $null } +} if ($LASTEXITCODE -ne 0) { Write-Host "" Write-Host "ERROR: Rebuild failed. Check the output above for details." -ForegroundColor Red diff --git a/scripts/host/restore_from_backup.ps1 b/scripts/host/restore_from_backup.ps1 index d52c3076..303d503c 100644 --- a/scripts/host/restore_from_backup.ps1 +++ b/scripts/host/restore_from_backup.ps1 @@ -50,7 +50,11 @@ function Import-DaafSettingsInline { if ($trimmed -eq "" -or $trimmed.StartsWith("#")) { continue } $eq = $line.IndexOf("=") if ($eq -lt 1) { continue } - $key = $line.Substring(0, $eq).Trim() + # Extract the key WITHOUT trimming: a leading or trailing space means the + # line is not flush at column 0, so it must fall through as unrecognized -- + # matching the bash loaders' column-0 `case` glob so a padded key like + # " DAAF_PROJECT_NAME=..." is rejected identically on both platforms. + $key = $line.Substring(0, $eq) if ($known -notcontains $key) { continue } $val = $line.Substring($eq + 1) if (($val.StartsWith('"') -and $val.EndsWith('"')) -or ($val.StartsWith("'") -and $val.EndsWith("'"))) { diff --git a/scripts/host/restore_from_backup.sh b/scripts/host/restore_from_backup.sh index 3a2cebdc..54c16e2e 100755 --- a/scripts/host/restore_from_backup.sh +++ b/scripts/host/restore_from_backup.sh @@ -396,6 +396,16 @@ trap 'docker rm -f "${CID:-}" > /dev/null 2>&1 || true' INT TERM # was already cleared in Step 1, so the user would be left with an empty volume # and no explanation. Capture into CID first, then check the status. if ! CID="$(docker create -v "${VOLUME_NAME}:/dest" busybox)"; then + # Best-effort reap: `docker create` can in principle fail AFTER creating the + # container while still printing its CID to stdout, which the command + # substitution above captured -- remove it so a launch failure does not leak a + # helper container (mirrors backup_daaf.sh's stage-start reap and + # backup_daaf.ps1's stage-start cleanup). `docker create` is single-phase, so a + # fail-with-CID is near-theoretical here, but this keeps parity with the twin and + # with the `docker cp` failure branch just below. The `${CID:-}` guard makes this + # a harmless no-op if no CID was captured. Reap first, then drop the trap (matches + # the cp-branch order). + docker rm -f "${CID:-}" > /dev/null 2>&1 || true trap - INT TERM echo "" >&2 echo "ERROR: Could not create the helper container for the volume copy." >&2 @@ -628,10 +638,21 @@ if [ "${HAS_CLAUDE_BACKUP}" -eq 1 ]; then trap 'docker rm -f "${CLAUDE_CID:-}" > /dev/null 2>&1 || true' INT TERM # Guard the create under `set -e`: a failure here is non-fatal (the data # volume restore already succeeded), so it must fall into the WARNING path - # below, not abort the whole script. `|| CLAUDE_CID=""` keeps the exit - # status from tripping `set -e` while leaving CLAUDE_CID empty on failure. - CLAUDE_CID="$(docker create -v "${CLAUDE_VOLUME_NAME}:/dest" busybox)" || CLAUDE_CID="" - if [ -n "${CLAUDE_CID}" ] \ + # below, not abort the whole script. Capture the launch outcome in a SEPARATE + # flag rather than blanking the CID: `docker create` can in principle fail + # after emitting a CID, and the old `|| CLAUDE_CID=""` discarded that CID + # before the trap above or the `docker rm -f` at the end of this block could + # reap it -- leaking the helper container. Keep the captured CID intact so both + # reap sites can remove it (mirrors backup_daaf.sh's CLAUDE_STAGE_START_OK flag + # and backup_daaf.ps1's $claudeStageStartOk + finally-reap pattern). Assigning + # inside the `if` condition keeps `set -e` from aborting on a launch failure + # while still capturing whatever the substitution produced. + if CLAUDE_CID="$(docker create -v "${CLAUDE_VOLUME_NAME}:/dest" busybox)"; then + CLAUDE_CREATE_OK=1 + else + CLAUDE_CREATE_OK=0 + fi + if [ "${CLAUDE_CREATE_OK}" -eq 1 ] \ && docker cp "${CLAUDE_BACKUP_PATH}/." "${CLAUDE_CID}:/dest/" \ && docker run --rm -v "${CLAUDE_VOLUME_NAME}:/dest" busybox chown -R 1000:1000 /dest; then # Replay the Claude volume's own symlink manifest (staged out by backup), diff --git a/scripts/host/run_daaf.ps1 b/scripts/host/run_daaf.ps1 index 838d23ea..3c61031f 100644 --- a/scripts/host/run_daaf.ps1 +++ b/scripts/host/run_daaf.ps1 @@ -43,7 +43,11 @@ function Import-DaafSettingsInline { if ($trimmed -eq "" -or $trimmed.StartsWith("#")) { continue } $eq = $line.IndexOf("=") if ($eq -lt 1) { continue } - $key = $line.Substring(0, $eq).Trim() + # Extract the key WITHOUT trimming: a leading or trailing space means the + # line is not flush at column 0, so it must fall through as unrecognized -- + # matching the bash loaders' column-0 `case` glob so a padded key like + # " DAAF_PROJECT_NAME=..." is rejected identically on both platforms. + $key = $line.Substring(0, $eq) if ($known -notcontains $key) { continue } $val = $line.Substring($eq + 1) if (($val.StartsWith('"') -and $val.EndsWith('"')) -or ($val.StartsWith("'") -and $val.EndsWith("'"))) { diff --git a/scripts/host/run_vscode.ps1 b/scripts/host/run_vscode.ps1 index 94fffcc0..169d640e 100644 --- a/scripts/host/run_vscode.ps1 +++ b/scripts/host/run_vscode.ps1 @@ -48,7 +48,11 @@ function Import-DaafSettingsInline { if ($trimmed -eq "" -or $trimmed.StartsWith("#")) { continue } $eq = $line.IndexOf("=") if ($eq -lt 1) { continue } - $key = $line.Substring(0, $eq).Trim() + # Extract the key WITHOUT trimming: a leading or trailing space means the + # line is not flush at column 0, so it must fall through as unrecognized -- + # matching the bash loaders' column-0 `case` glob so a padded key like + # " DAAF_PROJECT_NAME=..." is rejected identically on both platforms. + $key = $line.Substring(0, $eq) if ($known -notcontains $key) { continue } $val = $line.Substring($eq + 1) if (($val.StartsWith('"') -and $val.EndsWith('"')) -or ($val.StartsWith("'") -and $val.EndsWith("'"))) { diff --git a/scripts/host/update_daaf.ps1 b/scripts/host/update_daaf.ps1 index c240c062..30387dd2 100644 --- a/scripts/host/update_daaf.ps1 +++ b/scripts/host/update_daaf.ps1 @@ -148,7 +148,11 @@ function Import-DaafSettingsInline { if ($trimmed -eq "" -or $trimmed.StartsWith("#")) { continue } $eq = $line.IndexOf("=") if ($eq -lt 1) { continue } - $key = $line.Substring(0, $eq).Trim() + # Extract the key WITHOUT trimming: a leading or trailing space means the + # line is not flush at column 0, so it must fall through as unrecognized -- + # matching the bash loaders' column-0 `case` glob so a padded key like + # " DAAF_PROJECT_NAME=..." is rejected identically on both platforms. + $key = $line.Substring(0, $eq) if ($known -notcontains $key) { continue } $val = $line.Substring($eq + 1) if (($val.StartsWith('"') -and $val.EndsWith('"')) -or ($val.StartsWith("'") -and $val.EndsWith("'"))) { diff --git a/scripts/host/view_logs.ps1 b/scripts/host/view_logs.ps1 index 6efbf5fa..7a99dd42 100644 --- a/scripts/host/view_logs.ps1 +++ b/scripts/host/view_logs.ps1 @@ -49,7 +49,11 @@ function Import-DaafSettingsInline { if ($trimmed -eq "" -or $trimmed.StartsWith("#")) { continue } $eq = $line.IndexOf("=") if ($eq -lt 1) { continue } - $key = $line.Substring(0, $eq).Trim() + # Extract the key WITHOUT trimming: a leading or trailing space means the + # line is not flush at column 0, so it must fall through as unrecognized -- + # matching the bash loaders' column-0 `case` glob so a padded key like + # " DAAF_PROJECT_NAME=..." is rejected identically on both platforms. + $key = $line.Substring(0, $eq) if ($known -notcontains $key) { continue } $val = $line.Substring($eq + 1) if (($val.StartsWith('"') -and $val.EndsWith('"')) -or ($val.StartsWith("'") -and $val.EndsWith("'"))) { diff --git a/scripts/host/view_notebooks.ps1 b/scripts/host/view_notebooks.ps1 index 91cb2415..9493e8a4 100644 --- a/scripts/host/view_notebooks.ps1 +++ b/scripts/host/view_notebooks.ps1 @@ -48,7 +48,11 @@ function Import-DaafSettingsInline { if ($trimmed -eq "" -or $trimmed.StartsWith("#")) { continue } $eq = $line.IndexOf("=") if ($eq -lt 1) { continue } - $key = $line.Substring(0, $eq).Trim() + # Extract the key WITHOUT trimming: a leading or trailing space means the + # line is not flush at column 0, so it must fall through as unrecognized -- + # matching the bash loaders' column-0 `case` glob so a padded key like + # " DAAF_PROJECT_NAME=..." is rejected identically on both platforms. + $key = $line.Substring(0, $eq) if ($known -notcontains $key) { continue } $val = $line.Substring($eq + 1) if (($val.StartsWith('"') -and $val.EndsWith('"')) -or ($val.StartsWith("'") -and $val.EndsWith("'"))) { diff --git a/scripts/host/view_quarto.ps1 b/scripts/host/view_quarto.ps1 index 1f918e07..84fab3e2 100644 --- a/scripts/host/view_quarto.ps1 +++ b/scripts/host/view_quarto.ps1 @@ -81,7 +81,11 @@ function Import-DaafSettingsInline { if ($trimmed -eq "" -or $trimmed.StartsWith("#")) { continue } $eq = $line.IndexOf("=") if ($eq -lt 1) { continue } - $key = $line.Substring(0, $eq).Trim() + # Extract the key WITHOUT trimming: a leading or trailing space means the + # line is not flush at column 0, so it must fall through as unrecognized -- + # matching the bash loaders' column-0 `case` glob so a padded key like + # " DAAF_PROJECT_NAME=..." is rejected identically on both platforms. + $key = $line.Substring(0, $eq) if ($known -notcontains $key) { continue } $val = $line.Substring($eq + 1) if (($val.StartsWith('"') -and $val.EndsWith('"')) -or ($val.StartsWith("'") -and $val.EndsWith("'"))) { diff --git a/tests/bash/backup_daaf.bats b/tests/bash/backup_daaf.bats index 43984333..7e2ae55f 100755 --- a/tests/bash/backup_daaf.bats +++ b/tests/bash/backup_daaf.bats @@ -965,3 +965,137 @@ teardown() { assert_failure assert_output --partial "ERROR" } + +# ========================================================================= +# --- Helper-container reap on staging launch failure --- +# ========================================================================= +# `docker run -d` can fail AFTER creating the container (Created/Exited) while +# still printing its CID to stdout, which the command substitution captures. The +# launch guards must reap that captured CID (docker rm -f ) so a launch +# failure does not leak a helper container. These mocks log every docker call to a +# file so the reap can be asserted after `run` (the DOCKER_CALLS array does not +# survive bats's `run` subshell). Each test would FAIL against the pre-fix code +# (main branch: no reap before exit 1; Claude branch: `|| CLAUDE_STAGE_CID=""` +# blanked the CID before any cleanup). + +@test "backup: main staging launch failure reaps the captured helper container" { + export DAAF_NESTED=1 + + # The data-volume `docker run -d` prints a CID but exits nonzero. The error + # branch must run `docker rm -f stagecid0000` before `exit 1`. + docker() { + printf '%s\n' "$*" >> "${TEST_DIR}/docker_calls.log" + case "$1" in + info) return 0 ;; + volume) return 0 ;; + run) + if [ "${2:-}" = "-d" ]; then echo "stagecid0000"; return 1; fi + printf '5\n512\t/source\n500K\t/source\n500\n' + return 0 + ;; + wait) echo "0"; return 0 ;; + cp) return 0 ;; + logs) echo ""; return 0 ;; + rm) return 0 ;; + *) return 0 ;; + esac + } + export -f docker + + run bash "${REPO_ROOT}/scripts/host/backup_daaf.sh" + assert_failure + assert_output --partial "Could not start the staging container" + # The captured CID must have been reaped in the error branch. + run grep -F 'rm -f stagecid0000' "${TEST_DIR}/docker_calls.log" + assert_success +} + +@test "backup: Claude staging launch failure preserves and reaps the captured CID (WARNING, not fatal)" { + export DAAF_NESTED=1 + + # Data-volume backup succeeds; the Claude-volume `docker run -d` prints a CID but + # exits nonzero. The fix keeps the captured CID (not `|| CLAUDE_STAGE_CID=""`) so + # the end-of-block `docker rm -f claudestagecid` reaps it. Claude failure stays a + # WARNING; the script still completes (the data backup is valid). Logical KB 0 + # skips the size check. The `run -d` arm distinguishes the two volumes by the + # "claude-config" substring in the volume mount argument. + docker() { + printf '%s\n' "$*" >> "${TEST_DIR}/docker_calls.log" + case "$1" in + info) return 0 ;; + volume) return 0 ;; + run) + if [ "${2:-}" = "-d" ]; then + if printf '%s' "$*" | grep -q 'claude-config'; then + echo "claudestagecid"; return 1 + fi + echo "stagecid0000"; return 0 + fi + printf '2\n512\t/source\n500K\t/source\n0\n' + return 0 + ;; + wait) echo "0"; return 0 ;; + cp) + local dest_dir="" + for arg in "$@"; do dest_dir="${arg}"; done + dest_dir="${dest_dir%/}" + if [ -n "${dest_dir}" ]; then + mkdir -p "${dest_dir}" + touch "${dest_dir}/file1" "${dest_dir}/file2" + fi + return 0 + ;; + logs) echo ""; return 0 ;; + rm) return 0 ;; + *) return 0 ;; + esac + } + export -f docker + + run bash "${REPO_ROOT}/scripts/host/backup_daaf.sh" + assert_success + assert_output --partial "Failed to back up the Claude Code state volume" + # The captured Claude CID must be reaped -- proving it was NOT blanked before cleanup. + run grep -F 'rm -f claudestagecid' "${TEST_DIR}/docker_calls.log" + assert_success +} + +@test "backup: file-count tolerance floors to 1 for a small backup (clamp parity with PS)" { + export DAAF_NESTED=1 + + # For <100 files, TOTAL_FILES/100 truncates to 0; the clamp raises it to 1 so a + # 1-file difference is tolerated (parity with the PS [math]::Max(1, ...) floor). + # Scan reports 2 files; the data copy lands 3 files (diff = 1). With the clamp the + # difference is within tolerance -> NO file-count-mismatch WARNING. Without the + # clamp (tolerance 0) a diff of 1 would warn. Logical KB 0 skips the size check, + # isolating the count signal. + docker() { + case "$1" in + info) return 0 ;; + volume) return 0 ;; + run) + if [ "${2:-}" = "-d" ]; then echo "stagecid0000"; return 0; fi + printf '2\n512\t/source\n500K\t/source\n0\n' + return 0 + ;; + wait) echo "0"; return 0 ;; + cp) + local dest_dir="" + for arg in "$@"; do dest_dir="${arg}"; done + dest_dir="${dest_dir%/}" + if [ -n "${dest_dir}" ]; then + mkdir -p "${dest_dir}" + touch "${dest_dir}/file1" "${dest_dir}/file2" "${dest_dir}/file3" + fi + return 0 + ;; + rm) return 0 ;; + *) return 0 ;; + esac + } + export -f docker + + run bash "${REPO_ROOT}/scripts/host/backup_daaf.sh" + assert_success + refute_output --partial "file-count mismatch" +} diff --git a/tests/bash/backup_twin_parity.bats b/tests/bash/backup_twin_parity.bats index 8c9b4ca3..efbf3394 100755 --- a/tests/bash/backup_twin_parity.bats +++ b/tests/bash/backup_twin_parity.bats @@ -285,3 +285,88 @@ dquote_count() { tr -cd '"' | wc -c | tr -d '[:space:]'; } run diff "${TEST_DIR}/ref_r_sh.txt" "${TEST_DIR}/drift_r_ps1.txt" assert_failure } + +# =========================================================================== +# 5. HOST-SIDE LAUNCH-FAILURE REAP: both twins reap a captured staging CID +# =========================================================================== +# A `docker run -d` can fail AFTER creating the container (Created/Exited) while +# still printing its CID. Both twins must reap that captured CID on a launch +# failure rather than leaking a helper container. This is a host-side control-flow +# parity concern the container-program diffs above do not cover -- the historical +# gap where the .ps1 already reaped but the .sh did not. + +@test "twin-parity: both backup twins reap the captured MAIN staging CID on a launch failure" { + # .sh main launch: the `if ! STAGE_CID="$(docker run -d ...)"` error branch reaps + # the captured CID before `exit 1` (also used by the traps -- presence suffices). + run grep -F 'docker rm -f "${STAGE_CID:-}"' "${REPO_ROOT}/${BACKUP_SH}" + assert_success + # .ps1 main launch: the stage-start guard reaps $StageCid when the launch failed. + run grep -F 'docker rm -f $StageCid' "${REPO_ROOT}/${BACKUP_PS1}" + assert_success +} + +@test "twin-parity: both backup twins preserve + reap the CLAUDE staging CID (no blanking)" { + # .sh: the launch OUTCOME is captured in a separate flag so the CID stays intact + # for the end-of-block reap. The old `|| CLAUDE_STAGE_CID=""` blanked the CID + # before any cleanup could run and MUST be gone. + run grep -F 'CLAUDE_STAGE_START_OK=1' "${REPO_ROOT}/${BACKUP_SH}" + assert_success + run grep -F 'docker rm -f "${CLAUDE_STAGE_CID}"' "${REPO_ROOT}/${BACKUP_SH}" + assert_success + # The fixed launch uses an `if CLAUDE_STAGE_CID="$(docker run -d ...)"` guard. The + # buggy version was a `CLAUDE_STAGE_CID="$(...)" || CLAUDE_STAGE_CID=""` one-liner + # with NO leading `if ` -- so this positive assertion discriminates fix from bug. + # (A negative grep for the old `|| CLAUDE_STAGE_CID=""` idiom would false-match the + # explanatory comment that documents the historical pattern.) + run grep -F 'if CLAUDE_STAGE_CID="$(docker run -d' "${REPO_ROOT}/${BACKUP_SH}" + assert_success + # .ps1: the finally block reaps $ClaudeStageCid on every exit path. + run grep -F 'docker rm -f $ClaudeStageCid' "${REPO_ROOT}/${BACKUP_PS1}" + assert_success +} + +# =========================================================================== +# 6. RESTORE-PAIR LAUNCH-FAILURE REAP: both twins reap a captured create CID +# =========================================================================== +# Same host-side control-flow parity concern as section 5, applied to the restore +# pair's `docker create` helper-container guards. `docker create` is single-phase +# so a fail-with-CID is near-theoretical, but the .ps1 twin already reaps the CID in +# a try/finally at BOTH volumes, so the .sh must match: reap the captured CID on a +# create failure rather than leaking a helper container. These are source-anchor +# pins (fix-specific, comment-safe positive anchors), complementing the behavioral +# reap tests in restore_from_backup.bats. + +@test "twin-parity: both restore twins reap the captured MAIN helper CID on a create failure" { + # .sh main guard: the `${CID:-}` reap form must appear EXACTLY twice -- once in + # the INT/TERM trap and once in the `if ! CID="$(docker create ...)"` error branch + # (the fix). Pre-fix the trap alone gave a single occurrence, so a bare presence + # grep would not discriminate; the count of 2 is fix-specific. (The cp-failure and + # post-copy cleanups use the unguarded `${CID}` form, so they are not counted.) + local n + n="$(grep -cF 'docker rm -f "${CID:-}"' "${REPO_ROOT}/${RESTORE_SH}")" + [ "${n}" -eq 2 ] || { echo "FAIL: restore_from_backup.sh has ${n} \${CID:-} reap sites, expected 2 (INT/TERM trap + create-failure error branch). A missing error-branch reap leaks the helper container."; return 1; } + # .ps1 main guard: the try/finally reaps $Cid on every exit path (incl. a create + # that failed after emitting a CID). + run grep -F 'docker rm -f $Cid' "${REPO_ROOT}/${RESTORE_PS1}" + assert_success +} + +@test "twin-parity: both restore twins preserve + reap the CLAUDE helper CID (no blanking)" { + # .sh: the create OUTCOME is captured in a separate flag so the CID stays intact + # for the end-of-block reap. The old `|| CLAUDE_CID=""` blanked the CID before any + # cleanup could run and MUST be gone. + run grep -F 'CLAUDE_CREATE_OK=1' "${REPO_ROOT}/${RESTORE_SH}" + assert_success + run grep -F 'docker rm -f "${CLAUDE_CID}"' "${REPO_ROOT}/${RESTORE_SH}" + assert_success + # The fixed guard uses an `if CLAUDE_CID="$(docker create ...)"` form. The buggy + # version was a `CLAUDE_CID="$(...)" || CLAUDE_CID=""` one-liner with NO leading + # `if ` -- so this positive assertion discriminates fix from bug. (A negative grep + # for the old `|| CLAUDE_CID=""` idiom would false-match the explanatory comment + # that documents the historical pattern.) + run grep -F 'if CLAUDE_CID="$(docker create' "${REPO_ROOT}/${RESTORE_SH}" + assert_success + # .ps1: the finally block reaps $ClaudeCid on every exit path. + run grep -F 'docker rm -f $ClaudeCid' "${REPO_ROOT}/${RESTORE_PS1}" + assert_success +} diff --git a/tests/bash/daaf_lib.bats b/tests/bash/daaf_lib.bats index f2acb656..0aa7507f 100755 --- a/tests/bash/daaf_lib.bats +++ b/tests/bash/daaf_lib.bats @@ -471,6 +471,23 @@ teardown() { [ ! -f /tmp/daaf_should_not_exist ] } +@test "load_daaf_settings rejects a whitespace-padded key (column-0 strict, parity with PS)" { + # A leading-space " DAAF_PROJECT_NAME=..." line is not flush at column 0, so the + # column-0 `case` glob rejects it. This is the conformance anchor the PowerShell + # loaders were fixed to match (PS now extracts the key WITHOUT .Trim()), so a padded + # key resolves to nothing on BOTH platforms rather than silently working on one. + printf ' DAAF_PROJECT_NAME=padded\n' > "${TEST_DIR}/environment_settings.txt" + run bash -c ' + unset _DAAF_LIB_LOADED + source "'"${REPO_ROOT}"'/scripts/host/daaf_lib.sh" + unset DAAF_PROJECT_NAME + load_daaf_settings "./environment_settings.txt" + echo "NAME=${DAAF_PROJECT_NAME:-UNSET}" + ' + assert_success + assert_output --partial "NAME=UNSET" +} + # ========================================================================= # docker-compose.yml multi-instance interpolation defaults # ========================================================================= diff --git a/tests/bash/restore_from_backup.bats b/tests/bash/restore_from_backup.bats index 49e89236..f36ed2d3 100755 --- a/tests/bash/restore_from_backup.bats +++ b/tests/bash/restore_from_backup.bats @@ -836,3 +836,103 @@ teardown() { assert_output --partial "may predate" assert_output --partial "permission preservation" } + +# ========================================================================= +# --- Helper-container reap on `docker create` launch failure --- +# ========================================================================= +# `docker create` echoes a CID to stdout; if the command then exits nonzero the +# command substitution has ALREADY captured that CID. The two launch guards must +# reap it (docker rm -f ) so a launch failure does not leak a helper container. +# `docker create` is single-phase, so a fail-with-CID is near-theoretical here -- +# these tests pin insurance parity with backup_daaf.sh's fixed launch guards and +# with the .ps1 twin's finally-reap. The mocks log every docker call to a file so +# the reap can be asserted after `run` (a bats array does not survive the `run` +# subshell). Each test FAILS against the pre-fix restore code (main branch: no reap +# before exit 1; Claude branch: `|| CLAUDE_CID=""` blanked the CID before the +# end-of-block reap). + +@test "restore: main create launch failure reaps the captured helper container" { + mkdir -p "${TEST_DIR}/2026-01-01_daaf_backup" + touch "${TEST_DIR}/2026-01-01_daaf_backup/f1" + + export DAAF_NESTED=1 + + # Step 1 volume-clear (`docker run --rm ... rm -rf`) succeeds; the Site A + # `docker create` for the data volume prints a CID but exits nonzero. The error + # branch must run `docker rm -f mockcid0000` before `exit 1`. + docker() { + printf '%s\n' "$*" >> "${TEST_DIR}/docker_calls.log" + case "$1" in + info) return 0 ;; + volume) return 0 ;; + ps) return 0 ;; + create) echo "mockcid0000"; return 1 ;; + cp) return 0 ;; + rm) return 0 ;; + run) return 0 ;; + *) return 0 ;; + esac + } + export -f docker + + run bash -c 'printf "1\nRESTORE\n" | bash "'"${REPO_ROOT}"'/scripts/host/restore_from_backup.sh"' + assert_failure + assert_output --partial "Could not create the helper container" + # The captured CID must have been reaped in the error branch (fix-specific). + run grep -F 'rm -f mockcid0000' "${TEST_DIR}/docker_calls.log" + assert_success +} + +@test "restore: Claude create launch failure preserves and reaps the captured CID (WARNING, not fatal)" { + mkdir -p "${TEST_DIR}/2026-01-01_daaf_backup" + touch "${TEST_DIR}/2026-01-01_daaf_backup/f1" + # A non-empty Claude subfolder => HAS_CLAUDE_BACKUP=1, so the Claude restore path + # runs after the data-volume restore succeeds. + mkdir -p "${TEST_DIR}/2026-01-01_daaf_backup/.daaf-claude-config" + touch "${TEST_DIR}/2026-01-01_daaf_backup/.daaf-claude-config/creds" + + export DAAF_NESTED=1 + + # Data-volume restore succeeds (its `docker create` -> CID, exit 0). The + # Claude-volume `docker create` prints a CID but exits nonzero; the fix keeps that + # CID (no `|| CLAUDE_CID=""`) so the end-of-block `docker rm -f claudecid1111` + # reaps it. The two creates are distinguished by the "claude-config" volume + # substring. Both manifests probe ABSENT (test -f -> 1); verification returns a + # positive count so the data restore succeeds. Claude failure stays a WARNING and + # the script still completes (overall success). + docker() { + printf '%s\n' "$*" >> "${TEST_DIR}/docker_calls.log" + case "$1" in + info) return 0 ;; + volume) return 0 ;; + ps) return 0 ;; + create) + if printf '%s' "$*" | grep -q 'claude-config'; then + echo "claudecid1111"; return 1 + fi + echo "datacid0000"; return 0 + ;; + cp) return 0 ;; + rm) return 0 ;; + run) + shift + local args_str="$*" + if [[ "${args_str}" == *"test -f"* ]]; then + return 1 + elif [[ "${args_str}" == *"find /dest"* ]] && [[ "${args_str}" == *"wc -l"* ]]; then + echo "1"; return 0 + fi + return 0 + ;; + *) return 0 ;; + esac + } + export -f docker + + run bash -c 'printf "1\nRESTORE\n" | bash "'"${REPO_ROOT}"'/scripts/host/restore_from_backup.sh"' + assert_success + assert_output --partial "Failed to restore the Claude Code state volume" + # The captured Claude CID must be reaped -- proving it was NOT blanked before cleanup. + run grep -F 'rm -f claudecid1111' "${TEST_DIR}/docker_calls.log" + assert_success +} diff --git a/tests/powershell/backup_daaf.Tests.ps1 b/tests/powershell/backup_daaf.Tests.ps1 index c0972e8d..269ad89d 100644 --- a/tests/powershell/backup_daaf.Tests.ps1 +++ b/tests/powershell/backup_daaf.Tests.ps1 @@ -235,6 +235,50 @@ Describe "backup_daaf.ps1" { # No PS7-only null-coalescing operator $Content | Should -Not -Match '\?\?' } + + # ----------------------------------------------------------------- + # Regression: fail-closed staging-status parse (boolean-gated TryParse) + # ----------------------------------------------------------------- + # [int]::TryParse writes 0 into the [ref] on a parse FAILURE, so the old + # `$null = [int]::TryParse(..., [ref]$StageStatus)` form silently overwrote the + # fail-closed default with 0 whenever `docker wait` emitted a non-numeric last + # line (a daemon error merged via 2>&1) -- flipping the fatal gate from "abort" + # to "proceed to copy a partial staged tree." The fix parses into a temp var and + # adopts it ONLY when TryParse returns $true (mirrors restore_from_backup.ps1:279). + Context "Regression: fail-closed staging-status parse (boolean-gated TryParse)" { + It "gates the data-volume staging status on TryParse's boolean via a temp var" { + $Content | Should -Match '\[int\]::TryParse\("\$StageStatusRaw"\.Trim\(\), \[ref\]\$StageStatusParsed\)' + $Content | Should -Match '\$StageStatus = \$StageStatusParsed' + # The fail-open idiom for the status var must be gone. + $Content | Should -Not -Match '\$null = \[int\]::TryParse\("\$StageStatusRaw"' + } + It "gates the Claude-volume staging status the same fail-closed way" { + $Content | Should -Match '\[int\]::TryParse\("\$ClaudeStageStatusRaw"\.Trim\(\), \[ref\]\$ClaudeStageStatusParsed\)' + $Content | Should -Match '\$ClaudeStageStatus = \$ClaudeStageStatusParsed' + $Content | Should -Not -Match '\$null = \[int\]::TryParse\("\$ClaudeStageStatusRaw"' + } + It "keeps the explicit fail-closed default of 1 for both staging statuses" { + $Content | Should -Match '\$StageStatus = 1' + $Content | Should -Match '\$ClaudeStageStatus = 1' + } + } + + # ----------------------------------------------------------------- + # Regression: floored 1% tolerances (no banker's rounding) + # ----------------------------------------------------------------- + # `[long]($TotalFiles / 100)` round-half-to-even diverges from the bash twin's + # truncating integer division (350 -> 4 vs floor 3). The fix uses [math]::Floor + # to match bash (backup_daaf.sh:454/477) and restore_from_backup.ps1:621. + Context "Regression: floored 1% tolerances (no banker's rounding)" { + It "floors the file-count tolerance with [math]::Floor (not a bare [long] cast)" { + $Content | Should -Match '\[math\]::Max\(1, \[long\]\[math\]::Floor\(\$TotalFiles / 100\)\)' + $Content | Should -Not -Match '\[long\]\(\$TotalFiles / 100\)' + } + It "floors the size tolerance with [math]::Floor (not a bare [long] cast)" { + $Content | Should -Match '\[math\]::Max\(1, \[long\]\[math\]::Floor\(\$SourceSizeKB / 100\)\)' + $Content | Should -Not -Match '\[long\]\(\$SourceSizeKB / 100\)' + } + } } } @@ -301,6 +345,17 @@ Describe "backup_daaf.ps1 Import-DaafSettingsInline" { $env:DAAF_PROJECT_NAME | Should -Be "safe" $env:ANTHROPIC_API_KEY | Should -BeNullOrEmpty } + + It "rejects a whitespace-padded key (column-0 strict, parity with bash)" { + # A leading-space key like " DAAF_PROJECT_NAME=..." is not flush at column 0. + # The bash loaders' `case` glob rejects it; the PS loader must too (key extracted + # WITHOUT .Trim()), so a padded key resolves identically -- to nothing -- on both + # platforms rather than silently working on Windows only. Pre-fix, the trimmed key + # matched the whitelist and the value was adopted. + Set-Content -Path $script:SettingsFile -Value " DAAF_PROJECT_NAME=padded" + Import-DaafSettingsInline -SettingsFile $script:SettingsFile + $env:DAAF_PROJECT_NAME | Should -BeNullOrEmpty + } } # ============================================================================ @@ -398,6 +453,41 @@ Describe "backup_daaf.ps1 behavioral tests" { $Content | Should -Match '\$AvailableKB -lt \$RequiredKB' } } + + # ----------------------------------------------------------------- + # Executable semantics: fail-closed parse + floored tolerance + # ----------------------------------------------------------------- + # These run the FIXED snippets directly so a behavioral regression (not just a + # text edit) is caught. Each would produce the wrong value under the pre-fix form. + Context "Fail-closed parse + floored tolerance (executable semantics)" { + It "keeps the staging status fail-closed at 1 on non-numeric docker wait output" { + # Reproduce the exact parse snippet and feed it a merged daemon error line. + # The pre-fix `$null = [int]::TryParse(...)` overwrote the default with 0. + $StageStatusRaw = "Error response from daemon: no such container" + $StageStatus = 1 + $StageStatusParsed = 0 + if ($null -ne $StageStatusRaw -and [int]::TryParse("$StageStatusRaw".Trim(), [ref]$StageStatusParsed)) { + $StageStatus = $StageStatusParsed + } + $StageStatus | Should -Be 1 + } + It "adopts a genuine numeric zero status when docker wait succeeds cleanly" { + $StageStatusRaw = "0" + $StageStatus = 1 + $StageStatusParsed = 0 + if ($null -ne $StageStatusRaw -and [int]::TryParse("$StageStatusRaw".Trim(), [ref]$StageStatusParsed)) { + $StageStatus = $StageStatusParsed + } + $StageStatus | Should -Be 0 + } + It "floors an odd-hundred-half tolerance to 3, not banker's-rounded 4" { + # 350/100: [long](...) banker's-rounds to 4; [math]::Floor gives 3, matching + # the bash twin's truncating integer division. The second assertion documents + # (with executable evidence) exactly the divergence the fix removes. + ([long][math]::Floor(350 / 100)) | Should -Be 3 + ([long](350 / 100)) | Should -Be 4 + } + } } # ============================================================================ diff --git a/tests/powershell/rebuild_daaf.Tests.ps1 b/tests/powershell/rebuild_daaf.Tests.ps1 index d9bf4050..a378e2e2 100644 --- a/tests/powershell/rebuild_daaf.Tests.ps1 +++ b/tests/powershell/rebuild_daaf.Tests.ps1 @@ -93,6 +93,16 @@ Describe "rebuild_daaf.ps1" { It "waits for container readiness after rebuild" { $Content | Should -Match 'Waiting for container to be ready' } + + It "extracts the settings key column-0 strict (no .Trim(), rejects padded keys like bash)" { + # Import-DaafSettingsInline must extract the key WITHOUT .Trim() so a + # whitespace-padded " DAAF_PROJECT_NAME=..." line falls through as + # unrecognized -- matching the bash loaders' column-0 `case` glob. The + # pre-fix `.Substring(0, $eq).Trim()` accepted padded keys, diverging from + # bash across all ten PS loader copies. + $Content | Should -Match '\$key = \$line\.Substring\(0, \$eq\)' + $Content | Should -Not -Match '\$key = \$line\.Substring\(0, \$eq\)\.Trim\(\)' + } } } @@ -246,9 +256,19 @@ Describe "rebuild_daaf.ps1 error paths" { $Content | Should -Match 'BUILDKIT_STEP_LOG_MAX_SPEED=10485760' } - It "selects the builder via BUILDX_BUILDER and clears it after the build" { + It "selects the builder via BUILDX_BUILDER and RESTORES the prior value after the build (non-destructive)" { + # Regression: the pre-fix code cleared the var unconditionally with + # `if ($UseDiagBuilder) { $env:BUILDX_BUILDER = $null }`, which REMOVES the + # variable and clobbers a user's own pre-exported BUILDX_BUILDER. The fix + # saves the prior value first and restores it (removing only when the user had + # none). Assert the save + conditional restore, and that the destructive + # single-line unconditional clear is gone. (`$env:BUILDX_BUILDER = $null` still + # appears in the else-branch and in the hazard comment, so we negative-match + # the specific one-line `if ($UseDiagBuilder) { ... = $null }` form instead.) $Content | Should -Match '\$env:BUILDX_BUILDER = "daaf-diag-builder"' - $Content | Should -Match '\$env:BUILDX_BUILDER = \$null' + $Content | Should -Match '\$priorBuildxBuilder = \$env:BUILDX_BUILDER' + $Content | Should -Match 'if \(\$null -ne \$priorBuildxBuilder\) \{ \$env:BUILDX_BUILDER = \$priorBuildxBuilder \}' + $Content | Should -Not -Match 'if \(\$UseDiagBuilder\) \{ \$env:BUILDX_BUILDER = \$null \}' } It "falls open to the default builder when the builder cannot be created" { diff --git a/tests/powershell/run_daaf.Tests.ps1 b/tests/powershell/run_daaf.Tests.ps1 index 1b1caabf..816081b2 100644 --- a/tests/powershell/run_daaf.Tests.ps1 +++ b/tests/powershell/run_daaf.Tests.ps1 @@ -74,6 +74,16 @@ Describe "run_daaf.ps1" { It "starts container if not running" { $Content | Should -Match 'Starting DAAF container' } + + It "extracts the settings key column-0 strict (no .Trim(), rejects padded keys like bash)" { + # Import-DaafSettingsInline must extract the key WITHOUT .Trim() so a + # whitespace-padded " DAAF_PROJECT_NAME=..." line falls through as + # unrecognized -- matching the bash loaders' column-0 `case` glob. The + # pre-fix `.Substring(0, $eq).Trim()` accepted padded keys, diverging from + # bash across the PS loader copies. + $Content | Should -Match '\$key = \$line\.Substring\(0, \$eq\)' + $Content | Should -Not -Match '\$key = \$line\.Substring\(0, \$eq\)\.Trim\(\)' + } } } diff --git a/tests/powershell/run_vscode.Tests.ps1 b/tests/powershell/run_vscode.Tests.ps1 index 1903362e..977450b7 100644 --- a/tests/powershell/run_vscode.Tests.ps1 +++ b/tests/powershell/run_vscode.Tests.ps1 @@ -98,6 +98,16 @@ Describe "run_vscode.ps1" { It "reports container already running" { $Content | Should -Match 'DAAF container is running' } + + It "extracts the settings key column-0 strict (no .Trim(), rejects padded keys like bash)" { + # Import-DaafSettingsInline must extract the key WITHOUT .Trim() so a + # whitespace-padded " DAAF_PROJECT_NAME=..." line falls through as + # unrecognized -- matching the bash loaders' column-0 `case` glob. The + # pre-fix `.Substring(0, $eq).Trim()` accepted padded keys, diverging from + # bash across the PS loader copies. + $Content | Should -Match '\$key = \$line\.Substring\(0, \$eq\)' + $Content | Should -Not -Match '\$key = \$line\.Substring\(0, \$eq\)\.Trim\(\)' + } } } diff --git a/tests/powershell/update_daaf.Tests.ps1 b/tests/powershell/update_daaf.Tests.ps1 index 43affbba..bbfae3ac 100644 --- a/tests/powershell/update_daaf.Tests.ps1 +++ b/tests/powershell/update_daaf.Tests.ps1 @@ -139,6 +139,16 @@ Describe "update_daaf.ps1" { It "detects ahead/behind state" { $Content | Should -Match 'rev-list --count' } + + It "extracts the settings key column-0 strict (no .Trim(), rejects padded keys like bash)" { + # Import-DaafSettingsInline must extract the key WITHOUT .Trim() so a + # whitespace-padded " DAAF_PROJECT_NAME=..." line falls through as + # unrecognized -- matching the bash loaders' column-0 `case` glob. The + # pre-fix `.Substring(0, $eq).Trim()` accepted padded keys, diverging from + # bash across the PS loader copies. + $Content | Should -Match '\$key = \$line\.Substring\(0, \$eq\)' + $Content | Should -Not -Match '\$key = \$line\.Substring\(0, \$eq\)\.Trim\(\)' + } } } @@ -828,6 +838,46 @@ Describe "update_daaf.ps1 behavioral tests" { } } + # ----------------------------------------------------------------- + # Import-DaafSettingsInline (column-0 key strictness) + # ----------------------------------------------------------------- + # The settings loader must extract the key WITHOUT .Trim() so a whitespace- + # padded " DAAF_PROJECT_NAME=..." line is NOT flush at column 0 and falls + # through as unrecognized -- matching the bash loaders' column-0 `case` glob. + # A flush (column-0) key is still adopted. The function is called with an + # explicit -SettingsFile so no CWD juggling is needed; the probed env vars are + # cleared before and restored after so a real container env is not clobbered. + Context "Import-DaafSettingsInline column-0 key strictness" { + BeforeEach { + $script:IdsDir = New-Item -ItemType Directory -Path (Join-Path ([System.IO.Path]::GetTempPath()) "daaf-ids-$(Get-Random)") + } + AfterEach { + Remove-Item -Recurse -Force $script:IdsDir -ErrorAction SilentlyContinue + } + + It "rejects a whitespace-padded key and adopts a flush key (like bash column-0)" { + $settings = Join-Path $script:IdsDir "environment_settings.txt" + Set-Content -Path $settings -Value @( + " DAAF_PORT_MARIMO=9999", + "DAAF_PROJECT_NAME=flushvalue" + ) + $origName = $env:DAAF_PROJECT_NAME + $origMarimo = $env:DAAF_PORT_MARIMO + Remove-Item Env:DAAF_PROJECT_NAME -ErrorAction SilentlyContinue + Remove-Item Env:DAAF_PORT_MARIMO -ErrorAction SilentlyContinue + try { + Import-DaafSettingsInline -SettingsFile $settings + # Flush key adopted; padded key (" DAAF_PORT_MARIMO") off-whitelist -> unset. + $env:DAAF_PROJECT_NAME | Should -Be "flushvalue" + $env:DAAF_PORT_MARIMO | Should -BeNullOrEmpty + } + finally { + if ($null -ne $origName) { $env:DAAF_PROJECT_NAME = $origName } else { Remove-Item Env:DAAF_PROJECT_NAME -ErrorAction SilentlyContinue } + if ($null -ne $origMarimo) { $env:DAAF_PORT_MARIMO = $origMarimo } else { Remove-Item Env:DAAF_PORT_MARIMO -ErrorAction SilentlyContinue } + } + } + } + # ----------------------------------------------------------------- # Resolve-Conflict # ----------------------------------------------------------------- diff --git a/tests/powershell/view_logs.Tests.ps1 b/tests/powershell/view_logs.Tests.ps1 index f1ebcdc2..af425519 100644 --- a/tests/powershell/view_logs.Tests.ps1 +++ b/tests/powershell/view_logs.Tests.ps1 @@ -93,6 +93,16 @@ Describe "view_logs.ps1" { $Content | Should -Match 'DAAF_NESTED' $Content | Should -Match '\$SkipMenu' } + + It "extracts the settings key column-0 strict (no .Trim(), rejects padded keys like bash)" { + # Import-DaafSettingsInline must extract the key WITHOUT .Trim() so a + # whitespace-padded " DAAF_PROJECT_NAME=..." line falls through as + # unrecognized -- matching the bash loaders' column-0 `case` glob. The + # pre-fix `.Substring(0, $eq).Trim()` accepted padded keys, diverging from + # bash across the PS loader copies. + $Content | Should -Match '\$key = \$line\.Substring\(0, \$eq\)' + $Content | Should -Not -Match '\$key = \$line\.Substring\(0, \$eq\)\.Trim\(\)' + } } } diff --git a/tests/powershell/view_notebooks.Tests.ps1 b/tests/powershell/view_notebooks.Tests.ps1 index f6040828..5128dd3a 100644 --- a/tests/powershell/view_notebooks.Tests.ps1 +++ b/tests/powershell/view_notebooks.Tests.ps1 @@ -98,6 +98,16 @@ Describe "view_notebooks.ps1" { It "reports container already running" { $Content | Should -Match 'DAAF container is running' } + + It "extracts the settings key column-0 strict (no .Trim(), rejects padded keys like bash)" { + # Import-DaafSettingsInline must extract the key WITHOUT .Trim() so a + # whitespace-padded " DAAF_PROJECT_NAME=..." line falls through as + # unrecognized -- matching the bash loaders' column-0 `case` glob. The + # pre-fix `.Substring(0, $eq).Trim()` accepted padded keys, diverging from + # bash across the PS loader copies. + $Content | Should -Match '\$key = \$line\.Substring\(0, \$eq\)' + $Content | Should -Not -Match '\$key = \$line\.Substring\(0, \$eq\)\.Trim\(\)' + } } } diff --git a/tests/powershell/view_quarto.Tests.ps1 b/tests/powershell/view_quarto.Tests.ps1 index d872e9f7..f5e98476 100644 --- a/tests/powershell/view_quarto.Tests.ps1 +++ b/tests/powershell/view_quarto.Tests.ps1 @@ -107,6 +107,16 @@ Describe "view_quarto.ps1" { It "reports container already running" { $Content | Should -Match 'DAAF container is running' } + + It "extracts the settings key column-0 strict (no .Trim(), rejects padded keys like bash)" { + # Import-DaafSettingsInline must extract the key WITHOUT .Trim() so a + # whitespace-padded " DAAF_PROJECT_NAME=..." line falls through as + # unrecognized -- matching the bash loaders' column-0 `case` glob. The + # pre-fix `.Substring(0, $eq).Trim()` accepted padded keys, diverging from + # bash across the PS loader copies. + $Content | Should -Match '\$key = \$line\.Substring\(0, \$eq\)' + $Content | Should -Not -Match '\$key = \$line\.Substring\(0, \$eq\)\.Trim\(\)' + } } } From d67ee46e201846f96918d7b7bd6679be5247a94f Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Sun, 19 Jul 2026 13:08:48 +0000 Subject: [PATCH 05/66] fix(shim): release v1.2.13 provider hardening Add multimodal request translation, privacy-safe transport diagnostics, strict lifecycle readiness and atomic restart, complete package accounting, and restore Claude Code system-role compatibility. Co-Authored-By: Claude --- CONTRIBUTING.md | 15 +- Dockerfile | 12 + benchmarks/harness/route_provenance.py | 3 + benchmarks/tests/test_chatgpt_route.py | 2 + scripts/deploy_smoke/smoke_probes.py | 62 +- .../provider_shim/anthropic_openai_shim.py | 573 +++++++-- scripts/provider_shim/start_shim.sh | 1144 ++++++++++++++--- scripts/smoke_tests/smoke_imports.py | 186 ++- scripts/smoke_tests/smoke_imports_a.py | 258 ++++ tests/bash/start_shim.bats | 744 +++++++++++ tests/provider_shim/_loopback_harness.py | 136 +- .../test_historical_regressions.py | 622 +++++++++ .../test_reasoning_formatting.py | 6 +- .../provider_shim/test_request_translation.py | 629 +++++++++ tests/provider_shim/test_stream_hardening.py | 129 +- tests/python/test_deploy_smoke.py | 168 +++ .../01_installation_and_quickstart.md | 23 +- user_reference/07_faq_technical.md | 38 +- 18 files changed, 4366 insertions(+), 384 deletions(-) create mode 100644 scripts/smoke_tests/smoke_imports_a.py create mode 100644 tests/bash/start_shim.bats create mode 100644 tests/provider_shim/test_historical_regressions.py create mode 100644 tests/provider_shim/test_request_translation.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 282fba1e..a7ed7f16 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -427,12 +427,21 @@ pwsh -NoProfile -Command "Get-ChildItem ./scripts/host/*.ps1 | ForEach-Object { `DAAF_DEV` is an opt-in build flag: when it is unset or `0` (the default), none of this tooling is installed and the image is identical to a standard build. See `user_reference/01_installation_and_quickstart.md` ("Building with the developer test toolchain") for details. -**Working on R support?** The R execution environment (R, the R package set, and Quarto) is part of the standard DAAF image — there is nothing to enable. The R smoke tests in `scripts/smoke_tests/` exercise each R library skill and run in any container; a Python import-smoke (`smoke_imports.py`) additionally verifies every pinned Python analysis package imports. Both run via `run_all_smoke_tests.sh`. (The per-skill R smokes came first because the newly added R package set needed install verification the mature Python stack had long since earned; the Python import-smoke rounds out the suite.) See `user_reference/01_installation_and_quickstart.md` ("R support (included)") for what the image ships with. +**Working on R support?** The R execution environment (R, the R package set, and Quarto) is part of the standard DAAF image — there is nothing to enable. The R smoke tests in `scripts/smoke_tests/` exercise each R library skill and run in any container; the Python import smoke (`smoke_imports.py`) covers explicit Docker pins and asserts the exact provider-shim runtime versions `httpx==0.28.1` and `uvicorn==0.51.0`. Both run via `run_all_smoke_tests.sh`. (The per-skill R smokes came first because the newly added R package set needed install verification the mature Python stack had long since earned; the Python import smoke rounds out the suite.) See `user_reference/01_installation_and_quickstart.md` ("R support (included)") for what the image ships with. ### For Deployment or Provider-Configuration Changes If your change touches how DAAF talks to a model provider -- a new or changed API key, a model remap (`ANTHROPIC_DEFAULT_*`), a provider-shim change, a base-URL switch -- or you just want to confirm a rebuilt install still works end-to-end, use the **deployment smoke suite** (`scripts/deploy_smoke/`). Unlike the R/Python library smokes above (which verify packages load) or DAAFBench (which scores model *behavior*), this suite verifies the *deployment itself* functions in whatever provider route it is configured for. It auto-detects the active route (anthropic-subscription, OpenRouter, or the chatgpt-subscription/openai-api shim routes) and runs tiered probes. It requires the `DAAF_DEV=1` image. +**Provider-shim changes have a stricter release gate.** A reviewable shim PR should show all of the following, with commands and observed results rather than a remembered pass claim: + +1. **Production-path deterministic coverage.** Exercise the real translator/ASGI/subprocess paths in `tests/provider_shim/`, not a duplicate implementation inside a test. Run the provider-shim suite, the lifecycle suite at `tests/bash/start_shim.bats`, the DAAFBench route/provenance tests under `benchmarks/tests/`, and the deploy-smoke harness self-tests at `tests/python/test_deploy_smoke.py`. +2. **Lifecycle and privacy invariants.** Preserve strict readiness (exact service identity, `ok` status, source-derived version, normalized backend mode), the nonfatal machine-readable `--auto` boot contract, serialized `--restart`/`--start`/`--stop` ownership, exactly-once restart begin/result records with distinct stop/launch/readiness failures, serialized single-write logging, private bounded retention, safe PID/PGID identity checks, and exactly-once terminal/cleanup accounting. Diagnostics may record local/client/provider request IDs where available, transport class/phase, bounded timings, and retry attribution—but never prompts/text, tool inputs/arguments, image data or URLs/query strings, file IDs/names, credentials/cookies, raw SSE, complete bodies, or free-form backend prose. +3. **Image-contract evidence.** User-message and tool-result images must preserve source ordering and fail locally on unsupported or malformed content rather than silently dropping it. The public OpenAI Responses schema is the documented basis. The private ChatGPT/Codex wire is not an official image guarantee: fixtures for it must come from observed live evidence, never a guessed documentation shape, and the capture process must not log payloads. Run a privacy-safe live image probe on that lane before claiming ChatGPT image compatibility. +4. **Image and deployment validation.** If Docker dependencies change, run `scripts/smoke_tests/run_all_smoke_tests.sh`; its Python gate verifies 50 import targets plus two metadata-only distributions — 52 explicit pinned distributions in total. It preserves exact-version checks for `httpx==0.28.1` and `uvicorn==0.51.0` and uses metadata-only presence/exact-version checks for `svy-rs==0.10.0` and `svy-io==0.1.1` rather than inventing import names. After deploying the changed shim, restart it with the single atomic command `bash /daaf/scripts/provider_shim/start_shim.sh --restart`, confirm the source version through strict `/health`, then run deployment-smoke Tier 0 and Tier 1 against the active route. When the current Claude Code session itself routes through the shim, never schedule `--stop` and `--start` as separate assistant turns—the first turn can remove the transport needed for the second. Deterministic success is necessary, but it does not substitute for this deployed round-trip. + +For tolerated private-wire differences, aggregate bounded diagnostics rather than flooding logs: the known `arguments.done`-without-`name` case records the first occurrence and periodic count summaries. Hard or conflicting protocol errors must remain visible and fail explicitly. + Start with the free, no-API preflight, then add live tiers as needed: ```bash @@ -476,10 +485,10 @@ DAAF's components are wired together at many small registration points -- an age | A **reference file** in `agent_reference/` | § 4 (R1-R6 / RM1-RM4) -- CLAUDE.md table plus every "When to Read" trigger | Link and cross-reference checks; render the Markdown | | A **hook** (`.claude/hooks/`) | § 5 (H1-H5) -- note hooks are human-deployed (deny-protected) | The safety regression batteries `bash scripts/test_safety_hooks.sh` and `bash scripts/test_enforce_single_command.sh`, plus both allow and block scenarios | | A **host-facing script** (`scripts/host/`) | § 6 (HS1-HS11) -- install lists, updater filter, CI smoke lists, `.sh`/`.ps1` parity | `bats tests/bash/`, Pester, `shellcheck`, `PSScriptAnalyzer`, and `bash tests/lint/check-daaf-conventions.sh` (all run in CI) | -| The **provider shim** (`scripts/provider_shim/`) | § 7 (P1-P11 / PM1-PM6) -- `/health` schema, `SHIM_*` env-var lockstep, lane-gate triplication, `SHIM_VERSION` bump | The shim and benchmark suites, plus a deploy-smoke run: Tier 0 `/health` preflight and a Tier 1 live round-trip; confirm the new `SHIM_VERSION` via `GET /health` | +| The **provider shim** (`scripts/provider_shim/`) | § 7 (P1-P11 / PM1-PM6) -- `/health` schema, `SHIM_*` env-var lockstep, lane-gate triplication, `SHIM_VERSION` bump | Production-path shim tests, lifecycle Bats, route/provenance and deploy-smoke self-tests; package smoke when Docker pins change; after deployment, Tier 0 + Tier 1 and strict `/health` version confirmation; privacy-safe live image validation before a ChatGPT compatibility claim | | A **deployment / provider configuration** (API key, model remap, base URL) | Route/env coherence | deploy-smoke Tier 0 (free preflight), adding Tier 1/2 as needed (Tier 2 ≈ $0.60-3.00) | | **Documentation only** | -- | Read it aloud, check links and cross-references, render the Markdown | -| **Any of the above** | § 8 cross-cutting (CC1-CC7) | Sweep for stale count words, broken cross-references, and naming drift | +| **Any of the above** | § 8 cross-cutting (CC1-CC8) | Sweep for stale count words, broken cross-references, naming drift, and stale examples/templates | Two zero-cost batteries are worth knowing regardless of what you touched: the deployment smoke suite's **Tier D** deterministic battery (bats, Pester, lint, and safety-hook tests) and the conventions linter both run without any provider cost, and both run in CI on every push that touches a script. diff --git a/Dockerfile b/Dockerfile index ce159d8a..cc565f35 100644 --- a/Dockerfile +++ b/Dockerfile @@ -789,6 +789,18 @@ ARG CLAUDE_CODE_VERSION=2.1.202 RUN curl -fsSL https://claude.ai/install.sh | bash -s ${CLAUDE_CODE_VERSION} ENV PATH="/home/appuser/.local/bin:${PATH}" +# Provider-shim direct runtime dependencies. These were already installed +# transitively (httpx via svy; uvicorn via marimo), but the persistent shim imports +# them directly, so its contract must not depend on unrelated packages retaining +# those dependencies. Keep this framework-owned pin layer late: neither package is +# needed to build the Python/R stacks above, and placing it here preserves Docker's +# expensive-layer cache when only the shim runtime pins change. +USER root +RUN uv pip install --system \ + httpx==0.28.1 \ + uvicorn==0.51.0 +USER appuser + # ============================================ # USER ADDITIONS — add your own packages and tools here # ============================================ diff --git a/benchmarks/harness/route_provenance.py b/benchmarks/harness/route_provenance.py index 22697b33..af1945ba 100644 --- a/benchmarks/harness/route_provenance.py +++ b/benchmarks/harness/route_provenance.py @@ -24,6 +24,7 @@ DEFAULT_SHIM_PORT = 4141 CHATGPT_PROVIDER = "chatgpt-subscription" EXPECTED_CODEX_BACKEND = "https://chatgpt.com/backend-api/codex" +EXPECTED_SHIM_SERVICE = "daaf-anthropic-openai-shim" LOCAL_AUTH_PLACEHOLDER = "daaf-shim-local" # Deliberately explicit rather than derived from the dataclass: adding an # internal field must not silently widen the serialization boundary. @@ -158,6 +159,8 @@ def _read_health_response(response) -> dict: def _validate_health_payload(health: Mapping[str, object], endpoint: str) -> RouteProvenance: problems = [] + if health.get("service") != EXPECTED_SHIM_SERVICE: + problems.append(f"service must equal '{EXPECTED_SHIM_SERVICE}'") if health.get("status") != "ok": problems.append("status must equal 'ok'") if health.get("backend_mode") != "chatgpt": diff --git a/benchmarks/tests/test_chatgpt_route.py b/benchmarks/tests/test_chatgpt_route.py index 8b922e2c..51bfbdf6 100644 --- a/benchmarks/tests/test_chatgpt_route.py +++ b/benchmarks/tests/test_chatgpt_route.py @@ -88,6 +88,7 @@ def coherent_env(**overrides): def healthy_payload(**overrides): payload = { + "service": "daaf-anthropic-openai-shim", "status": "ok", "backend": "https://chatgpt.com/backend-api/codex", "backend_mode": "chatgpt", @@ -264,6 +265,7 @@ def test_route_coherence_failures(self): def test_health_contract_failures(self): bad_payloads = [ + healthy_payload(service="unrelated-service"), healthy_payload(status="degraded"), healthy_payload(backend_mode="openai"), healthy_payload(sanitize_tools=False), diff --git a/scripts/deploy_smoke/smoke_probes.py b/scripts/deploy_smoke/smoke_probes.py index 77519c53..c8b4a840 100644 --- a/scripts/deploy_smoke/smoke_probes.py +++ b/scripts/deploy_smoke/smoke_probes.py @@ -65,6 +65,8 @@ # interpolated into f-strings, wrapped in Path(...), and passed as cwd= throughout. BASE_DIR = str(Path(__file__).resolve().parents[2]) SHIM_HEALTH_URL = "http://127.0.0.1:4141/health" +EXPECTED_SHIM_SERVICE = "daaf-anthropic-openai-shim" +_SAFE_SHIM_VERSION_RE = re.compile(r"[A-Za-z0-9._+-]{1,64}") _GLM52_STATIC_ID = re.compile(r"z-ai/glm-5\.2(?:-[0-9]{8})?") @@ -486,8 +488,8 @@ def probe_statuslines(base_dir: str = BASE_DIR) -> ProbeResult: def probe_shim_health(route_info: RouteInfo) -> ProbeResult: - """GET the shim /health endpoint (shim routes only) and verify backend_mode - matches the detected route; report sanitize_tools, codex_home_present, version.""" + """GET the shim /health endpoint (shim routes only) and fail closed on its + provenance schema before reporting bounded configuration evidence.""" r = ProbeResult(probe_id="T0.8", name="Provider shim /health", tier="0") if route_info.detected_route not in SHIM_ROUTES: r.verdict = Verdict.SKIP @@ -499,35 +501,67 @@ def probe_shim_health(route_info: RouteInfo) -> ProbeResult: with urlopen(SHIM_HEALTH_URL, timeout=10) as resp: body = resp.read().decode("utf-8", "replace") health = json.loads(body) - # Scrub any secret env value that the /health JSON might echo verbatim - # before it enters evidence (defense-in-depth; the placeholder keeps the - # blob parseable for the evidence/ snapshot). - r.add_evidence(f"GET {SHIM_HEALTH_URL}", - output=scrub_secret_values(json.dumps(health, indent=2)[:1500])) - except (URLError, socket.timeout, json.JSONDecodeError, OSError) as e: + if not isinstance(health, dict): + raise ValueError("shim /health JSON must be an object") + except (URLError, socket.timeout, json.JSONDecodeError, OSError, ValueError) as e: r.verdict = Verdict.FAIL - r.detail = f"shim /health unreachable/invalid: {type(e).__name__}: {e}" - r.add_evidence(f"GET {SHIM_HEALTH_URL}", note=str(e)) + r.detail = f"shim /health unreachable/invalid: {type(e).__name__}: {str(e)[:200]}" + r.add_evidence(f"GET {SHIM_HEALTH_URL}", note=str(e)[:200]) return r + version = health.get("version") + version_is_safe = ( + isinstance(version, str) + and _SAFE_SHIM_VERSION_RE.fullmatch(version) is not None + ) + version_marker = version if version_is_safe else "" + + codex_home_present = health.get("codex_home_present") + codex_home_marker = ( + codex_home_present if isinstance(codex_home_present, bool) else "" + ) + + # Preserve the response shape for audit evidence, but never reflect an unsafe + # version or non-boolean auth-presence value. The local markers are bounded and + # contain no endpoint-controlled text; secret-value scrubbing remains a second + # defense before the snapshot enters the report. + health_evidence = dict(health) + if not version_is_safe: + health_evidence["version"] = version_marker + if "codex_home_present" in health and not isinstance(codex_home_present, bool): + health_evidence["codex_home_present"] = codex_home_marker + r.add_evidence( + f"GET {SHIM_HEALTH_URL}", + output=scrub_secret_values(json.dumps(health_evidence, indent=2)[:1500]), + ) + expected_mode = "chatgpt" if route_info.detected_route == ROUTE_CHATGPT else "openai" actual_mode = health.get("backend_mode") problems = [] + if health.get("service") != EXPECTED_SHIM_SERVICE: + problems.append(f"service must equal '{EXPECTED_SHIM_SERVICE}'") + if health.get("status") != "ok": + problems.append("status must equal 'ok'") if actual_mode != expected_mode: problems.append(f"backend_mode='{actual_mode}' but route expects '{expected_mode}'") - if route_info.detected_route == ROUTE_CHATGPT and not health.get("codex_home_present"): - problems.append("codex_home_present=false (auth.json missing/unreadable) for chatgpt route") + if not version_is_safe: + problems.append("version must match [A-Za-z0-9._+-]{1,64}") + if route_info.detected_route == ROUTE_CHATGPT and codex_home_present is not True: + problems.append( + "codex_home_present must be boolean true " + "(auth.json missing/unreadable) for chatgpt route" + ) r.add_evidence("", note=( f"backend_mode={actual_mode} sanitize_tools={health.get('sanitize_tools')} " - f"codex_home_present={health.get('codex_home_present')} version={health.get('version')}" + f"codex_home_present={codex_home_marker} version={version_marker}" )) if problems: r.verdict = Verdict.FAIL r.detail = "; ".join(problems) else: r.verdict = Verdict.PASS - r.detail = f"shim healthy: backend_mode={actual_mode}, version={health.get('version')}." + r.detail = f"shim healthy: backend_mode={actual_mode}, version={version_marker}." return r diff --git a/scripts/provider_shim/anthropic_openai_shim.py b/scripts/provider_shim/anthropic_openai_shim.py index 5a784c73..86e76c67 100644 --- a/scripts/provider_shim/anthropic_openai_shim.py +++ b/scripts/provider_shim/anthropic_openai_shim.py @@ -63,6 +63,29 @@ # * GET /health endpoint for the manager's idempotency/status checks. # # Changelog: +# v1.2.13 (2026-07-19): System-role message admission (live-outage fix). Current +# Claude Code appends a role:"system" message inside `messages`; the v1.2.12 +# request-validation role check (user/assistant only) and the translator's +# matching raise rejected every real conversation turn with the static 400 +# "invalid request structure" while small side requests (no system-role +# message) still passed. Validation now admits "system" and the translator +# folds system-role messages to user-role input — the exact pre-v1.2.12 +# mapping proven against the live Codex backend. Captured-payload repro and +# bisection: scripts/scratch/replay_captured_request.py. All other v1.2.12 +# hardening, translation, and lifecycle behavior unchanged. +# SHIM_VERSION -> 1.2.13. +# v1.2.12 (2026-07-18): Content-blind image fidelity and bounded translator +# diagnostics. User and tool-result image blocks now map to Responses +# input_image parts for the shared OpenAI/ChatGPT payload shape, with strict +# local validation and explicit 400s for unsupported sources, assistant images, +# and unknown history blocks. Transport lifecycle records now attribute bounded +# exception classes/phases, sent and returned request IDs, and retry sources. +# The live Codex arguments.done missing-name divergence is process-aggregated +# (first occurrence plus periodic summaries) instead of warning per event. +# /health now carries an exact service identity so the lifecycle manager can +# reject an unrelated HTTP 200 before declaring readiness. Retry counts, +# backoff policy, response translation, reasoning continuity, and cancellation/ +# cleanup ownership are unchanged. SHIM_VERSION -> 1.2.12. # v1.2.11 (2026-07-17): Correlated request-lifecycle observability and # structured backend-error normalization. Every /v1/messages request now owns # an internally generated 32-hex correlation ID in a ContextVar before body @@ -519,10 +542,9 @@ # (default; still overridable via SHIM_BACKEND_BASE_URL). # Auth = Bearer of the OAuth access_token read from # $CODEX_HOME/auth.json (NOT an API key); the outbound -# request emits ONLY the 3-header floor (authorization, -# content-type: application/json, accept: -# text/event-stream) — the backend gates on nothing -# else (notes/08). REQUIRES CODEX_HOME set and a +# request emits the validated auth/content/accept floor +# plus the privacy-safe X-Client-Request-Id used for +# transport correlation. REQUIRES CODEX_HOME set and a # readable auth.json; if either is missing the shim # fails fast with the re-login message rather than # inventing a default path. Request translation, tools, @@ -617,6 +639,7 @@ import random import asyncio import logging +import threading import urllib.parse from collections import OrderedDict from contextvars import ContextVar @@ -625,7 +648,8 @@ import uvicorn # --- Config --- -SHIM_VERSION = "1.2.11" +SHIM_VERSION = "1.2.13" +SHIM_SERVICE_ID = "daaf-anthropic-openai-shim" SHIM_PORT = int(os.environ.get("SHIM_PORT", "4141")) @@ -812,6 +836,10 @@ def _resolve_startup_verbosity(): _UPSTREAM_REQUEST_ID_MAXLEN = 200 _CLIENT_ERROR_MESSAGE_MAXLEN = 200 _MACHINE_FIELD_VALUE_MAXLEN = 1000 +_WIRE_DIVERGENCE_SUMMARY_INTERVAL = 100 +_WIRE_DIVERGENCE_COUNTS = {} +_WIRE_DIVERGENCE_LAST_EMITTED = {} +_WIRE_DIVERGENCE_LOCK = threading.Lock() class _RequestLifecycleState: @@ -833,6 +861,10 @@ def __init__(self): self.reasoning_cache_misses = 0 self.attempts = 0 self.retries = 0 + self.last_retry_reason = "-" + self.last_retry_source = "-" + self.client_request_id = self.req_id + self.transport_exception = "-" self.upstream_request_id = "-" self.upstream_request_id_header = "-" self.upstream_http_version = "unknown" @@ -971,13 +1003,33 @@ def _record_attempt(transport): _lifecycle_event("upstream_attempt", attempt=state.attempts, transport=transport) -def _record_retry(reason): +def _record_retry(reason, source="shim_policy"): state = _request_state() if state is None: return state.phase = "upstream_retry" state.retries += 1 - _lifecycle_event("upstream_retry", retry=state.retries, reason=reason) + state.last_retry_reason = _scrub_metadata(reason, 100) + state.last_retry_source = _scrub_metadata(source, 100) + _lifecycle_event( + "upstream_retry", retry=state.retries, reason=state.last_retry_reason, + source=state.last_retry_source, + ) + + +def _record_transport_failure(error, phase, retryable): + state = _request_state() + if state is None: + return + exception_class = _scrub_metadata(type(error).__name__, 100) + state.transport_exception = exception_class + if not retryable and state.failure_phase == "-": + state.failure_phase = phase + _lifecycle_event( + "transport_failure", exception_class=exception_class, + failure_phase=phase, attempt=state.attempts, + retryable="y" if retryable else "n", + ) def _record_disconnect(phase): @@ -1070,6 +1122,11 @@ def _log_terminal_once(): state.effort_value, state.effort_source), reasoning_cache_miss=state.reasoning_cache_misses, attempts=state.attempts, retries=state.retries, + retry_reason=state.last_retry_reason, + retry_source=state.last_retry_source, + transport_exception=state.transport_exception, + client_req_id=state.client_request_id, + upstream_req_id=state.upstream_request_id, failure_phase=state.failure_phase, terminal_frame_send=state.terminal_frame_send, body_close_send=state.body_close_send, @@ -1135,6 +1192,10 @@ class _ProtocolError(Exception): """Controlled upstream Responses protocol/schema violation.""" +class _InvalidRequestError(Exception): + """Sanitized local Anthropic request rejection before any backend call.""" + + class _ClientDisconnected(Exception): """Response-local control signal for a downstream client disconnect.""" @@ -1356,6 +1417,33 @@ def _scrub_log_token(value): # Non-strings pass through for existing %s rendering semantics. return _sanitize_sensitive_text(value) if isinstance(value, str) else value + +def _record_wire_divergence(kind): + # Process-level aggregation is protected even when validators run from multiple + # server tasks or test threads. Emit one discovery warning, then one bounded + # summary per fixed interval; no event IDs or payload values enter the record. + # The summary reports records suppressed strictly BETWEEN emitted records: the + # first summary at observation 100 therefore covers observations 2..99 (98), + # while later summaries cover 101..199, 201..299, and so on (99 each). Keep the + # warning writes inside the same lock so concurrent observations cannot reorder + # an emitted summary ahead of the discovery record it follows. + with _WIRE_DIVERGENCE_LOCK: + count = _WIRE_DIVERGENCE_COUNTS.get(kind, 0) + 1 + _WIRE_DIVERGENCE_COUNTS[kind] = count + if count == 1: + log.warning("wire-divergence: %s observed=1", kind) + _WIRE_DIVERGENCE_LAST_EMITTED[kind] = count + elif count % _WIRE_DIVERGENCE_SUMMARY_INTERVAL == 0: + previous_emitted = _WIRE_DIVERGENCE_LAST_EMITTED.get(kind, 0) + suppressed = count - previous_emitted - 1 + log.warning( + "wire-divergence-summary kind=%s observed=%d suppressed=%d", + kind, count, suppressed, + ) + _WIRE_DIVERGENCE_LAST_EMITTED[kind] = count + return count + + # v1.2.0: reasoning-item cache (module-level, bounded LRU). # INTENT: the Responses API pairs a `reasoning` output item with the # function_call(s) that follow it. On store:false stateless replay the caller @@ -1971,133 +2059,274 @@ def _system_to_text(system): return "\n".join(p for p in parts if p) -def _flatten_tool_result_content(tr_content): - # tool_result content may be a string OR an array of blocks. Flatten to text. - # INTENT: the Responses `function_call_output.output` field is a plain string, - # so we reduce the Anthropic tool_result content to text (same flattening - # the chat lane used, preserved verbatim for behavioral continuity). - if isinstance(tr_content, list): - flat = [] - for sub in tr_content: - if isinstance(sub, dict) and sub.get("type") == "text": - flat.append(sub.get("text", "")) - elif isinstance(sub, str): - flat.append(sub) - return "\n".join(flat) +_IMAGE_MEDIA_TYPES = frozenset({ + "image/jpeg", "image/png", "image/gif", "image/webp", +}) +_BASE64_IMAGE_RE = re.compile( + r"(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?\Z" +) +_CONTENT_LABEL_RE = re.compile(r"[^A-Za-z0-9_.-]+") + + +def _content_label(value): + # Content errors may identify only structural coordinates and type labels. Keep + # arbitrary request text, URLs, file IDs, and image bytes structurally unreachable. + if not isinstance(value, str) or not value: + return "unknown" + return _CONTENT_LABEL_RE.sub("_", value)[:40] or "unknown" + + +def _content_error(message_index, block_index, role, block_type, reason): + return _InvalidRequestError( + "message %d block %s role %s type %s: %s" % ( + message_index, block_index, _content_label(role), + _content_label(block_type), reason, + ) + ) + + +def _image_to_responses_part(block, message_index, block_index, role): + # INTENT: translate visual bytes without inspecting, decoding, or logging them. + # REASONING: MIME prefixes come only from an exact allowlist; strict RFC-4648 + # syntax is checked with a bounded-character regex so validation never creates + # a second decoded image copy. GIF is accepted syntactically; this stdlib-only + # shim does not claim animated-GIF detection, so the later live lane smoke is + # the release gate for each provider endpoint. + if role != "user": + raise _content_error( + message_index, block_index, role, "image", + "assistant/history images are unsupported", + ) + source = block.get("source") + if not isinstance(source, dict): + raise _content_error( + message_index, block_index, role, "image", "invalid image source", + ) + source_type = source.get("type") + if source_type == "base64": + media_type = source.get("media_type") + data = source.get("data") + if media_type not in _IMAGE_MEDIA_TYPES: + raise _content_error( + message_index, block_index, role, "image", + "unsupported image media type", + ) + if ( + not isinstance(data, str) + or not data + or len(data) % 4 != 0 + or _BASE64_IMAGE_RE.fullmatch(data) is None + ): + raise _content_error( + message_index, block_index, role, "image", "invalid base64 image data", + ) + return { + "type": "input_image", + "image_url": "data:%s;base64,%s" % (media_type, data), + "detail": "auto", + } + if source_type == "url": + url = source.get("url") + valid = isinstance(url, str) and bool(url) and not any( + char.isspace() or ord(char) < 0x20 or ord(char) == 0x7f + for char in url + ) and "\\" not in url + if valid: + try: + parsed = urllib.parse.urlsplit(url) + valid = ( + parsed.scheme.lower() in {"http", "https"} + and bool(parsed.netloc) + and bool(parsed.hostname) + and parsed.username is None + and parsed.password is None + ) + if valid: + parsed.port # force invalid-port validation + except (TypeError, ValueError): + valid = False + if not valid: + raise _content_error( + message_index, block_index, role, "image", "invalid image URL", + ) + return {"type": "input_image", "image_url": url, "detail": "auto"} + if source_type == "file": + raise _content_error( + message_index, block_index, role, "image", + "provider-scoped file image sources are unsupported", + ) + raise _content_error( + message_index, block_index, role, "image", "unsupported image source type", + ) + + +def _tool_result_output(tr_content, message_index, block_index, role): + # Preserve the historical string output for text-only results. The Responses + # function-call output item-list form is used only when an image is present. + # Other container/scalar types are malformed Anthropic history, not values to + # serialize opportunistically into a backend function-call output. if isinstance(tr_content, str): return tr_content - return json.dumps(tr_content) + if not isinstance(tr_content, list): + raise _content_error( + message_index, block_index, role, "tool_result", + "content must be a string or list (got %s)" % _content_label( + type(tr_content).__name__ + ), + ) + parts = [] + flat_text = [] + saw_image = False + for sub_index, sub in enumerate(tr_content): + nested_index = "%d.%d" % (block_index, sub_index) + if isinstance(sub, str): + text = sub + parts.append({"type": "input_text", "text": text}) + flat_text.append(text) + continue + if not isinstance(sub, dict): + raise _content_error( + message_index, nested_index, role, "non_object", + "unsupported tool-result content block", + ) + sub_type = sub.get("type") + if sub_type == "text": + text = sub.get("text", "") + if not isinstance(text, str): + raise _content_error( + message_index, nested_index, role, "text", "invalid text block", + ) + parts.append({"type": "input_text", "text": text}) + flat_text.append(text) + elif sub_type == "image": + saw_image = True + parts.append(_image_to_responses_part( + sub, message_index, nested_index, role, + )) + else: + raise _content_error( + message_index, nested_index, role, sub_type, + "unsupported tool-result content block", + ) + return parts if saw_image else "\n".join(flat_text) def _messages_to_input(messages): - # Translate Anthropic messages[] into a Responses `input[]` array, threading - # cached reasoning items in before the function_calls they justify. - # - # INTENT: produce the ordered list of Responses input items: - # * user text -> {"role":"user","content":[{"type":"input_text",...}]} - # * assistant text -> {"type":"message","role":"assistant", - # "content":[{"type":"output_text",...}]} - # * assistant tool_use -> [cached reasoning item?] then - # {"type":"function_call","call_id","name","arguments"} - # * user tool_result -> {"type":"function_call_output","call_id","output"} - # REASONING: assistant history content parts MUST be `output_text`, NOT - # `input_text` — the Responses API rejects input_text on resent assistant - # messages (spec file 04 §1 "Important deprecation"). Reasoning items are - # injected from the cache (see _REASONING_CACHE) immediately before their - # paired function_call so gpt-5.x's required-reasoning-item invariant holds - # on stateless replay. - # ASSUMES: image/thinking/other unknown blocks are ignored gracefully (chat- - # lane behavior preserved). Thinking blocks that Claude Code resends from - # OUR own thinking emission are DROPPED here — the reasoning cache, not - # resent thinking text, is the continuity mechanism. + # Translate in source order. User text/images remain adjacent content parts; + # assistant tool replay keeps reasoning-cache injection immediately before calls. input_items = [] - injected_reasoning_ids = set() # dedupe: a reasoning item shared by parallel - # calls is injected ONCE per request build. - missing_reasoning = 0 # count cache misses for the request log line. - - for m in messages: - role = m.get("role", "user") - content = m.get("content", "") - - # Plain string content -> a single message item for this role. + injected_reasoning_ids = set() + missing_reasoning = 0 + + for message_index, message in enumerate(messages): + role = message.get("role") + # v1.2.13: fold role:"system" messages to user-role input. Live Claude + # Code appends a system-role message to `messages`; pre-v1.2.12 code + # mapped every non-assistant role to user and that behavior is proven + # against the live Codex backend, so system reuses it verbatim rather + # than inventing untested native system/developer-role translation. + # ORDERING DEPENDENCY: the request-validation gauntlet checks RAW roles + # and pre-rejects system-role messages carrying tool_use/tool_result; + # this fold runs after it, so the post-fold tool checks below never see + # them. Callers must not invoke this translator without that gauntlet. + if role == "system": + role = "user" + if role not in ("user", "assistant"): + raise _InvalidRequestError( + "message %d role invalid: expected user or assistant" % message_index + ) + content = message.get("content", "") if isinstance(content, str): - if role == "assistant": - input_items.append({ - "type": "message", "role": "assistant", - "content": [{"type": "output_text", "text": content}], - }) - else: - input_items.append({ - "role": "user", - "content": [{"type": "input_text", "text": content}], - }) + input_items.append({ + **({"type": "message"} if role == "assistant" else {}), + "role": "assistant" if role == "assistant" else "user", + "content": [{ + "type": "output_text" if role == "assistant" else "input_text", + "text": content, + }], + }) continue - text_parts = [] - for block in content: - if not isinstance(block, dict): - continue - btype = block.get("type") - - if btype == "text": - text_parts.append(block.get("text", "")) - - # Emit the message item (text) for this turn FIRST, so text precedes the - # function_call items within an assistant turn (mirrors output order). - if text_parts: - joined = "\n".join(text_parts) - if role == "assistant": - input_items.append({ - "type": "message", "role": "assistant", - "content": [{"type": "output_text", "text": joined}], - }) - else: - input_items.append({ - "role": "user", - "content": [{"type": "input_text", "text": joined}], - }) + pending_parts = [] - # Second pass over blocks for tool_use / tool_result (order-preserving). - for block in content: + def flush_message_parts(): + if not pending_parts: + return + input_items.append({ + **({"type": "message"} if role == "assistant" else {}), + "role": "assistant" if role == "assistant" else "user", + "content": list(pending_parts), + }) + pending_parts.clear() + + for block_index, block in enumerate(content): if not isinstance(block, dict): + raise _content_error( + message_index, block_index, role, "non_object", + "unsupported content block", + ) + block_type = block.get("type") + if block_type == "text": + text = block.get("text", "") + if not isinstance(text, str): + raise _content_error( + message_index, block_index, role, "text", "invalid text block", + ) + pending_parts.append({ + "type": "output_text" if role == "assistant" else "input_text", + "text": text, + }) + elif block_type == "image": + pending_parts.append(_image_to_responses_part( + block, message_index, block_index, role, + )) + elif block_type in {"thinking", "redacted_thinking"}: + # Known Claude continuity blocks are intentionally consumed rather + # than forwarded. Encrypted Responses reasoning cache entries, not + # replayed Anthropic thinking prose/signatures, preserve continuity. continue - btype = block.get("type") - - if btype == "tool_use": + elif block_type == "tool_use": + if role != "assistant": + raise _content_error( + message_index, block_index, role, "tool_use", + "tool_use is permitted only in assistant messages", + ) + flush_message_parts() call_id = block.get("id", "") - # REASONING-CACHE INJECTION: if we have a cached reasoning item - # for this call_id, insert it immediately BEFORE the function_call - # (unless an equivalent reasoning item — same id — was already - # injected for this request build; parallel calls share one). cached = _REASONING_CACHE.get(call_id) if cached is not None: - rs_id = cached.get("id") - if rs_id not in injected_reasoning_ids: + reasoning_id = cached.get("id") + if reasoning_id not in injected_reasoning_ids: input_items.append(cached) - injected_reasoning_ids.add(rs_id) + injected_reasoning_ids.add(reasoning_id) else: - # Cache miss (e.g. shim restarted mid-session, or unknown - # call_id). Omit the reasoning item and proceed — do NOT fail - # the request. Live testing will establish whether the API - # hard-rejects; the documented risk is the community-reported - # "function_call was provided without its required 'reasoning' - # item" error. missing_reasoning += 1 input_items.append({ "type": "function_call", "call_id": call_id, "name": block.get("name", ""), - # arguments is a JSON *string*, not an object. "arguments": json.dumps(block.get("input", {})), }) - - elif btype == "tool_result": + elif block_type == "tool_result": + if role != "user": + raise _content_error( + message_index, block_index, role, "tool_result", + "tool_result is permitted only in user messages", + ) + flush_message_parts() input_items.append({ "type": "function_call_output", "call_id": block.get("tool_use_id", ""), - "output": _flatten_tool_result_content(block.get("content", "")), + "output": _tool_result_output( + block.get("content", ""), message_index, block_index, role, + ), }) - # Unknown block types (image, thinking) are ignored gracefully. + else: + raise _content_error( + message_index, block_index, role, block_type, + "unsupported content block", + ) + flush_message_parts() return input_items, missing_reasoning @@ -2425,8 +2654,7 @@ def _validate_stream_event_fields(event): if "name" in event: _require_protocol_string(event.get("name"), "function name") else: - log.warning("wire-divergence: arguments.done missing name (item %s)", - _scrub_log_token(event.get("item_id"))) + _record_wire_divergence("arguments.done_missing_name") if "arguments" in event: _require_protocol_string( event.get("arguments"), "function arguments", allow_empty=True @@ -2642,7 +2870,22 @@ def _diag_headers(headers): return " ".join(parts) if parts else "(none)" -# --- HARDENING: retry helper (UNCHANGED) --- +# --- HARDENING: retry helper --- + + +def _transport_failure_phase(error, post_stream_start=False): + if post_stream_start: + return "post_stream_start_body_read" + if isinstance(error, (httpx.ConnectTimeout, httpx.ConnectError)): + return "connect" + if isinstance(error, httpx.PoolTimeout): + return "connection_pool_wait" + if isinstance(error, (httpx.WriteTimeout, httpx.WriteError)): + return "request_body_write" + if isinstance(error, (httpx.ReadTimeout, httpx.ReadError)): + return "header_wait_or_body_read" + return "connect_or_header_wait" + async def _discard_httpx_response(response): """Close a successful buffered POST result discarded by outer cancellation.""" @@ -2692,6 +2935,9 @@ async def _post_with_retry(url, headers, payload, disconnect_event): raise except httpx.HTTPError as e: last_exc = e + _record_transport_failure( + e, _transport_failure_phase(e), attempt < MAX_RETRIES, + ) if attempt < MAX_RETRIES: delay = _retry_delay(attempt, None) _record_retry("transport") @@ -2820,6 +3066,9 @@ async def _open_backend_stream(url, headers, payload, disconnect_event): raise cancellation except httpx.HTTPError as e: await _settle_stream_context(stream_cm) + _record_transport_failure( + e, _transport_failure_phase(e), attempt < MAX_RETRIES, + ) if attempt >= MAX_RETRIES: _record_backend_error( "api_error", backend_type=type(e).__name__, @@ -3174,16 +3423,17 @@ def _bearer_of(headers): async def _build_backend_headers(force_token_refresh=False, rejected_token=None): # v1.2.5: assemble the backend request headers for the active lane. - # INTENT: openai mode -> {Authorization: Bearer , Content-Type}. chatgpt - # mode -> the 3-header floor {authorization: Bearer , - # content-type: application/json, accept: text/event-stream} the Codex backend - # gates on (notes/08); NO api-key/openai-specific headers. + # INTENT: both lanes add the local X-Client-Request-Id for safe correlation. + # OpenAI mode otherwise sends Authorization + Content-Type; ChatGPT mode sends + # its validated authorization/content-type/accept floor and no API-key header. # REASONING: keeping this in one helper means the lazy-401 retry path can rebuild # the headers (with force_token_refresh=True + the rejected token for the # guarded-reload identity check) identically to the first attempt. # CREDENTIAL SAFETY: returns a dict CONTAINING the Bearer; callers must never log # it. In chatgpt mode a token-layer failure raises RuntimeError(_RELOGIN_MSG) # (no secret content) for the caller to surface as a clean client error. + state = _request_state() + client_request_id = state.client_request_id if state is not None else uuid.uuid4().hex if SHIM_BACKEND_MODE == "chatgpt": access_token = await _get_access_token( force_refresh=force_token_refresh, rejected_token=rejected_token) @@ -3191,14 +3441,34 @@ async def _build_backend_headers(force_token_refresh=False, rejected_token=None) "authorization": f"Bearer {access_token}", "content-type": "application/json", "accept": "text/event-stream", + "X-Client-Request-Id": client_request_id, } - # openai mode (unchanged behavior). return { "Authorization": f"Bearer {SHIM_BACKEND_API_KEY}", "Content-Type": "application/json", + "X-Client-Request-Id": client_request_id, } +def _pending_sse_data_bytes(line_buffer): + # Return the logical payload bytes currently buffered for a data field, or None + # when the partial line is not a data field. SSE framing ("data:", one optional + # space, and a possible trailing CR awaiting LF) is deliberately excluded. + # This makes the event cap independent of where upstream transport chunks split, + # including splits inside the field prefix and immediately before CRLF. + logical_end = len(line_buffer) + if logical_end and line_buffer[-1] == 0x0D: + logical_end -= 1 + if logical_end == 4 and line_buffer[:logical_end] == b"data": + return 0 + if logical_end < 5 or line_buffer[:5] != b"data:": + return None + value_start = 5 + if logical_end > value_start and line_buffer[value_start] == 0x20: + value_start += 1 + return logical_end - value_start + + def _consume_sse_line(raw_line, data_lines, event_size, event_open): # v1.2.7 bounded SSE framing primitive. SSE joins repeated data fields with a # newline and dispatches only on a REAL blank line. Comments do not open an @@ -3248,8 +3518,21 @@ async def _iter_bounded_sse_data(resp, disconnect_event): while True: newline_at = line_buffer.find(b"\n") if newline_at < 0: - if len(line_buffer) > MAX_RESPONSES_SSE_EVENT_BYTES: - raise ValueError("upstream SSE line exceeded size limit") + pending_data_bytes = _pending_sse_data_bytes(line_buffer) + if pending_data_bytes is None: + # Non-data fields still receive a raw-line bound. Data fields use + # the logical event bound below so their fixed SSE framing does not + # make an exactly-at-limit payload fail under one chunking pattern. + if len(line_buffer) > MAX_RESPONSES_SSE_EVENT_BYTES: + raise ValueError("upstream SSE line exceeded size limit") + else: + joined_size = ( + event_size + + (1 if data_lines else 0) + + pending_data_bytes + ) + if joined_size > MAX_RESPONSES_SSE_EVENT_BYTES: + raise ValueError("upstream SSE event exceeded size limit") break raw_line = bytes(line_buffer[:newline_at]) del line_buffer[:newline_at + 1] @@ -3305,6 +3588,11 @@ async def _accumulate_terminal_response(resp, disconnect_event): phase="upstream_stream", ) return None, message + except httpx.HTTPError as error: + _record_transport_failure( + error, _transport_failure_phase(error, post_stream_start=True), False, + ) + raise except _ProtocolError as error: return None, _scrub_and_trim_body(str(error))[:200] or "backend protocol error" except ValueError as error: @@ -3347,12 +3635,42 @@ async def _watch_disconnect(): invalid_structure = not isinstance(messages, list) if not invalid_structure: invalid_structure = any(not isinstance(message, dict) for message in messages) + if not invalid_structure: + # v1.2.13: "system" is admitted alongside user/assistant. Live Claude + # Code sends a trailing role:"system" message inside `messages`; the + # v1.2.12 two-role check rejected every real conversation turn with + # the static 400 (captured-payload repro: scripts/scratch/ + # replay_captured_request.py). Translation folds system to user-role + # input, the pre-v1.2.12 behavior proven against the live backend. + invalid_structure = any( + message.get("role") not in ("user", "assistant", "system") + for message in messages + ) if not invalid_structure: invalid_structure = any( "content" in message and not isinstance(message.get("content"), (str, list)) for message in messages ) + if not invalid_structure: + invalid_structure = any( + isinstance(message.get("content"), list) + and any( + isinstance(block, dict) + and ( + (block.get("type") == "tool_use" + and message.get("role") != "assistant") + or (block.get("type") == "tool_result" + and message.get("role") != "user") + or ( + block.get("type") == "tool_result" + and not isinstance(block.get("content", ""), (str, list)) + ) + ) + for block in message["content"] + ) + for message in messages + ) if not invalid_structure: invalid_structure = any( isinstance(message.get("content"), list) @@ -3400,8 +3718,17 @@ async def _watch_disconnect(): # every log line below, and the response echoed to Claude Code. `model` from # here on is suffix-free; `slug_effort_raw` is the parsed tier-2 token (or None). model, slug_effort_raw = _split_effort_suffix(req.get("model", "")) - responses_payload, missing_reasoning, effort_value, effort_source = \ - _anthropic_to_responses_request(req, model, slug_effort_raw) + try: + responses_payload, missing_reasoning, effort_value, effort_source = \ + _anthropic_to_responses_request(req, model, slug_effort_raw) + except _InvalidRequestError as error: + # The exception text is constructed exclusively from structural indexes, + # bounded type labels, and local constants. Never log or reflect block data. + _mark_error(phase="request_translation", error_type="invalid_request_error") + await _send_json(send, 400, {"type": "error", "error": { + "type": "invalid_request_error", "message": str(error), + }}) + return n_msgs = len(responses_payload["input"]) n_tools = len(responses_payload.get("tools", [])) state = _request_state() @@ -3461,9 +3788,9 @@ async def _watch_disconnect(): return # v1.2.5: build the backend auth headers by lane. - # INTENT: in openai mode, Bearer the env API key (unchanged). In chatgpt mode, - # Bearer the OAuth access_token from auth.json and emit ONLY the 3-header floor - # the Codex backend gates on (notes/08). Request translation, tools, and SSE + # INTENT: in openai mode, Bearer the env API key. In chatgpt mode, Bearer the + # OAuth access_token from auth.json and emit its validated header floor. Both + # lanes add the local X-Client-Request-Id for content-blind correlation. Tools and SSE # block lifecycle stay shared. v1.2.7 makes chatgpt transport always-streaming # (inbound non-stream callers are internally accumulated), while v1.2.6's one # route-specific response-formatting rule preserves reliable reasoning-summary @@ -4642,6 +4969,9 @@ async def _finalize_stream_failure(message, error_type="api_error"): elif isinstance(e, ValueError): failure_kind = "backend SSE framing error" elif isinstance(e, httpx.HTTPError): + _record_transport_failure( + e, _transport_failure_phase(e, post_stream_start=True), False, + ) failure_kind = "backend transport error" else: failure_kind = "backend stream translation error" @@ -4728,6 +5058,7 @@ async def _handle_models(send): async def _handle_health(send): # HARDENING: health endpoint for the manager's idempotency + --status checks. await _send_json(send, 200, { + "service": SHIM_SERVICE_ID, "status": "ok", "backend": SHIM_BACKEND_BASE_URL, "backend_mode": SHIM_BACKEND_MODE, diff --git a/scripts/provider_shim/start_shim.sh b/scripts/provider_shim/start_shim.sh index 346c87d7..8712e60f 100755 --- a/scripts/provider_shim/start_shim.sh +++ b/scripts/provider_shim/start_shim.sh @@ -4,29 +4,28 @@ # # The shim (anthropic_openai_shim.py) translates Anthropic Messages API calls # from Claude Code into OpenAI Responses API calls (POST /v1/responses) to a -# configured backend. -# This script starts it, stops it, reports status, and — under --auto — silently -# no-ops unless the user has opted in via DAAF_PROVIDER_SHIM=openai. +# configured backend. This manager starts it, atomically restarts it, stops it, +# reports status, rotates its local log, and supervises unexpected exits. # # Modes: # --auto Silent no-op unless DAAF_PROVIDER_SHIM=openai. If opted in, # start-if-not-running with a keepalive supervisor. Boot-safe: -# never fatal, never blocks (used by the container entrypoint). +# never fatal and never blocks container startup indefinitely. # --start Start the shim (with keepalive supervisor) if not already running. +# --restart Under one lifecycle lock: stop any identity-verified generation, +# wait for exit, launch one replacement, and require strict readiness. # --stop Stop the shim and its supervisor. -# --status Print running/stopped + health JSON. +# --status Print running/stopped + strictly validated health JSON. +# --help Print public action usage. # # Config (env, all optional — defaults mirror the shim's own): # DAAF_PROVIDER_SHIM activation switch; must equal "openai" to auto-start # SHIM_PORT default 4141 -# SHIM_BACKEND_MODE backend lane (v1.2.5): "openai" (default) API-key lane, -# or "chatgpt" ChatGPT-subscription Codex lane. In chatgpt -# mode the shim reads the OAuth access_token from -# $CODEX_HOME/auth.json (NOT an API key) and routes to -# https://chatgpt.com/backend-api/codex/responses with the -# 3-header floor. REQUIRES CODEX_HOME set + a readable -# auth.json (this manager warns loudly at start if absent; -# the shim itself fails fast per request). +# SHIM_BACKEND_MODE backend lane selector: "openai" (default) API-key +# lane, or "chatgpt" ChatGPT-subscription Codex lane. +# Resolution matches the shim: case-insensitive and +# whitespace-trimmed; an unknown value falls back to +# "openai" (the shim logs the configuration warning). # SHIM_BACKEND_BASE_URL default is mode-conditional: openai -> # https://api.openai.com/v1; chatgpt -> # https://chatgpt.com/backend-api/codex. Explicit value @@ -37,46 +36,37 @@ # OAuth token store (mode 0600). Compose sets it to # /home/appuser/.claude/codex-daaf. # SHIM_OAUTH_TOKEN_URL (chatgpt mode) TEST/STAGING OVERRIDE only. Production -# default is the hardcoded auth.openai.com/oauth/token; -# set only to point the token refresh at a mock/staging -# endpoint. Leave unset in production. +# default is the hardcoded auth.openai.com/oauth/token. # SHIM_OAUTH_CLIENT_ID (chatgpt mode) TEST/STAGING OVERRIDE only. Production -# default is the hardcoded codex first-party client_id; -# set only for a mock/staging endpoint expecting a -# different client_id. Leave unset in production. +# default is the hardcoded codex first-party client_id. # SHIM_STRIP_MODEL_PREFIX default "" -# SHIM_SANITIZE_TOOLS default "1" (enabled); strips known GPT tool-call -# quirks — set to 0 (and restart the shim) for +# SHIM_SANITIZE_TOOLS default "1" (enabled); set to 0 and restart for # DAAFBench runs of shim-routed models -# SHIM_REASONING_EFFORT tier 3 of the v1.2.3 effort precedence chain -# (per-request signal > "#" slug suffix > this -# env var > default "high"); sets reasoning.effort only -# when no per-request signal or slug suffix is present. -# An inbound per-request "high" is treated as unset -# since v1.2.3 (the client pins it for GPT slugs). -# Valid: none|low|medium|high|xhigh|max ("max" is -# gpt-5.6-only). Unset -> default "high" (the shim -# always sends an effort since v1.2.2) -# SHIM_TEXT_VERBOSITY response verbosity (v1.2.4+); the outbound request -# always carries text.verbosity. Valid low|medium|high; -# default "high" (warmth/volume, parity with DAAF's -# posture). Read once at startup like the flags above. +# SHIM_REASONING_EFFORT tier 3 of the effort precedence chain (per-request +# signal > "#" slug suffix > env > high) +# SHIM_TEXT_VERBOSITY low|medium|high; default high # -# Keepalive: a background supervisor restarts the shim if it exits, with a short -# sleep between restarts. A restart-storm guard stops after 10 crashes in 60s -# and logs loudly, so a misconfigured shim can't spin forever. +# Keepalive: the supervisor restarts the shim after an unexpected exit. A guard +# stops after 10 crashes in 60 seconds so a broken configuration cannot spin. # -# Logging: the shim's stderr and the supervisor's own messages are appended to -# logs/shim.log. The log dir is created on demand and is gitignored. +# Logging: shim stderr and supervisor records share logs/shim.log. Supervisor +# records are emitted once to stderr; the supervisor's single outer redirect is +# the only writer, avoiding the historical tee + redirect duplicate. Before a +# new supervisor starts, a >25 MiB log rotates to shim.log.1 with five retained +# generations. The 25 MiB / five-generation policy is fixed deliberately: the +# observed 12.7 MiB over nine days remains comfortably below one generation, +# while worst-case retained supervisor history stays bounded near 150 MiB. +# Rotation is serialized with lifecycle start/restart/stop, rejects symlink/non-regular +# targets, and creates the replacement log (0600) before the first new record. # ============================================================================= set -uo pipefail -# Deliberately omit -e: several paths (health probe, stale-pid cleanup) inspect -# failures and decide what to do rather than aborting. This is a lifecycle -# manager, not a safety hook — it must degrade gracefully, especially under -# --auto where it runs at container boot. +umask 077 +# Deliberately omit -e: health probes, stale-state cleanup, and process teardown +# inspect failures and choose an explicit recovery path. --auto must degrade +# gracefully rather than aborting container boot. -# --- Config --- +# --- Config ----------------------------------------------------------------- SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)" readonly SCRIPT_DIR readonly SHIM_PY="${SCRIPT_DIR}/anthropic_openai_shim.py" @@ -84,292 +74,1031 @@ readonly LOG_DIR="${SCRIPT_DIR}/logs" readonly LOG_FILE="${LOG_DIR}/shim.log" readonly PID_FILE="${LOG_DIR}/shim.pid" readonly SUP_PID_FILE="${LOG_DIR}/supervisor.pid" -# Records the supervisor's process-group id so --stop can reap the supervisor -# AND its shim child atomically (they share a session/group via setsid). A plain -# per-pid kill races the supervisor's restart loop, which can respawn a child in -# the kill window and orphan it — observed during live validation. readonly PGID_FILE="${LOG_DIR}/pgid" -# Sentinel written by --stop, polled by the supervisor loop, so a stop can never -# race a restart. readonly STOP_FILE="${LOG_DIR}/stop.requested" +readonly LOCK_DIR="${LOG_DIR}/lifecycle.lock" +readonly LOCK_OWNER_FILE="${LOCK_DIR}/owner.pid" +readonly LOG_WRITE_LOCK_DIR="${LOG_DIR}/log-write.lock" +readonly LOG_WRITE_OWNER_FILE="${LOG_WRITE_LOCK_DIR}/owner.pid" +readonly SHIM_SERVICE_ID="daaf-anthropic-openai-shim" SHIM_PORT="${SHIM_PORT:-4141}" +readonly SHIM_PORT readonly HEALTH_URL="http://127.0.0.1:${SHIM_PORT}/health" -# Restart-storm guard. -readonly STORM_LIMIT=10 # crashes... -readonly STORM_WINDOW=60 # ...within this many seconds -> give up +readonly LOG_MAX_BYTES=26214400 # 25 MiB; rotate only when strictly greater. +readonly LOG_GENERATIONS=5 +readonly STORM_LIMIT=10 +readonly STORM_WINDOW=60 +readonly RESTART_DELAY=2 +readonly READINESS_WAIT=15 -# --- Helpers --- -log_line() { - # Timestamped supervisor/manager line -> log file and stderr. - printf '%s SUPERVISOR %s\n' "$(date -u '+%Y-%m-%d %H:%M:%S')" "$*" | tee -a "$LOG_FILE" >&2 +# Test-only timing controls are intentionally outside the SHIM_* user-config +# namespace and activate only when the isolated Bats harness opts in. Production +# always uses the fixed policy above. +storm_limit="$STORM_LIMIT" +storm_window="$STORM_WINDOW" +restart_delay="$RESTART_DELAY" +readiness_wait="$READINESS_WAIT" +if [ "${DAAF_SHIM_TEST_MODE:-0}" = "1" ]; then + storm_limit="${DAAF_SHIM_TEST_STORM_LIMIT:-$STORM_LIMIT}" + storm_window="${DAAF_SHIM_TEST_STORM_WINDOW:-$STORM_WINDOW}" + restart_delay="${DAAF_SHIM_TEST_RESTART_DELAY:-$RESTART_DELAY}" + readiness_wait="${DAAF_SHIM_TEST_READINESS_WAIT:-$READINESS_WAIT}" +fi + +HEALTH_JSON="" +HEALTH_REASON="not_probed" +EXPECTED_SHIM_VERSION="" +EXPECTED_BACKEND_MODE="openai" +START_FAILURE_KIND="not_started" +ACTIVE_LAUNCH_PID="" +LIFECYCLE_LOCK_HELD=0 + +# --- Filesystem and logging helpers ----------------------------------------- +path_is_safe_file_target() { + # Missing is safe; an existing target must be a regular, non-symlink file. + local target="$1" + [ ! -L "$target" ] || return 1 + [ ! -e "$target" ] || [ -f "$target" ] } ensure_log_dir() { - mkdir -p "$LOG_DIR" 2>/dev/null || true + if [ -L "$LOG_DIR" ]; then + printf 'ERROR: shim log directory is a symlink: %s\n' "$LOG_DIR" >&2 + printf ' Fix: replace it with a real directory owned by the container user.\n' >&2 + return 1 + fi + if [ -e "$LOG_DIR" ] && [ ! -d "$LOG_DIR" ]; then + printf 'ERROR: shim log path exists but is not a directory: %s\n' "$LOG_DIR" >&2 + printf ' Fix: move that object aside and recreate a private directory.\n' >&2 + return 1 + fi + mkdir -p "$LOG_DIR" || return 1 + chmod 0700 "$LOG_DIR" || return 1 + return 0 +} + +state_targets_are_safe() { + local target generation + for target in "$LOG_FILE" "$PID_FILE" "$SUP_PID_FILE" "$PGID_FILE" "$STOP_FILE"; do + if ! path_is_safe_file_target "$target"; then + printf 'ERROR: refusing unsafe shim state target: %s\n' "$target" >&2 + printf ' Fix: remove the symlink/non-regular object and retry.\n' >&2 + return 1 + fi + done + generation=1 + while [ "$generation" -le "$LOG_GENERATIONS" ]; do + target="${LOG_FILE}.${generation}" + if ! path_is_safe_file_target "$target"; then + printf 'ERROR: refusing unsafe shim log-generation target: %s\n' "$target" >&2 + printf ' Fix: remove the symlink/non-regular object and retry.\n' >&2 + return 1 + fi + generation=$((generation + 1)) + done + if [ -L "$LOCK_DIR" ] || { [ -e "$LOCK_DIR" ] && [ ! -d "$LOCK_DIR" ]; }; then + printf 'ERROR: refusing unsafe shim lifecycle-lock target: %s\n' "$LOCK_DIR" >&2 + printf ' Fix: remove the symlink/non-directory object and retry.\n' >&2 + return 1 + fi + if [ -L "$LOG_WRITE_LOCK_DIR" ] || { + [ -e "$LOG_WRITE_LOCK_DIR" ] && [ ! -d "$LOG_WRITE_LOCK_DIR" ] + }; then + printf 'ERROR: refusing unsafe shim log-write-lock target: %s\n' \ + "$LOG_WRITE_LOCK_DIR" >&2 + printf ' Fix: remove the symlink/non-directory object and retry.\n' >&2 + return 1 + fi + return 0 +} + +acquire_log_write_lock() { + local owner attempts=0 + while [ "$attempts" -lt 200 ]; do + if mkdir "$LOG_WRITE_LOCK_DIR" 2>/dev/null; then + printf '%s\n' "${BASHPID:-$$}" > "$LOG_WRITE_OWNER_FILE" || return 1 + chmod 0700 "$LOG_WRITE_LOCK_DIR" || return 1 + chmod 0600 "$LOG_WRITE_OWNER_FILE" || return 1 + return 0 + fi + owner="$(read_pid_file "$LOG_WRITE_OWNER_FILE")" || owner="" + if [ -z "$owner" ] || ! kill -0 "$owner" 2>/dev/null; then + rm -f "$LOG_WRITE_OWNER_FILE" 2>/dev/null || true + rmdir "$LOG_WRITE_LOCK_DIR" 2>/dev/null || true + else + sleep 0.01 + fi + attempts=$((attempts + 1)) + done + printf 'ERROR: timed out acquiring shim log-write lock: %s\n' \ + "$LOG_WRITE_LOCK_DIR" >&2 + return 1 +} + +release_log_write_lock() { + rm -f "$LOG_WRITE_OWNER_FILE" 2>/dev/null || true + rmdir "$LOG_WRITE_LOCK_DIR" 2>/dev/null || true +} + +append_log_record() { + local record="$1" rc=0 + acquire_log_write_lock || return 1 + state_targets_are_safe || rc=1 + if [ "$rc" -eq 0 ]; then + if [ ! -e "$LOG_FILE" ]; then + : > "$LOG_FILE" || rc=1 + [ "$rc" -ne 0 ] || chmod 0600 "$LOG_FILE" || rc=1 + fi + fi + if [ "$rc" -eq 0 ] && [ "${DAAF_SHIM_TEST_MODE:-0}" = "1" ] && \ + [ "${DAAF_SHIM_TEST_RESULT_APPEND_FAILURE:-0}" = "1" ]; then + case "$record" in + *' MANAGER SHIM_RESTART_RESULT '*) rc=1 ;; + esac + fi + if [ "$rc" -eq 0 ]; then + printf '%s\n' "$record" >> "$LOG_FILE" || rc=1 + fi + release_log_write_lock + return "$rc" +} + +log_line() { + # Every supervisor event has one authoritative append. A separate terminal + # write occurs only when stderr is an interactive TTY; it never feeds back + # through the log writer and therefore cannot duplicate the stored record. + local record + record="$(printf '%s SUPERVISOR %s' "$(date -u '+%Y-%m-%d %H:%M:%S')" "$*")" + append_log_record "$record" || return 1 + [ -t 2 ] && printf '%s\n' "$record" >&2 + return 0 +} + +manager_log_line() { + # Foreground manager events append once and remain visible to their caller. + local record + record="$(printf '%s MANAGER %s' "$(date -u '+%Y-%m-%d %H:%M:%S')" "$*")" + append_log_record "$record" || return 1 + printf '%s\n' "$record" >&2 +} + +write_stream_to_log() { + # Preserve Python's structured records byte-for-byte while serializing each + # physical line through the same append lock as supervisor/manager records. + local line + while IFS= read -r line || [ -n "$line" ]; do + append_log_record "$line" || return 1 + done } -health_json() { - # Print the health JSON to stdout, or nothing. Returns 0 iff healthy. - local out - out="$(curl -fsS --max-time 2 "$HEALTH_URL" 2>/dev/null)" || return 1 - printf '%s' "$out" +rotate_log_if_needed_unlocked() { + ensure_log_dir || return 1 + state_targets_are_safe || return 1 + + if [ ! -e "$LOG_FILE" ]; then + : > "$LOG_FILE" || return 1 + chmod 0600 "$LOG_FILE" || return 1 + return 0 + fi + + local size generation previous + size="$(wc -c < "$LOG_FILE" 2>/dev/null)" || return 1 + case "$size" in + ''|*[!0-9]*) + printf 'ERROR: could not determine shim log size safely: %s\n' "$LOG_FILE" >&2 + return 1 + ;; + esac + + if [ "$size" -le "$LOG_MAX_BYTES" ]; then + chmod 0600 "$LOG_FILE" || return 1 + return 0 + fi + + # All targets were preflighted before mutation. Descending renames preserve + # generations deterministically; the oldest regular file is removed first. + if [ -e "${LOG_FILE}.${LOG_GENERATIONS}" ]; then + rm -f "${LOG_FILE}.${LOG_GENERATIONS}" || return 1 + fi + generation="$LOG_GENERATIONS" + while [ "$generation" -gt 1 ]; do + previous=$((generation - 1)) + if [ -e "${LOG_FILE}.${previous}" ]; then + mv "${LOG_FILE}.${previous}" "${LOG_FILE}.${generation}" || return 1 + chmod 0600 "${LOG_FILE}.${generation}" || return 1 + fi + generation=$((generation - 1)) + done + mv "$LOG_FILE" "${LOG_FILE}.1" || return 1 + chmod 0600 "${LOG_FILE}.1" || return 1 + : > "$LOG_FILE" || return 1 + chmod 0600 "$LOG_FILE" || return 1 + return 0 +} + +rotate_log_if_needed() { + local rc + acquire_log_write_lock || return 1 + rotate_log_if_needed_unlocked + rc=$? + release_log_write_lock + return "$rc" +} + +# --- Configuration and health helpers -------------------------------------- +source_shim_version() { + local version + version="$(awk -F'"' '/^SHIM_VERSION = "/ { print $2; exit }' "$SHIM_PY" 2>/dev/null)" + case "$version" in + ''|*[!0-9A-Za-z._+-]*) return 1 ;; + esac + printf '%s' "$version" +} + +resolve_backend_mode() { + local raw normalized + raw="${SHIM_BACKEND_MODE:-openai}" + # Match the shim's practical startup semantics: trim surrounding whitespace, + # lowercase, accept the two known values, otherwise fall back to openai. + normalized="${raw#"${raw%%[![:space:]]*}"}" + normalized="${normalized%"${normalized##*[![:space:]]}"}" + normalized="$(printf '%s' "$normalized" | tr '[:upper:]' '[:lower:]')" + case "$normalized" in + openai|chatgpt) EXPECTED_BACKEND_MODE="$normalized" ;; + *) EXPECTED_BACKEND_MODE="openai" ;; + esac +} + +load_expected_contract() { + EXPECTED_SHIM_VERSION="$(source_shim_version)" || { + printf 'ERROR: could not derive SHIM_VERSION from %s.\n' "$SHIM_PY" >&2 + printf ' Fix: restore the provider-shim source file and retry.\n' >&2 + return 1 + } + resolve_backend_mode + return 0 +} + +probe_health() { + HEALTH_JSON="" + HEALTH_REASON="unreachable" + local response http_status out + response="$(curl -sS --max-time 2 --write-out $'\n%{http_code}' \ + "$HEALTH_URL" 2>/dev/null)" + http_status="${response##*$'\n'}" + case "$http_status" in + [0-9][0-9][0-9]) ;; + *) return 1 ;; + esac + if [ "$http_status" = "000" ]; then + return 1 + fi + if [ "$http_status" != "200" ]; then + HEALTH_REASON="http_status_${http_status}" + return 1 + fi + # Any received 200 is reachable. Validate only its captured body and never + # expose that body in diagnostics, even if curl reported a transfer anomaly. + out="${response%$'\n'*}" + HEALTH_JSON="$out" + + if ! printf '%s' "$out" | jq -e 'type == "object"' >/dev/null 2>&1; then + HEALTH_REASON="malformed_json" + return 1 + fi + if ! printf '%s' "$out" | jq -e --arg service "$SHIM_SERVICE_ID" \ + '.service == $service and .status == "ok"' >/dev/null 2>&1; then + HEALTH_REASON="wrong_identity_or_status" + return 1 + fi + if ! printf '%s' "$out" | jq -e --arg version "$EXPECTED_SHIM_VERSION" \ + '.version == $version' >/dev/null 2>&1; then + HEALTH_REASON="version_mismatch" + return 1 + fi + if ! printf '%s' "$out" | jq -e --arg mode "$EXPECTED_BACKEND_MODE" \ + '.backend_mode == $mode' >/dev/null 2>&1; then + HEALTH_REASON="backend_mode_mismatch" + return 1 + fi + HEALTH_REASON="ready" return 0 } is_healthy() { - curl -fsS --max-time 2 "$HEALTH_URL" >/dev/null 2>&1 + probe_health >/dev/null +} + +emit_readiness_failure() { + printf 'SHIM_READINESS_FAILURE status=failed reason=%s expected_service=%s expected_version=%s expected_backend_mode=%s port=%s\n' \ + "$HEALTH_REASON" "$SHIM_SERVICE_ID" "${EXPECTED_SHIM_VERSION:--}" \ + "$EXPECTED_BACKEND_MODE" "$SHIM_PORT" >&2 +} + +# --- Process identity helpers ---------------------------------------------- +is_decimal_pid() { + case "${1:-}" in + ''|*[!0-9]*) return 1 ;; + *) [ "$1" -gt 1 ] 2>/dev/null ;; + esac +} + +read_pid_file() { + local path="$1" value + value="$(awk 'NR == 1 { print; exit }' "$path" 2>/dev/null)" + is_decimal_pid "$value" || return 1 + printf '%s' "$value" +} + +pid_has_exact_arg() { + local pid="$1" expected="$2" arg + [ -r "/proc/${pid}/cmdline" ] || return 1 + while IFS= read -r -d '' arg; do + [ "$arg" = "$expected" ] && return 0 + done < "/proc/${pid}/cmdline" + return 1 +} + +pid_is_supervisor() { + local pid="$1" + is_decimal_pid "$pid" || return 1 + kill -0 "$pid" 2>/dev/null || return 1 + pid_has_exact_arg "$pid" "$SHIM_PY" && return 1 + pid_has_exact_arg "$pid" "${BASH_SOURCE[0]}" || return 1 + pid_has_exact_arg "$pid" "__supervise" +} + +pid_is_shim() { + local pid="$1" + is_decimal_pid "$pid" || return 1 + kill -0 "$pid" 2>/dev/null || return 1 + pid_has_exact_arg "$pid" "$SHIM_PY" } supervisor_running() { - # 0 iff a live supervisor process is recorded and still alive. - [ -f "$SUP_PID_FILE" ] || return 1 local sp - sp="$(cat "$SUP_PID_FILE" 2>/dev/null || true)" - [ -n "$sp" ] || return 1 - kill -0 "$sp" 2>/dev/null + sp="$(read_pid_file "$SUP_PID_FILE")" || return 1 + pid_is_supervisor "$sp" } -# --- Keepalive supervisor ----------------------------------------------------- -# Runs in the background (invoked as: this_script __supervise). Loops: launch -# the shim, wait for it, restart on exit, honoring the storm guard. Writes the -# shim's PID to PID_FILE each cycle and its own PID to SUP_PID_FILE. +process_group_is_owned() { + local pgid="$1" supervisor_pid="$2" actual + is_decimal_pid "$pgid" || return 1 + [ "$pgid" = "$supervisor_pid" ] || return 1 + pid_is_supervisor "$supervisor_pid" || return 1 + actual="$(ps -o pgid= -p "$supervisor_pid" 2>/dev/null | tr -d ' ')" + [ "$actual" = "$pgid" ] +} + +terminate_verified_pid() { + local role="$1" pid="$2" waited=0 + case "$role" in + supervisor) pid_is_supervisor "$pid" || return 1 ;; + shim) pid_is_shim "$pid" || return 1 ;; + *) return 1 ;; + esac + kill -TERM "$pid" 2>/dev/null || true + while [ "$waited" -lt 5 ]; do + if [ "$role" = "supervisor" ]; then + pid_is_supervisor "$pid" || return 0 + else + pid_is_shim "$pid" || return 0 + fi + sleep 1 + waited=$((waited + 1)) + done + if [ "$role" = "supervisor" ]; then + pid_is_supervisor "$pid" && kill -KILL "$pid" 2>/dev/null || true + else + pid_is_shim "$pid" && kill -KILL "$pid" 2>/dev/null || true + fi + return 0 +} + +# --- Lifecycle lock --------------------------------------------------------- +acquire_lifecycle_lock() { + local wait_mode="${1:-wait}" owner attempts=0 + ensure_log_dir || return 1 + state_targets_are_safe || return 1 + + # Public lifecycle actions serialize by waiting for the current live owner. + # This lock is deliberately acquired only by outer action wrappers; helpers + # with a _locked suffix never recurse into this non-reentrant lock. + # Sixty seconds exceeds the manager's five-second stop plus 15-second + # readiness budget while remaining bounded if an owner is wedged. + while [ "$attempts" -lt 6000 ]; do + if mkdir "$LOCK_DIR" 2>/dev/null; then + LIFECYCLE_LOCK_HELD=1 + trap release_lifecycle_lock EXIT + trap 'exit 130' INT TERM HUP + if ! printf '%s\n' "$$" > "$LOCK_OWNER_FILE"; then + release_lifecycle_lock + return 1 + fi + if ! chmod 0700 "$LOCK_DIR" || ! chmod 0600 "$LOCK_OWNER_FILE"; then + release_lifecycle_lock + return 1 + fi + return 0 + fi + + owner="$(read_pid_file "$LOCK_OWNER_FILE")" || owner="" + if [ -z "$owner" ] || ! kill -0 "$owner" 2>/dev/null; then + # A dead/malformed owner means a prior manager died while holding the + # lock. Remove only the known owner file and empty lock directory. + rm -f "$LOCK_OWNER_FILE" 2>/dev/null || true + rmdir "$LOCK_DIR" 2>/dev/null || true + else + if [ "$wait_mode" = "no_wait" ]; then + printf 'ERROR: another shim lifecycle action is active (pid %s).\n' "$owner" >&2 + printf ' Fix: let it finish; boot will continue and a later health check can retry.\n' >&2 + return 1 + fi + sleep 0.01 + fi + attempts=$((attempts + 1)) + done + + printf 'ERROR: timed out acquiring shim lifecycle lock: %s\n' "$LOCK_DIR" >&2 + printf ' Fix: wait for the active lifecycle action to finish, then retry.\n' >&2 + return 1 +} + +release_lifecycle_lock() { + local owner="" + if [ "$LIFECYCLE_LOCK_HELD" -eq 1 ]; then + owner="$(read_pid_file "$LOCK_OWNER_FILE")" || owner="" + if [ "$owner" = "$$" ]; then + rm -f "$LOCK_OWNER_FILE" 2>/dev/null || true + rmdir "$LOCK_DIR" 2>/dev/null || true + fi + LIFECYCLE_LOCK_HELD=0 + fi + trap - EXIT INT TERM HUP +} + +# --- Keepalive supervisor --------------------------------------------------- run_supervisor() { - ensure_log_dir - echo "$$" > "$SUP_PID_FILE" - # Record our process-group id. When launched via `setsid` the supervisor is a - # group leader, so its PGID == its PID and the shim child inherits the group. - # --stop signals the whole group by this id. - local pgid + ensure_log_dir || exit 1 + state_targets_are_safe || exit 1 + printf '%s\n' "$$" > "$SUP_PID_FILE" || exit 1 + chmod 0600 "$SUP_PID_FILE" || exit 1 + + local pgid child_pid logger_pid log_pipe window_start crashes rc now pgid="$(ps -o pgid= -p "$$" 2>/dev/null | tr -d ' ')" - [ -n "$pgid" ] && echo "$pgid" > "$PGID_FILE" + if [ "$pgid" = "$$" ]; then + printf '%s\n' "$pgid" > "$PGID_FILE" || exit 1 + chmod 0600 "$PGID_FILE" || exit 1 + else + # nohup fallback may share the caller's process group. Never record or kill + # that broad group; verified per-pid teardown is safer. + rm -f "$PGID_FILE" 2>/dev/null || true + log_line "process-group isolation unavailable; using verified per-pid stop" + fi - # A stop sentinel lets --stop tell the supervisor to quit WITHOUT racing the - # restart loop: the loop checks for it before every (re)launch and sleep tick. rm -f "$STOP_FILE" 2>/dev/null || true + child_pid="" + logger_pid="" + log_pipe="${LOG_DIR}/shim.stream.$$" + if [ -e "$log_pipe" ] || [ -L "$log_pipe" ]; then + printf 'ERROR: refusing existing shim log-stream target: %s\n' "$log_pipe" >&2 + exit 1 + fi + mkfifo "$log_pipe" || exit 1 + chmod 0600 "$log_pipe" || exit 1 - # Clean up the shim child on supervisor termination. - local child_pid="" - cleanup_sup() { - if [ -n "$child_pid" ]; then - kill "$child_pid" 2>/dev/null || true + cleanup_supervisor() { + if [ -n "$child_pid" ] && pid_is_shim "$child_pid"; then + kill -TERM "$child_pid" 2>/dev/null || true + fi + if [ -n "$logger_pid" ] && kill -0 "$logger_pid" 2>/dev/null; then + kill -TERM "$logger_pid" 2>/dev/null || true + wait "$logger_pid" 2>/dev/null || true fi - rm -f "$PID_FILE" "$SUP_PID_FILE" "$PGID_FILE" 2>/dev/null || true - log_line "supervisor exiting" + rm -f "$log_pipe" "$PID_FILE" "$SUP_PID_FILE" "$PGID_FILE" 2>/dev/null || true + log_line "supervisor exiting" || true } - trap cleanup_sup EXIT INT TERM + stop_supervisor_signal() { + trap - INT TERM + exit 0 + } + trap cleanup_supervisor EXIT + trap stop_supervisor_signal INT TERM - local window_start crashes window_start="$(date +%s)" crashes=0 + log_line "starting shim on port ${SHIM_PORT} backend_mode=${EXPECTED_BACKEND_MODE} version=${EXPECTED_SHIM_VERSION} (keepalive on)" - log_line "starting shim on port ${SHIM_PORT} (backend from env; keepalive on)" while true; do - # Honor a stop request that arrived before (re)launch. if [ -f "$STOP_FILE" ]; then log_line "stop requested; supervisor loop ending" exit 0 fi - # Launch the shim; its stderr (structured logs) appends to the log file. - python3 "$SHIM_PY" >>"$LOG_FILE" 2>&1 & + + write_stream_to_log < "$log_pipe" & + logger_pid=$! + python3 "$SHIM_PY" > "$log_pipe" 2>&1 & child_pid=$! - echo "$child_pid" > "$PID_FILE" + printf '%s\n' "$child_pid" > "$PID_FILE" + chmod 0600 "$PID_FILE" 2>/dev/null || true wait "$child_pid" - local rc=$? + rc=$? child_pid="" + wait "$logger_pid" 2>/dev/null || true + logger_pid="" + rm -f "$PID_FILE" 2>/dev/null || true - # If a stop was requested while the shim ran, do not restart. if [ -f "$STOP_FILE" ]; then log_line "stop requested; not restarting shim" exit 0 fi - # Storm guard: count crashes within a rolling window. - local now now="$(date +%s)" - if [ $((now - window_start)) -gt "$STORM_WINDOW" ]; then + if [ $((now - window_start)) -gt "$storm_window" ]; then window_start="$now" crashes=0 fi crashes=$((crashes + 1)) - log_line "shim exited rc=${rc} (crash ${crashes}/${STORM_LIMIT} in ${STORM_WINDOW}s window)" - if [ "$crashes" -ge "$STORM_LIMIT" ]; then - log_line "RESTART STORM: ${STORM_LIMIT} crashes in ${STORM_WINDOW}s — giving up. Check config (SHIM_BACKEND_API_KEY, SHIM_PORT) and ${LOG_FILE}." + log_line "shim exited rc=${rc} (crash ${crashes}/${storm_limit} in ${storm_window}s window)" + if [ "$crashes" -ge "$storm_limit" ]; then + log_line "RESTART_STORM crashes=${crashes} window_s=${storm_window} action=give_up log=${LOG_FILE}" exit 1 fi - sleep 2 + + # Sleep in short, sentinel-aware ticks so --stop does not wait for the full + # restart delay. TERM also interrupts the sleep and exits via the trap. + local slept=0 + while [ "$slept" -lt 20 ]; do + [ -f "$STOP_FILE" ] && break + sleep "$(awk -v delay="$restart_delay" 'BEGIN { printf "%.3f", delay / 20 }')" + slept=$((slept + 1)) + done done } -# --- Public actions ----------------------------------------------------------- -do_start() { - ensure_log_dir +# --- Stop implementation (caller holds lifecycle lock) --------------------- +stop_processes_locked() { + local stopped=0 failed=0 pgid sp pp waited - if [ ! -f "$SHIM_PY" ]; then - echo "ERROR: shim not found at ${SHIM_PY}" >&2 - echo " Fix: ensure the provider_shim directory is intact." >&2 - return 1 + # Sentinel first: the supervisor checks it before every launch and sleep tick. + : > "$STOP_FILE" || return 2 + chmod 0600 "$STOP_FILE" 2>/dev/null || true + + sp="$(read_pid_file "$SUP_PID_FILE")" || sp="" + pp="$(read_pid_file "$PID_FILE")" || pp="" + pgid="$(read_pid_file "$PGID_FILE")" || pgid="" + + # Deterministic failure injection is confined to explicitly isolated tests. + if [ "${DAAF_SHIM_TEST_MODE:-0}" = "1" ] && \ + [ "${DAAF_SHIM_TEST_STOP_FAILURE:-0}" = "1" ] && \ + { { [ -n "$sp" ] && pid_is_supervisor "$sp"; } || \ + { [ -n "$pp" ] && pid_is_shim "$pp"; }; }; then + return 2 + fi + + if [ -n "$sp" ] && [ -n "$pgid" ] && process_group_is_owned "$pgid" "$sp"; then + kill -TERM "-${pgid}" 2>/dev/null || true + stopped=1 + waited=0 + while [ "$waited" -lt 5 ]; do + if ! pid_is_supervisor "$sp" && { [ -z "$pp" ] || ! pid_is_shim "$pp"; }; then + break + fi + sleep 1 + waited=$((waited + 1)) + done + if pid_is_supervisor "$sp" && process_group_is_owned "$pgid" "$sp"; then + kill -KILL "-${pgid}" 2>/dev/null || true + fi + fi + + # Missing/no-setsid PGID and stale-file cases use exact cmdline identity. A PID + # recycled by an unrelated process is never signalled. + if [ -n "$pp" ] && pid_is_shim "$pp"; then + terminate_verified_pid shim "$pp" || true + stopped=1 + fi + if [ -n "$sp" ] && pid_is_supervisor "$sp"; then + terminate_verified_pid supervisor "$sp" || true + stopped=1 + fi + + # A stop is successful only after every originally verified identity is gone. + # Preserve the sentinel and pid evidence on failure so a restart cannot launch + # a second generation while the first may still be live. + if [ -n "$pp" ] && pid_is_shim "$pp"; then + failed=1 fi - if ! command -v python3 >/dev/null 2>&1; then - echo "ERROR: python3 not found on PATH; cannot start shim." >&2 + if [ -n "$sp" ] && pid_is_supervisor "$sp"; then + failed=1 + fi + if [ "$failed" -eq 1 ]; then + return 2 + fi + + rm -f "$PID_FILE" "$SUP_PID_FILE" "$PGID_FILE" "$STOP_FILE" 2>/dev/null || return 2 + if [ "$stopped" -eq 1 ]; then + return 0 + fi + return 3 +} + +# --- Public actions --------------------------------------------------------- +preflight_start_contract() { + if [ ! -f "$SHIM_PY" ] || [ -L "$SHIM_PY" ]; then + printf 'ERROR: shim source is missing or unsafe: %s\n' "$SHIM_PY" >&2 + printf ' Fix: restore the provider_shim directory from the DAAF repository.\n' >&2 return 1 fi + local dependency + for dependency in python3 curl jq ps awk wc; do + if ! command -v "$dependency" >/dev/null 2>&1; then + printf 'ERROR: required shim-manager dependency is unavailable: %s\n' "$dependency" >&2 + printf ' Fix: rebuild from the current DAAF Dockerfile.\n' >&2 + return 1 + fi + done + load_expected_contract +} + +do_start_locked() { + local launch_kind="${1:-explicit}" waited=0 spawned=0 supervisor_pid="" + START_FAILURE_KIND="launch" + ACTIVE_LAUNCH_PID="" - # v1.2.5: surface the backend mode, and in chatgpt mode preflight the OAuth - # auth store. This is a WARNING, not a hard stop — the shim itself fails fast - # per request with the re-login message, and a transient auth.json (e.g. mid - # codex login) should not block the manager. But surfacing it here gives the - # operator an immediate, actionable signal instead of a per-request 401. - local backend_mode="${SHIM_BACKEND_MODE:-openai}" - echo "Shim backend mode: ${backend_mode}." >&2 - if [ "${backend_mode}" = "chatgpt" ]; then + preflight_start_contract || return 1 + + printf 'Shim backend mode: %s.\n' "$EXPECTED_BACKEND_MODE" >&2 + if [ "$EXPECTED_BACKEND_MODE" = "chatgpt" ]; then if [ -z "${CODEX_HOME:-}" ]; then - echo "WARNING: SHIM_BACKEND_MODE=chatgpt but CODEX_HOME is unset." >&2 - echo " The shim needs \$CODEX_HOME/auth.json for the OAuth Bearer." >&2 - echo " Fix: set CODEX_HOME and run 'codex login --device-auth' inside the container." >&2 + printf 'WARNING: SHIM_BACKEND_MODE=chatgpt but CODEX_HOME is unset.\n' >&2 + printf ' Fix: set CODEX_HOME and run codex login --device-auth.\n' >&2 elif [ ! -r "${CODEX_HOME}/auth.json" ]; then - echo "WARNING: chatgpt mode auth store not found/readable at \$CODEX_HOME/auth.json." >&2 - echo " Fix: run 'codex login --device-auth' inside the container to authenticate." >&2 + printf 'WARNING: chatgpt auth store is missing/unreadable at $CODEX_HOME/auth.json.\n' >&2 + printf ' Fix: run codex login --device-auth inside the container.\n' >&2 else - echo "chatgpt auth store present at \$CODEX_HOME/auth.json." >&2 + printf 'chatgpt auth store present at $CODEX_HOME/auth.json.\n' >&2 fi fi - # Idempotency: already healthy? Nothing to do. if is_healthy; then - echo "Shim already running and healthy on port ${SHIM_PORT}." >&2 + START_FAILURE_KIND="none" + printf 'Shim already running and strictly ready on port %s.\n' "$SHIM_PORT" >&2 return 0 fi - # Supervisor alive but not yet healthy (still booting)? Leave it. if supervisor_running; then - echo "Shim supervisor already running (pid $(cat "$SUP_PID_FILE" 2>/dev/null))." >&2 - return 0 + START_FAILURE_KIND="readiness" + printf 'WARNING: shim supervisor is running but strict readiness failed.\n' >&2 + emit_readiness_failure + return 1 + fi + # If another HTTP service answered /health, do not start a crash loop against + # its occupied port. Only an unreachable endpoint is eligible for launch. + if [ "$HEALTH_REASON" != "unreachable" ]; then + START_FAILURE_KIND="readiness" + printf 'ERROR: port %s answered /health but not with this shim contract.\n' "$SHIM_PORT" >&2 + emit_readiness_failure + return 1 fi - # Clean any stale pidfiles/sentinel from a prior crash or stop. rm -f "$PID_FILE" "$SUP_PID_FILE" "$PGID_FILE" "$STOP_FILE" 2>/dev/null || true + rotate_log_if_needed || return 1 - # Launch the supervisor in its OWN session/process-group via setsid, fully - # detached so it survives this shell returning. Being a group leader lets - # --stop reap the supervisor and its shim child together by process group, - # which is what closes the orphan-child race a plain per-pid kill leaves open. - if command -v setsid >/dev/null 2>&1; then - setsid "${BASH_SOURCE[0]}" __supervise >>"$LOG_FILE" 2>&1 & + if command -v setsid >/dev/null 2>&1 && [ "${DAAF_SHIM_TEST_NO_SETSID:-0}" != "1" ]; then + setsid "${BASH_SOURCE[0]}" __supervise >/dev/null 2>&1 & else - # Fallback: nohup + disown. --stop still works via the recorded PGID, - # which the supervisor derives from its own `ps` lookup either way. - nohup "${BASH_SOURCE[0]}" __supervise >>"$LOG_FILE" 2>&1 & + nohup "${BASH_SOURCE[0]}" __supervise >/dev/null 2>&1 & fi + supervisor_pid=$! + ACTIVE_LAUNCH_PID="$supervisor_pid" + spawned=1 disown 2>/dev/null || true - # Wait (bounded) for health. - local waited=0 - while [ "$waited" -lt 15 ]; do + while [ "$waited" -lt "$readiness_wait" ]; do if is_healthy; then - echo "Shim started and healthy on port ${SHIM_PORT}." >&2 + START_FAILURE_KIND="none" + ACTIVE_LAUNCH_PID="" + printf 'Shim started and strictly ready on port %s.\n' "$SHIM_PORT" >&2 return 0 fi + # The detached process needs a scheduling turn to create supervisor.pid. + # Keep waiting while the exact PID returned by the launch is alive; once + # the pidfile exists, supervisor_running adds the cmdline identity check. + if ! supervisor_running && ! kill -0 "$supervisor_pid" 2>/dev/null; then + break + fi sleep 1 waited=$((waited + 1)) done - echo "WARNING: shim did not report healthy within 15s. Check ${LOG_FILE}." >&2 + + if [ "$HEALTH_REASON" = "unreachable" ] && ! supervisor_running; then + START_FAILURE_KIND="launch" + else + START_FAILURE_KIND="readiness" + fi + printf 'WARNING: shim failed strict readiness within %ss. Check %s.\n' \ + "$readiness_wait" "$LOG_FILE" >&2 + emit_readiness_failure + if [ "$launch_kind" != "auto" ] && [ "$spawned" -eq 1 ]; then + stop_processes_locked >/dev/null 2>&1 || true + fi + ACTIVE_LAUNCH_PID="" return 1 } +do_start() { + local launch_kind="${1:-explicit}" rc + if [ "$launch_kind" = "auto" ]; then + acquire_lifecycle_lock no_wait || return 1 + else + acquire_lifecycle_lock wait || return 1 + fi + do_start_locked "$launch_kind" + rc=$? + release_lifecycle_lock + return "$rc" +} + do_stop() { - local stopped=0 - - # 1. Raise the stop sentinel FIRST so the supervisor's restart loop will not - # respawn a child while we tear things down (closes the observed race). - ensure_log_dir - : > "$STOP_FILE" 2>/dev/null || true - - # 2. Signal the whole process group if we recorded one. `kill -- -PGID` - # delivers to every member (supervisor + shim child + any sleep), so the - # child cannot outlive the supervisor. - local pgid - pgid="$(cat "$PGID_FILE" 2>/dev/null || true)" - if [ -n "$pgid" ] && kill -0 "-${pgid}" 2>/dev/null; then - kill -TERM "-${pgid}" 2>/dev/null || true - stopped=1 - # Give the group a moment to exit, then escalate if anything survives. - local waited=0 - while [ "$waited" -lt 5 ]; do - if ! kill -0 "-${pgid}" 2>/dev/null; then - break - fi - sleep 1 - waited=$((waited + 1)) - done - if kill -0 "-${pgid}" 2>/dev/null; then - kill -KILL "-${pgid}" 2>/dev/null || true - fi + local stop_rc + ensure_log_dir || return 1 + state_targets_are_safe || return 1 + acquire_lifecycle_lock || return 1 + stop_processes_locked + stop_rc=$? + release_lifecycle_lock + case "$stop_rc" in + 0) + printf 'Shim stopped.\n' >&2 + return 0 + ;; + 3) + printf 'Shim was not running.\n' >&2 + return 0 + ;; + *) + printf 'ERROR: shim stop did not terminate every verified process.\n' >&2 + printf ' Fix: inspect %s and retry --stop before starting another generation.\n' \ + "$LOG_FILE" >&2 + return 1 + ;; + esac +} + +# Restart exit codes are intentionally distinct for automation: +# 40 stop failure, 41 launch failure, 42 strict-readiness failure, +# 43 manager-record failure, and 130 interruption. +RESTART_PHASE="not_started" +RESTART_RESULT_WRITE_IN_PROGRESS=0 +RESTART_RESULT_WRITTEN=0 +RESTART_OLD_SUPERVISOR_PID="-" +RESTART_OLD_SHIM_PID="-" + +emit_restart_result_locked() { + local status="$1" stage="$2" reason="$3" exit_code="$4" new_supervisor_pid="${5:--}" + local record + [ "$RESTART_RESULT_WRITTEN" -eq 0 ] || return 0 + if [ "$RESTART_RESULT_WRITE_IN_PROGRESS" -eq 1 ]; then + printf 'SHIM_RESTART_RESULT status=failed stage=record reason=write_interrupted exit_code=130 old_supervisor_pid=%s old_shim_pid=%s new_supervisor_pid=%s port=%s record_persisted=unknown\n' \ + "$RESTART_OLD_SUPERVISOR_PID" "$RESTART_OLD_SHIM_PID" \ + "$new_supervisor_pid" "$SHIM_PORT" >&2 + return 1 fi - # 3. Fallback / belt-and-suspenders: also target the recorded individual pids - # in case the PGID file is missing (e.g. setsid unavailable at start). - local sp pp - sp="$(cat "$SUP_PID_FILE" 2>/dev/null || true)" - if [ -n "$sp" ] && kill -0 "$sp" 2>/dev/null; then - kill -TERM "$sp" 2>/dev/null || true - stopped=1 + record="SHIM_RESTART_RESULT status=${status} stage=${stage} reason=${reason} exit_code=${exit_code} old_supervisor_pid=${RESTART_OLD_SUPERVISOR_PID} old_shim_pid=${RESTART_OLD_SHIM_PID} new_supervisor_pid=${new_supervisor_pid} port=${SHIM_PORT}" + RESTART_RESULT_WRITE_IN_PROGRESS=1 + if [ "${DAAF_SHIM_TEST_MODE:-0}" = "1" ] && \ + [ "${DAAF_SHIM_TEST_SIGNAL_RESULT_IN_PROGRESS:-0}" = "1" ]; then + DAAF_SHIM_TEST_SIGNAL_RESULT_IN_PROGRESS=0 + kill -TERM "$$" fi - pp="$(cat "$PID_FILE" 2>/dev/null || true)" - if [ -n "$pp" ] && kill -0 "$pp" 2>/dev/null; then - kill -TERM "$pp" 2>/dev/null || true - stopped=1 + if ! manager_log_line "$record"; then + RESTART_RESULT_WRITE_IN_PROGRESS=0 + printf 'SHIM_RESTART_RESULT status=failed stage=record reason=manager_log_write exit_code=43 old_supervisor_pid=%s old_shim_pid=%s new_supervisor_pid=%s port=%s record_persisted=no\n' \ + "$RESTART_OLD_SUPERVISOR_PID" "$RESTART_OLD_SHIM_PID" \ + "$new_supervisor_pid" "$SHIM_PORT" >&2 + return 1 + fi + RESTART_RESULT_WRITE_IN_PROGRESS=0 + RESTART_RESULT_WRITTEN=1 + return 0 +} + +handle_restart_interrupt() { + local signal_name="$1" launch_waited=0 interrupted_during_write=0 + trap - INT TERM HUP + RESTART_PHASE="interrupted_${RESTART_PHASE}" + if [ "$RESTART_RESULT_WRITE_IN_PROGRESS" -eq 1 ]; then + interrupted_during_write=1 + printf 'SHIM_RESTART_RESULT status=failed stage=record reason=write_interrupted exit_code=130 old_supervisor_pid=%s old_shim_pid=%s new_supervisor_pid=- port=%s record_persisted=unknown\n' \ + "$RESTART_OLD_SUPERVISOR_PID" "$RESTART_OLD_SHIM_PID" "$SHIM_PORT" >&2 + fi + # Once an atomic restart has begun, interruption never leaves a newly launched + # unverified generation behind. The direct launch PID closes the short window + # before supervisor.pid is written; cmdline verification still precedes signal. + while [ -n "$ACTIVE_LAUNCH_PID" ] && kill -0 "$ACTIVE_LAUNCH_PID" 2>/dev/null && \ + ! pid_is_supervisor "$ACTIVE_LAUNCH_PID" && [ "$launch_waited" -lt 100 ]; do + sleep 0.01 + launch_waited=$((launch_waited + 1)) + done + if [ -n "$ACTIVE_LAUNCH_PID" ] && pid_is_supervisor "$ACTIVE_LAUNCH_PID"; then + terminate_verified_pid supervisor "$ACTIVE_LAUNCH_PID" || true fi + stop_processes_locked >/dev/null 2>&1 || true + if [ "$interrupted_during_write" -eq 0 ]; then + emit_restart_result_locked failed interrupt "signal_${signal_name}" 130 - || true + fi + release_lifecycle_lock + exit 130 +} - rm -f "$PID_FILE" "$SUP_PID_FILE" "$PGID_FILE" "$STOP_FILE" 2>/dev/null || true +do_restart() { + local prior="stopped" stop_rc start_rc new_supervisor_pid="-" - if [ "$stopped" -eq 1 ]; then - echo "Shim stopped." >&2 - else - echo "Shim was not running." >&2 + ensure_log_dir || return 43 + state_targets_are_safe || return 43 + acquire_lifecycle_lock || return 43 + preflight_start_contract || { + manager_log_line "SHIM_RESTART_BEGIN status=begin prior=unknown old_supervisor_pid=- old_shim_pid=- port=${SHIM_PORT}" || true + emit_restart_result_locked failed launch launch_preflight_failed 41 - || true + release_lifecycle_lock + return 41 + } + + RESTART_PHASE="inspect" + RESTART_RESULT_WRITE_IN_PROGRESS=0 + RESTART_RESULT_WRITTEN=0 + RESTART_OLD_SUPERVISOR_PID="$(read_pid_file "$SUP_PID_FILE")" || \ + RESTART_OLD_SUPERVISOR_PID="-" + RESTART_OLD_SHIM_PID="$(read_pid_file "$PID_FILE")" || RESTART_OLD_SHIM_PID="-" + + if is_healthy; then + prior="running_ready" + elif supervisor_running; then + prior="running_unready" + elif [ "$HEALTH_REASON" != "unreachable" ]; then + prior="unexpected_service" + fi + + if ! manager_log_line "SHIM_RESTART_BEGIN status=begin prior=${prior} old_supervisor_pid=${RESTART_OLD_SUPERVISOR_PID} old_shim_pid=${RESTART_OLD_SHIM_PID} port=${SHIM_PORT}"; then + printf 'SHIM_RESTART_RESULT status=failed stage=record reason=manager_log_write exit_code=43 old_supervisor_pid=%s old_shim_pid=%s new_supervisor_pid=- port=%s record_persisted=no\n' \ + "$RESTART_OLD_SUPERVISOR_PID" "$RESTART_OLD_SHIM_PID" "$SHIM_PORT" >&2 + release_lifecycle_lock + return 43 fi + + trap 'handle_restart_interrupt INT' INT + trap 'handle_restart_interrupt TERM' TERM + trap 'handle_restart_interrupt HUP' HUP + + # A healthy endpoint without manager-owned process identity must never be + # signalled or mistaken for a generation this manager may replace. + if [ "$prior" = "running_ready" ] && \ + [ "$RESTART_OLD_SUPERVISOR_PID" = "-" ] && \ + [ "$RESTART_OLD_SHIM_PID" = "-" ]; then + emit_restart_result_locked failed stop unmanaged_ready_service 40 - || true + release_lifecycle_lock + return 40 + fi + + RESTART_PHASE="stop" + stop_processes_locked + stop_rc=$? + case "$stop_rc" in + 0|3) + ;; + *) + emit_restart_result_locked failed stop termination_failed 40 - || true + release_lifecycle_lock + return 40 + ;; + esac + + RESTART_PHASE="launch" + do_start_locked restart + start_rc=$? + if [ "$start_rc" -ne 0 ]; then + stop_processes_locked >/dev/null 2>&1 || true + if [ "$START_FAILURE_KIND" = "launch" ]; then + emit_restart_result_locked failed launch launch_failed 41 - || true + release_lifecycle_lock + return 41 + fi + emit_restart_result_locked failed readiness strict_readiness_failed 42 - || true + release_lifecycle_lock + return 42 + fi + + RESTART_PHASE="verify" + if ! probe_health; then + emit_readiness_failure + stop_processes_locked >/dev/null 2>&1 || true + emit_restart_result_locked failed readiness strict_readiness_failed 42 - || true + release_lifecycle_lock + return 42 + fi + + new_supervisor_pid="$(read_pid_file "$SUP_PID_FILE")" || new_supervisor_pid="-" + if [ "$new_supervisor_pid" = "-" ] || ! pid_is_supervisor "$new_supervisor_pid" || \ + { [ "$RESTART_OLD_SUPERVISOR_PID" != "-" ] && \ + [ "$new_supervisor_pid" = "$RESTART_OLD_SUPERVISOR_PID" ]; }; then + stop_processes_locked >/dev/null 2>&1 || true + emit_restart_result_locked failed readiness supervisor_identity_invalid 42 \ + "$new_supervisor_pid" || true + release_lifecycle_lock + return 42 + fi + + RESTART_PHASE="ready" + if ! emit_restart_result_locked ready readiness ready 0 "$new_supervisor_pid"; then + release_lifecycle_lock + return 43 + fi + release_lifecycle_lock + printf 'Shim restarted and strictly ready on port %s.\n' "$SHIM_PORT" >&2 return 0 } do_status() { + if ! command -v curl >/dev/null 2>&1 || ! command -v jq >/dev/null 2>&1; then + printf 'STATUS: unavailable (curl and jq are required for strict readiness)\n' + return 1 + fi + if [ ! -f "$SHIM_PY" ] || ! load_expected_contract; then + printf 'STATUS: unavailable (shim source/version contract missing)\n' + return 1 + fi if is_healthy; then - echo "STATUS: running (port ${SHIM_PORT})" - echo "HEALTH: $(health_json)" + printf 'STATUS: running and strictly ready (port %s)\n' "$SHIM_PORT" + printf 'HEALTH: %s\n' "$HEALTH_JSON" return 0 fi if supervisor_running; then - echo "STATUS: supervisor up but shim not yet healthy (port ${SHIM_PORT})" - return 0 + printf 'STATUS: supervisor up but shim not strictly ready (port %s)\n' "$SHIM_PORT" + emit_readiness_failure + return 1 + fi + if [ "$HEALTH_REASON" != "unreachable" ]; then + printf 'STATUS: unexpected service on shim port %s\n' "$SHIM_PORT" + emit_readiness_failure + return 1 fi - echo "STATUS: stopped" + printf 'STATUS: stopped\n' return 0 } do_auto() { - # Silent no-op unless the user opted in. This is the container-boot path, so - # it must never be noisy or fatal. if [ "${DAAF_PROVIDER_SHIM:-}" != "openai" ]; then - # Not opted in — exit silently, success. exit 0 fi - ensure_log_dir - log_line "--auto: DAAF_PROVIDER_SHIM=openai detected; ensuring shim is running." - # start-if-not-running; swallow failures so boot always proceeds. - do_start || log_line "--auto: do_start reported non-zero; continuing boot anyway." + ensure_log_dir || { + printf 'SHIM_AUTO_FAILURE status=failed reason=unsafe_log_directory\n' >&2 + exit 0 + } + if ! do_start auto; then + manager_log_line "SHIM_AUTO_FAILURE status=failed reason=start_or_readiness_check continuing_boot=y" || \ + printf 'SHIM_AUTO_FAILURE status=failed reason=manager_log_write continuing_boot=y\n' >&2 + else + manager_log_line "SHIM_AUTO_READY status=ready continuing_boot=y" || \ + printf 'SHIM_AUTO_FAILURE status=failed reason=manager_log_write continuing_boot=y\n' >&2 + fi exit 0 } -# --- Dispatch --- +# --- Internal deterministic-test action ------------------------------------ +do_rotate_only() { + acquire_lifecycle_lock || return 1 + rotate_log_if_needed + local rc=$? + release_lifecycle_lock + return "$rc" +} + +# --- Dispatch --------------------------------------------------------------- +print_usage() { + printf 'Usage: %s {--auto|--start|--restart|--stop|--status|--help}\n' \ + "$(basename "${BASH_SOURCE[0]}")" +} + ACTION="${1:-}" case "$ACTION" in __supervise) + load_expected_contract || exit 1 run_supervisor ;; + __rotate_logs) + do_rotate_only + ;; --auto) do_auto ;; --start) - do_start + do_start explicit + ;; + --restart) + do_restart ;; --stop) do_stop @@ -377,8 +1106,11 @@ case "$ACTION" in --status) do_status ;; + --help|-h) + print_usage + ;; *) - echo "Usage: $(basename "${BASH_SOURCE[0]}") {--auto|--start|--stop|--status}" >&2 + print_usage >&2 exit 1 ;; esac diff --git a/scripts/smoke_tests/smoke_imports.py b/scripts/smoke_tests/smoke_imports.py index a536a7b1..58d38e86 100644 --- a/scripts/smoke_tests/smoke_imports.py +++ b/scripts/smoke_tests/smoke_imports.py @@ -1,21 +1,28 @@ # scripts/smoke_tests/smoke_imports.py -# Import-smoke test: verifies every pinned Python analysis package from the -# Dockerfile pip install blocks imports cleanly and reports its version. +# Package smoke test: verifies all 52 explicitly pinned Python distributions from +# the Dockerfile pip install blocks. Fifty import targets must import cleanly and +# report their installed versions; two distributions without supported import +# targets are checked through package metadata only. # Sequential inline script (DAAF code style): no functions, section separators, -# print + assert for validation. Exits nonzero on any import failure. +# print + assert for validation. Exits nonzero on any failed check. # -# Package list is derived from the five `uv pip install --system` blocks in -# /daaf/Dockerfile (core data science, econometrics, geospatial, visualization, -# ML interpretation/fairness). When the Dockerfile pins change, update this list -# to match. Entries are (import_name, pip_name) because several import names -# differ from their pip package name (e.g. scikit-learn -> sklearn). +# The inventory is derived from every explicit Python pin in the +# `uv pip install --system` blocks in /daaf/Dockerfile (analysis packages plus +# direct provider-shim runtime dependencies). When the Dockerfile pins change, +# update both inventories to match. Import entries are (import_name, pip_name) +# because several import names differ from their pip package name (e.g. +# scikit-learn -> sklearn). The provider shim's direct runtime dependencies carry +# exact version assertions, as do the two metadata-only distributions, making +# those pins image-identity gates rather than only presence checks. # --- Config --- import importlib +import importlib.metadata import sys -# (import_name, pip_name) for every pinned package in the Dockerfile pip blocks. -# pip_name is only used for reporting so a failure names the installable package. +# (import_name, pip_name) for the 50 pins with supported import targets. +# pip_name is used for metadata lookup and reporting so failures name the +# installable distribution. packages = [ # Block 1: core data science ("numpy", "numpy"), @@ -68,9 +75,33 @@ ("shap", "shap"), ("fairlearn", "fairlearn"), ("lightgbm", "lightgbm"), + # Block 6: network analysis + ("igraph", "igraph"), + # Block 7: synthetic data generation + ("faker", "faker"), + # Late direct provider-shim runtime dependency layer + ("httpx", "httpx"), + ("uvicorn", "uvicorn"), ] -print(f"--- Python import smoke: {len(packages)} pinned packages ---") +required_versions = { + "httpx": "0.28.1", + "uvicorn": "0.51.0", +} + +# These distributions intentionally have no fabricated import target. Their +# installed presence and exact pins are validated through distribution metadata. +metadata_only_distributions = { + "svy-rs": "0.10.0", + "svy-io": "0.1.1", +} + +explicit_distribution_total = len(packages) + len(metadata_only_distributions) +print( + f"--- Python package smoke: {len(packages)} import targets + " + f"{len(metadata_only_distributions)} metadata-only distributions = " + f"{explicit_distribution_total} explicit pinned distributions ---" +) # --- Import & Report --- # INTENT: import each package and capture its version so a version-visible @@ -83,22 +114,145 @@ for import_name, pip_name in packages: try: mod = importlib.import_module(import_name) - version = getattr(mod, "__version__", "no __version__ attr") - print(f"OK {pip_name:<16} (import {import_name}) version {version}") + version = importlib.metadata.version(pip_name) + expected_version = required_versions.get(pip_name) + if expected_version is not None and version != expected_version: + raise AssertionError( + f"expected pinned version {expected_version}, found {version}" + ) + pin_note = f" [pinned={expected_version}]" if expected_version else "" + print( + f"OK {pip_name:<16} (import {import_name}) " + f"version {version}{pin_note}" + ) except Exception as exc: # noqa: BLE001 -- report any import-time failure failures.append((pip_name, import_name, repr(exc))) print(f"FAIL {pip_name:<16} (import {import_name}) -> {exc!r}") +# INTENT: verify presence and exact versions for distributions that do not expose +# supported import targets. +# REASONING: inventing module names would turn an image-inventory check into a +# false import contract; distribution metadata verifies the pins directly. +# ASSUMES: importlib.metadata.version() raises PackageNotFoundError when an +# expected distribution is absent. +metadata_failures = [] +for pip_name, expected_version in metadata_only_distributions.items(): + try: + version = importlib.metadata.version(pip_name) + if version != expected_version: + raise AssertionError( + f"expected pinned version {expected_version}, found {version}" + ) + print( + f"OK {pip_name:<16} (metadata only) " + f"version {version} [pinned={expected_version}]" + ) + except Exception as exc: # noqa: BLE001 -- report every metadata failure + metadata_failures.append((pip_name, repr(exc))) + print(f"FAIL {pip_name:<16} (metadata only) -> {exc!r}") + # --- Validate --- -# Every listed package must import. Report the full failure list before asserting -# so the execution log names all broken packages, not just the first. +# Every import target and metadata-only distribution must pass. Report complete +# failure lists before asserting so one broken distribution does not mask others. if failures: print(f"--- {len(failures)} package(s) failed to import ---") for pip_name, import_name, err in failures: print(f" - {pip_name} (import {import_name}): {err}") +if metadata_failures: + print( + f"--- {len(metadata_failures)} metadata-only distribution(s) failed ---" + ) + for pip_name, err in metadata_failures: + print(f" - {pip_name} (metadata only): {err}") + assert not failures, f"{len(failures)} pinned package(s) failed to import" +assert not metadata_failures, ( + f"{len(metadata_failures)} metadata-only distribution(s) failed" +) # --- Summary --- -print(f"All {len(packages)} pinned Python packages imported successfully.") +print(f"All {len(packages)} import targets imported successfully.") +print( + f"All {len(metadata_only_distributions)} metadata-only distributions " + "were present at their exact pinned versions." +) +print( + f"PASS: {len(packages)} import targets + " + f"{len(metadata_only_distributions)} metadata-only distributions = " + f"{explicit_distribution_total} explicit pinned distributions verified." +) sys.exit(0) + + +# ============================================================================= +# EXECUTION LOG +# ============================================================================= +# +# Executed: 2026-07-18 23:56:14 +# Command: python3 /daaf/scripts/smoke_tests/smoke_imports.py +# Duration: 18s +# Exit code: 0 +# +# --- STDOUT --- +# --- Python package smoke: 50 import targets + 2 metadata-only distributions = 52 explicit pinned distributions --- +# OK numpy (import numpy) version 2.4.2 +# OK pandas (import pandas) version 3.0.0 +# OK polars (import polars) version 1.39.3 +# OK scipy (import scipy) version 1.17.0 +# OK openpyxl (import openpyxl) version 3.1.5 +# OK fastexcel (import fastexcel) version 0.19.0 +# OK xlrd (import xlrd) version 2.0.2 +# OK requests (import requests) version 2.32.5 +# OK pyarrow (import pyarrow) version 23.0.0 +# OK urllib3 (import urllib3) version 2.6.3 +# OK pre-commit (import pre_commit) version 4.5.1 +# OK scikit-learn (import sklearn) version 1.8.0 +# OK umap-learn (import umap) version 0.5.11 +# OK pyyaml (import yaml) version 6.0.3 +# OK statsmodels (import statsmodels) version 0.14.6 +# OK pyfixest (import pyfixest) version 0.40.0 +# OK tabulate (import tabulate) version 0.10.0 +# OK great-tables (import great_tables) version 0.21.0 +# OK wildboottest (import wildboottest) version 0.3.2 +# OK linearmodels (import linearmodels) version 7.0 +# OK rdrobust (import rdrobust) version 1.3.0 +# OK marginaleffects (import marginaleffects) version 0.5.0 +# OK arch (import arch) version 8.0.0 +# OK pydynpd (import pydynpd) version 0.2.1 +# OK svy (import svy) version 0.19.0 +# OK geopandas (import geopandas) version 1.1.3 +# OK rasterio (import rasterio) version 1.5.0 +# OK xarray (import xarray) version 2026.2.0 +# OK rioxarray (import rioxarray) version 0.22.0 +# OK contextily (import contextily) version 1.7.0 +# OK folium (import folium) version 0.20.0 +# OK libpysal (import libpysal) version 4.14.1 +# OK esda (import esda) version 2.9.0 +# OK spreg (import spreg) version 1.9.0 +# OK mapclassify (import mapclassify) version 2.10.0 +# OK rasterstats (import rasterstats) version 0.20.0 +# OK geopy (import geopy) version 2.4.1 +# OK osmnx (import osmnx) version 2.1.0 +# OK matplotlib (import matplotlib) version 3.10.8 +# OK seaborn (import seaborn) version 0.13.2 +# OK plotnine (import plotnine) version 0.15.3 +# OK plotly (import plotly) version 6.5.2 +# OK marimo (import marimo) version 0.19.11 +# OK shap (import shap) version 0.51.0 +# OK fairlearn (import fairlearn) version 0.12.0 +# OK lightgbm (import lightgbm) version 4.6.0 +# OK igraph (import igraph) version 1.0.0 +# OK faker (import faker) version 40.31.0 +# OK httpx (import httpx) version 0.28.1 [pinned=0.28.1] +# OK uvicorn (import uvicorn) version 0.51.0 [pinned=0.51.0] +# OK svy-rs (metadata only) version 0.10.0 [pinned=0.10.0] +# OK svy-io (metadata only) version 0.1.1 [pinned=0.1.1] +# All 50 import targets imported successfully. +# All 2 metadata-only distributions were present at their exact pinned versions. +# PASS: 50 import targets + 2 metadata-only distributions = 52 explicit pinned distributions verified. +# +# --- STDERR --- +# (captured in STDOUT above via 2>&1) +# +# ============================================================================= diff --git a/scripts/smoke_tests/smoke_imports_a.py b/scripts/smoke_tests/smoke_imports_a.py new file mode 100644 index 00000000..7720f774 --- /dev/null +++ b/scripts/smoke_tests/smoke_imports_a.py @@ -0,0 +1,258 @@ +# scripts/smoke_tests/smoke_imports.py +# Package smoke test: verifies all 52 explicitly pinned Python distributions from +# the Dockerfile pip install blocks. Fifty import targets must import cleanly and +# report their installed versions; two distributions without supported import +# targets are checked through package metadata only. +# Sequential inline script (DAAF code style): no functions, section separators, +# print + assert for validation. Exits nonzero on any failed check. +# +# The inventory is derived from every explicit Python pin in the +# `uv pip install --system` blocks in /daaf/Dockerfile (analysis packages plus +# direct provider-shim runtime dependencies). When the Dockerfile pins change, +# update both inventories to match. Import entries are (import_name, pip_name) +# because several import names differ from their pip package name (e.g. +# scikit-learn -> sklearn). The provider shim's direct runtime dependencies carry +# exact version assertions, as do the two metadata-only distributions, making +# those pins image-identity gates rather than only presence checks. + +# --- Config --- +import importlib +import importlib.metadata +import sys + +# (import_name, pip_name) for the 50 pins with supported import targets. +# pip_name is used for metadata lookup and reporting so failures name the +# installable distribution. +packages = [ + # Block 1: core data science + ("numpy", "numpy"), + ("pandas", "pandas"), + ("polars", "polars"), + ("scipy", "scipy"), + ("openpyxl", "openpyxl"), + ("fastexcel", "fastexcel"), + ("xlrd", "xlrd"), + ("requests", "requests"), + ("pyarrow", "pyarrow"), + ("urllib3", "urllib3"), + ("pre_commit", "pre-commit"), + ("sklearn", "scikit-learn"), + ("umap", "umap-learn"), + ("yaml", "pyyaml"), + ("statsmodels", "statsmodels"), + ("pyfixest", "pyfixest"), + ("tabulate", "tabulate"), + ("great_tables", "great-tables"), + ("wildboottest", "wildboottest"), + # Block 2: econometrics & statistical modeling + ("linearmodels", "linearmodels"), + ("rdrobust", "rdrobust"), + ("marginaleffects", "marginaleffects"), + ("arch", "arch"), + ("pydynpd", "pydynpd"), + ("svy", "svy"), + # Block 3: geospatial + ("geopandas", "geopandas"), + ("rasterio", "rasterio"), + ("xarray", "xarray"), + ("rioxarray", "rioxarray"), + ("contextily", "contextily"), + ("folium", "folium"), + ("libpysal", "libpysal"), + ("esda", "esda"), + ("spreg", "spreg"), + ("mapclassify", "mapclassify"), + ("rasterstats", "rasterstats"), + ("geopy", "geopy"), + ("osmnx", "osmnx"), + # Block 4: visualization + ("matplotlib", "matplotlib"), + ("seaborn", "seaborn"), + ("plotnine", "plotnine"), + ("plotly", "plotly"), + ("marimo", "marimo"), + # Block 5: ML interpretation & fairness + ("shap", "shap"), + ("fairlearn", "fairlearn"), + ("lightgbm", "lightgbm"), + # Block 6: network analysis + ("igraph", "igraph"), + # Block 7: synthetic data generation + ("faker", "faker"), + # Late direct provider-shim runtime dependency layer + ("httpx", "httpx"), + ("uvicorn", "uvicorn"), +] + +required_versions = { + "httpx": "0.28.1", + "uvicorn": "0.51.0", +} + +# These distributions intentionally have no fabricated import target. Their +# installed presence and exact pins are validated through distribution metadata. +metadata_only_distributions = { + "svy-rs": "0.10.0", + "svy-io": "0.1.1", +} + +explicit_distribution_total = len(packages) + len(metadata_only_distributions) +print( + f"--- Python package smoke: {len(packages)} import targets + " + f"{len(metadata_only_distributions)} metadata-only distributions = " + f"{explicit_distribution_total} explicit pinned distributions ---" +) + +# --- Import & Report --- +# INTENT: import each package and capture its version so a version-visible +# record lands in the appended execution log. +# REASONING: a bare import check hides which build was tested; printing the +# version makes the smoke result auditable against the Dockerfile pins. +# ASSUMES: import failures raise ImportError/ModuleNotFoundError, which we +# collect rather than let abort the loop, so one failure does not mask others. +failures = [] +for import_name, pip_name in packages: + try: + mod = importlib.import_module(import_name) + version = importlib.metadata.version(pip_name) + expected_version = required_versions.get(pip_name) + if expected_version is not None and version != expected_version: + raise AssertionError( + f"expected pinned version {expected_version}, found {version}" + ) + pin_note = f" [pinned={expected_version}]" if expected_version else "" + print( + f"OK {pip_name:<16} (import {import_name}) " + f"version {version}{pin_note}" + ) + except Exception as exc: # noqa: BLE001 -- report any import-time failure + failures.append((pip_name, import_name, repr(exc))) + print(f"FAIL {pip_name:<16} (import {import_name}) -> {exc!r}") + +# INTENT: verify presence and exact versions for distributions that do not expose +# supported import targets. +# REASONING: inventing module names would turn an image-inventory check into a +# false import contract; distribution metadata verifies the pins directly. +# ASSUMES: importlib.metadata.version() raises PackageNotFoundError when an +# expected distribution is absent. +metadata_failures = [] +for pip_name, expected_version in metadata_only_distributions.items(): + try: + version = importlib.metadata.version(pip_name) + if version != expected_version: + raise AssertionError( + f"expected pinned version {expected_version}, found {version}" + ) + print( + f"OK {pip_name:<16} (metadata only) " + f"version {version} [pinned={expected_version}]" + ) + except Exception as exc: # noqa: BLE001 -- report every metadata failure + metadata_failures.append((pip_name, repr(exc))) + print(f"FAIL {pip_name:<16} (metadata only) -> {exc!r}") + +# --- Validate --- +# Every import target and metadata-only distribution must pass. Report complete +# failure lists before asserting so one broken distribution does not mask others. +if failures: + print(f"--- {len(failures)} package(s) failed to import ---") + for pip_name, import_name, err in failures: + print(f" - {pip_name} (import {import_name}): {err}") + +if metadata_failures: + print( + f"--- {len(metadata_failures)} metadata-only distribution(s) failed ---" + ) + for pip_name, err in metadata_failures: + print(f" - {pip_name} (metadata only): {err}") + +assert not failures, f"{len(failures)} pinned package(s) failed to import" +assert not metadata_failures, ( + f"{len(metadata_failures)} metadata-only distribution(s) failed" +) + +# --- Summary --- +print(f"All {len(packages)} import targets imported successfully.") +print( + f"All {len(metadata_only_distributions)} metadata-only distributions " + "were present at their exact pinned versions." +) +print( + f"PASS: {len(packages)} import targets + " + f"{len(metadata_only_distributions)} metadata-only distributions = " + f"{explicit_distribution_total} explicit pinned distributions verified." +) +sys.exit(0) + + +# ============================================================================= +# EXECUTION LOG +# ============================================================================= +# +# Executed: 2026-07-19 00:42:48 +# Command: python3 /daaf/scripts/smoke_tests/smoke_imports_a.py +# Duration: 11s +# Exit code: 0 +# +# --- STDOUT --- +# --- Python package smoke: 50 import targets + 2 metadata-only distributions = 52 explicit pinned distributions --- +# OK numpy (import numpy) version 2.4.2 +# OK pandas (import pandas) version 3.0.0 +# OK polars (import polars) version 1.39.3 +# OK scipy (import scipy) version 1.17.0 +# OK openpyxl (import openpyxl) version 3.1.5 +# OK fastexcel (import fastexcel) version 0.19.0 +# OK xlrd (import xlrd) version 2.0.2 +# OK requests (import requests) version 2.32.5 +# OK pyarrow (import pyarrow) version 23.0.0 +# OK urllib3 (import urllib3) version 2.6.3 +# OK pre-commit (import pre_commit) version 4.5.1 +# OK scikit-learn (import sklearn) version 1.8.0 +# OK umap-learn (import umap) version 0.5.11 +# OK pyyaml (import yaml) version 6.0.3 +# OK statsmodels (import statsmodels) version 0.14.6 +# OK pyfixest (import pyfixest) version 0.40.0 +# OK tabulate (import tabulate) version 0.10.0 +# OK great-tables (import great_tables) version 0.21.0 +# OK wildboottest (import wildboottest) version 0.3.2 +# OK linearmodels (import linearmodels) version 7.0 +# OK rdrobust (import rdrobust) version 1.3.0 +# OK marginaleffects (import marginaleffects) version 0.5.0 +# OK arch (import arch) version 8.0.0 +# OK pydynpd (import pydynpd) version 0.2.1 +# OK svy (import svy) version 0.19.0 +# OK geopandas (import geopandas) version 1.1.3 +# OK rasterio (import rasterio) version 1.5.0 +# OK xarray (import xarray) version 2026.2.0 +# OK rioxarray (import rioxarray) version 0.22.0 +# OK contextily (import contextily) version 1.7.0 +# OK folium (import folium) version 0.20.0 +# OK libpysal (import libpysal) version 4.14.1 +# OK esda (import esda) version 2.9.0 +# OK spreg (import spreg) version 1.9.0 +# OK mapclassify (import mapclassify) version 2.10.0 +# OK rasterstats (import rasterstats) version 0.20.0 +# OK geopy (import geopy) version 2.4.1 +# OK osmnx (import osmnx) version 2.1.0 +# OK matplotlib (import matplotlib) version 3.10.8 +# OK seaborn (import seaborn) version 0.13.2 +# OK plotnine (import plotnine) version 0.15.3 +# OK plotly (import plotly) version 6.5.2 +# OK marimo (import marimo) version 0.19.11 +# OK shap (import shap) version 0.51.0 +# OK fairlearn (import fairlearn) version 0.12.0 +# OK lightgbm (import lightgbm) version 4.6.0 +# OK igraph (import igraph) version 1.0.0 +# OK faker (import faker) version 40.31.0 +# OK httpx (import httpx) version 0.28.1 [pinned=0.28.1] +# OK uvicorn (import uvicorn) version 0.51.0 [pinned=0.51.0] +# OK svy-rs (metadata only) version 0.10.0 [pinned=0.10.0] +# OK svy-io (metadata only) version 0.1.1 [pinned=0.1.1] +# All 50 import targets imported successfully. +# All 2 metadata-only distributions were present at their exact pinned versions. +# PASS: 50 import targets + 2 metadata-only distributions = 52 explicit pinned distributions verified. +# +# --- STDERR --- +# (captured in STDOUT above via 2>&1) +# +# ============================================================================= diff --git a/tests/bash/start_shim.bats b/tests/bash/start_shim.bats new file mode 100644 index 00000000..5eb84ab7 --- /dev/null +++ b/tests/bash/start_shim.bats @@ -0,0 +1,744 @@ +#!/usr/bin/env bats +# ============================================================================= +# Deterministic lifecycle tests for scripts/provider_shim/start_shim.sh. +# +# Every test stages the manager and a fabricated loopback-only shim under +# BATS_TEST_TMPDIR. The production process, source, state files, and shim.log are +# never touched. teardown stops only the staged manager's identity-verified pids, +# removes any symlink fixture, and deletes the isolated directory. +# ============================================================================= + +load 'test_helper' + +START_SHIM_SH="${START_SHIM_SH:-${REPO_ROOT}/scripts/provider_shim/start_shim.sh}" + +setup() { + SHIM_TEST_ROOT="${BATS_TEST_TMPDIR}/provider_shim" + mkdir -p "$SHIM_TEST_ROOT" + cp "$START_SHIM_SH" "${SHIM_TEST_ROOT}/start_shim.sh" + chmod 0755 "${SHIM_TEST_ROOT}/start_shim.sh" + MANAGER="${SHIM_TEST_ROOT}/start_shim.sh" + LOG_DIR="${SHIM_TEST_ROOT}/logs" + LOG_FILE="${LOG_DIR}/shim.log" + + cat > "${SHIM_TEST_ROOT}/anthropic_openai_shim.py" <<'PY' +#!/usr/bin/env python3 +SHIM_VERSION = "1.2.12" + +import json +import os +import signal +import time +from http.server import BaseHTTPRequestHandler, HTTPServer + +behavior = os.environ.get("FAKE_SHIM_BEHAVIOR", "serve") +if behavior == "crash": + raise SystemExit(int(os.environ.get("FAKE_SHIM_CRASH_CODE", "42"))) +time.sleep(float(os.environ.get("FAKE_SHIM_START_DELAY", "0"))) + +stopping = False + + +def stop_server(_signum, _frame): + global stopping + stopping = True + + +class Handler(BaseHTTPRequestHandler): + def do_GET(self): + if self.path != "/health": + self.send_response(404) + self.end_headers() + return + raw = os.environ.get("FAKE_HEALTH_RAW") + if raw is None: + mode = os.environ.get("FAKE_HEALTH_BACKEND_MODE", os.environ.get("SHIM_BACKEND_MODE", "openai")) + payload = { + "service": os.environ.get("FAKE_HEALTH_SERVICE", "daaf-anthropic-openai-shim"), + "status": os.environ.get("FAKE_HEALTH_STATUS", "ok"), + "version": os.environ.get("FAKE_HEALTH_VERSION", SHIM_VERSION), + "backend_mode": mode, + "backend": "https://example.invalid/v1", + "codex_home_present": True, + "sanitize_tools": True, + "reasoning_effort": None, + "text_verbosity": "high", + } + raw = json.dumps(payload) + body = raw.encode("utf-8") + self.send_response(int(os.environ.get("FAKE_HEALTH_HTTP_STATUS", "200"))) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, _format, *_args): + return + + +signal.signal(signal.SIGTERM, stop_server) +signal.signal(signal.SIGINT, stop_server) +server = HTTPServer(("127.0.0.1", int(os.environ.get("SHIM_PORT", "4141"))), Handler) +server.timeout = 0.05 +while not stopping: + server.handle_request() +server.server_close() +PY + chmod 0755 "${SHIM_TEST_ROOT}/anthropic_openai_shim.py" + + SHIM_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')" + export SHIM_PORT + export DAAF_SHIM_TEST_MODE=1 + export DAAF_SHIM_TEST_READINESS_WAIT=5 + export DAAF_SHIM_TEST_RESTART_DELAY=0.2 + export DAAF_SHIM_TEST_STORM_LIMIT=4 + export DAAF_SHIM_TEST_STORM_WINDOW=60 + export START_SHIM_SH MANAGER LOG_DIR LOG_FILE + unset FAKE_SHIM_BEHAVIOR FAKE_SHIM_CRASH_CODE FAKE_SHIM_START_DELAY FAKE_HEALTH_RAW + unset FAKE_HEALTH_SERVICE FAKE_HEALTH_STATUS FAKE_HEALTH_VERSION + unset FAKE_HEALTH_BACKEND_MODE FAKE_HEALTH_HTTP_STATUS DAAF_SHIM_TEST_NO_SETSID + unset DAAF_SHIM_TEST_STOP_FAILURE DAAF_SHIM_TEST_RESULT_APPEND_FAILURE + unset DAAF_SHIM_TEST_SIGNAL_RESULT_IN_PROGRESS SHIM_BACKEND_MODE +} + +teardown() { + unset DAAF_SHIM_TEST_STOP_FAILURE DAAF_SHIM_TEST_RESULT_APPEND_FAILURE + unset DAAF_SHIM_TEST_SIGNAL_RESULT_IN_PROGRESS + if [ -x "${MANAGER:-}" ]; then + "$MANAGER" --stop >/dev/null 2>&1 || true + fi + if [ -n "${EXTERNAL_SERVER_PID:-}" ]; then + kill "$EXTERNAL_SERVER_PID" 2>/dev/null || true + wait "$EXTERNAL_SERVER_PID" 2>/dev/null || true + fi + if [ -n "${VICTIM_PID:-}" ]; then + kill "$VICTIM_PID" 2>/dev/null || true + wait "$VICTIM_PID" 2>/dev/null || true + fi + if [ -n "${DIRECT_SUPERVISOR_PID:-}" ]; then + kill "$DIRECT_SUPERVISOR_PID" 2>/dev/null || true + wait "$DIRECT_SUPERVISOR_PID" 2>/dev/null || true + fi + find "${SHIM_TEST_ROOT:-${BATS_TEST_TMPDIR}}" -type l -delete 2>/dev/null || true + rm -rf "${SHIM_TEST_ROOT:-${BATS_TEST_TMPDIR}/provider_shim}" +} + +wait_for_file() { + local path="$1" attempts=0 + while [ "$attempts" -lt 100 ]; do + [ -s "$path" ] && return 0 + sleep 0.05 + attempts=$((attempts + 1)) + done + return 1 +} + +wait_for_log_text() { + local text="$1" attempts=0 + while [ "$attempts" -lt 100 ]; do + if [ -f "$LOG_FILE" ] && grep -F "$text" "$LOG_FILE" >/dev/null 2>&1; then + return 0 + fi + sleep 0.05 + attempts=$((attempts + 1)) + done + return 1 +} + +wait_for_dead() { + local pid="$1" attempts=0 + while [ "$attempts" -lt 100 ]; do + kill -0 "$pid" 2>/dev/null || return 0 + sleep 0.05 + attempts=$((attempts + 1)) + done + return 1 +} + +wait_for_ready() { + local attempts=0 + while [ "$attempts" -lt 100 ]; do + if curl -fsS --max-time 1 "http://127.0.0.1:${SHIM_PORT}/health" 2>/dev/null | \ + jq -e '.service == "daaf-anthropic-openai-shim" and .status == "ok"' \ + >/dev/null 2>&1; then + return 0 + fi + sleep 0.05 + attempts=$((attempts + 1)) + done + return 1 +} + +count_exact_arg_processes() { + local expected="$1" path arg found count=0 + for path in /proc/[0-9]*/cmdline; do + [ -r "$path" ] || continue + found=0 + while IFS= read -r -d '' arg; do + if [ "$arg" = "$expected" ]; then + found=1 + break + fi + done < "$path" + [ "$found" -eq 0 ] || count=$((count + 1)) + done + printf '%s\n' "$count" +} + +assert_manager_success() { + if [ "$status" -ne 0 ] && [ -f "$LOG_FILE" ]; then + printf '%s\n' '--- isolated lifecycle log ---' >&3 + while IFS= read -r line; do + printf '%s\n' "$line" >&3 + done < "$LOG_FILE" + fi + assert_success +} + +start_external_fixture() { + python3 "${SHIM_TEST_ROOT}/anthropic_openai_shim.py" >/dev/null 2>&1 & + EXTERNAL_SERVER_PID=$! + export EXTERNAL_SERVER_PID + local attempts=0 http_status + while [ "$attempts" -lt 100 ]; do + http_status="$(curl -sS --max-time 1 --output /dev/null --write-out '%{http_code}' \ + "http://127.0.0.1:${SHIM_PORT}/health" 2>/dev/null)" || http_status="000" + case "$http_status" in + [0-9][0-9][0-9]) [ "$http_status" = "000" ] || return 0 ;; + esac + sleep 0.05 + attempts=$((attempts + 1)) + done + return 1 +} + +# --- Syntax and deterministic rotation -------------------------------------- + +@test "start_shim.sh parses without errors" { + run bash -n "$MANAGER" + assert_success +} + +@test "usage and help recognize restart as a public action" { + run "$MANAGER" --help + assert_success + assert_output --partial "--restart" + run "$MANAGER" --unknown + assert_failure + assert_output --partial "--restart" +} + +@test "missing log is created private without rotation" { + run "$MANAGER" __rotate_logs + assert_success + [ -f "$LOG_FILE" ] + [ "$(stat -c '%a' "$LOG_FILE")" = "600" ] + [ ! -e "${LOG_FILE}.1" ] +} + +@test "existing log at exact 25 MiB boundary does not rotate" { + mkdir -p "$LOG_DIR" + truncate -s 26214400 "$LOG_FILE" + run "$MANAGER" __rotate_logs + assert_success + [ "$(wc -c < "$LOG_FILE")" -eq 26214400 ] + [ ! -e "${LOG_FILE}.1" ] + [ "$(stat -c '%a' "$LOG_FILE")" = "600" ] +} + +@test "log one byte over boundary rotates before creating private replacement" { + mkdir -p "$LOG_DIR" + truncate -s 26214401 "$LOG_FILE" + chmod 0644 "$LOG_FILE" + run "$MANAGER" __rotate_logs + assert_success + [ "$(wc -c < "$LOG_FILE")" -eq 0 ] + [ "$(wc -c < "${LOG_FILE}.1")" -eq 26214401 ] + [ "$(stat -c '%a' "$LOG_FILE")" = "600" ] + [ "$(stat -c '%a' "${LOG_FILE}.1")" = "600" ] +} + +@test "over-boundary start rotates before preserving the supervisor first record" { + mkdir -p "$LOG_DIR" + truncate -s 26214401 "$LOG_FILE" + run "$MANAGER" --start + assert_manager_success + grep -F "SUPERVISOR starting shim" "$LOG_FILE" + [ "$(wc -c < "${LOG_FILE}.1")" -eq 26214401 ] + run "$MANAGER" --stop + assert_success +} + +@test "rotation retains exactly five descending generations" { + mkdir -p "$LOG_DIR" + truncate -s 26214401 "$LOG_FILE" + printf 'old-1\n' > "${LOG_FILE}.1" + printf 'old-2\n' > "${LOG_FILE}.2" + printf 'old-3\n' > "${LOG_FILE}.3" + printf 'old-4\n' > "${LOG_FILE}.4" + printf 'old-5-discard\n' > "${LOG_FILE}.5" + run "$MANAGER" __rotate_logs + assert_success + grep -Fx 'old-1' "${LOG_FILE}.2" + grep -Fx 'old-2' "${LOG_FILE}.3" + grep -Fx 'old-3' "${LOG_FILE}.4" + grep -Fx 'old-4' "${LOG_FILE}.5" + ! grep -R -F 'old-5-discard' "$LOG_DIR" >/dev/null 2>&1 + [ ! -e "${LOG_FILE}.6" ] +} + +@test "unsafe symlink log target is rejected without touching its referent" { + mkdir -p "$LOG_DIR" + local referent="${SHIM_TEST_ROOT}/outside.log" + printf 'referent-unchanged\n' > "$referent" + ln -s "$referent" "$LOG_FILE" + run "$MANAGER" __rotate_logs + assert_failure + assert_output --partial "refusing unsafe shim state target" + grep -Fx 'referent-unchanged' "$referent" + [ -L "$LOG_FILE" ] +} + +@test "unsafe symlink generation is rejected before current log mutation" { + mkdir -p "$LOG_DIR" + truncate -s 26214401 "$LOG_FILE" + local referent="${SHIM_TEST_ROOT}/generation-referent.log" + printf 'generation-referent\n' > "$referent" + ln -s "$referent" "${LOG_FILE}.3" + run "$MANAGER" __rotate_logs + assert_failure + [ "$(wc -c < "$LOG_FILE")" -eq 26214401 ] + grep -Fx 'generation-referent' "$referent" +} + +# --- Strict readiness ------------------------------------------------------- + +@test "cleanly stopped status exits zero without launching a process" { + run "$MANAGER" --status + assert_success + assert_output "STATUS: stopped" + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 0 ] + [ "$(count_exact_arg_processes "$MANAGER")" -eq 0 ] + [ ! -e "${LOG_DIR}/supervisor.pid" ] + [ ! -e "${LOG_DIR}/shim.pid" ] +} + +@test "strict readiness accepts exact identity version normalized mode and status" { + export SHIM_BACKEND_MODE=' ChatGPT ' + export FAKE_HEALTH_BACKEND_MODE=chatgpt + run "$MANAGER" --start + assert_manager_success + assert_output --partial "strictly ready" + run "$MANAGER" --status + assert_success + assert_output --partial '"service": "daaf-anthropic-openai-shim"' + assert_output --partial '"version": "1.2.12"' + assert_output --partial '"backend_mode": "chatgpt"' +} + +@test "non-200 health response is reachable unexpected and blocks manager launch" { + export FAKE_HEALTH_HTTP_STATUS=404 + start_external_fixture + + run "$MANAGER" --start + + assert_failure + assert_output --partial "port ${SHIM_PORT} answered /health" + assert_output --partial "reason=http_status_404" + refute_output --partial "reason=unreachable" + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 1 ] + [ "$(count_exact_arg_processes "$MANAGER")" -eq 0 ] + [ ! -e "${LOG_DIR}/supervisor.pid" ] + [ ! -e "${LOG_DIR}/shim.pid" ] +} + +@test "malformed JSON wrong identity wrong version and wrong mode are rejected" { + local name + for name in malformed identity version mode; do + unset FAKE_HEALTH_RAW FAKE_HEALTH_SERVICE FAKE_HEALTH_VERSION FAKE_HEALTH_BACKEND_MODE + case "$name" in + malformed) export FAKE_HEALTH_RAW='{not-json' ;; + identity) export FAKE_HEALTH_SERVICE='unrelated-service' ;; + version) export FAKE_HEALTH_VERSION='1.2.11' ;; + mode) export FAKE_HEALTH_BACKEND_MODE='chatgpt' ;; + esac + start_external_fixture + run "$MANAGER" --start + assert_failure + assert_output --partial "SHIM_READINESS_FAILURE status=failed" + case "$name" in + malformed) assert_output --partial "reason=malformed_json" ;; + identity) assert_output --partial "reason=wrong_identity_or_status" ;; + version) assert_output --partial "reason=version_mismatch" ;; + mode) assert_output --partial "reason=backend_mode_mismatch" ;; + esac + kill "$EXTERNAL_SERVER_PID" 2>/dev/null || true + wait "$EXTERNAL_SERVER_PID" 2>/dev/null || true + EXTERNAL_SERVER_PID="" + done +} + +@test "auto remains boot-nonfatal but emits machine-readable readiness failure" { + export DAAF_PROVIDER_SHIM=openai + export FAKE_HEALTH_SERVICE=unrelated-service + start_external_fixture + run "$MANAGER" --auto + assert_success + assert_output --partial "SHIM_READINESS_FAILURE status=failed" + assert_output --partial "SHIM_AUTO_FAILURE status=failed" + [ "$(grep -c 'MANAGER SHIM_AUTO_FAILURE status=failed' "$LOG_FILE")" -eq 1 ] +} + +@test "auto remains nonblocking while a restart owns the lifecycle lock" { + run "$MANAGER" --start + assert_manager_success + export DAAF_PROVIDER_SHIM=openai + export FAKE_SHIM_START_DELAY=5 + export DAAF_SHIM_TEST_READINESS_WAIT=10 + local restart_pid before after + + "$MANAGER" --restart > "${SHIM_TEST_ROOT}/restart-for-auto.out" 2>&1 & + restart_pid=$! + wait_for_log_text "MANAGER SHIM_RESTART_BEGIN status=begin" + before="$(date +%s)" + run "$MANAGER" --auto + after="$(date +%s)" + + assert_success + assert_output --partial "SHIM_AUTO_FAILURE status=failed" + [ $((after - before)) -lt 3 ] + wait "$restart_pid" + wait_for_ready +} + +# --- Process identity, idempotency, restart, and stop ----------------------- + +@test "restart replaces a running-ready generation and passes strict readiness" { + run "$MANAGER" --start + assert_manager_success + local old_sp old_pp new_sp new_pp + old_sp="$(< "${LOG_DIR}/supervisor.pid")" + old_pp="$(< "${LOG_DIR}/shim.pid")" + + run "$MANAGER" --restart + assert_manager_success + assert_output --partial "SHIM_RESTART_BEGIN status=begin prior=running_ready" + assert_output --partial "SHIM_RESTART_RESULT status=ready stage=readiness reason=ready exit_code=0" + assert_output --partial "Shim restarted and strictly ready" + new_sp="$(< "${LOG_DIR}/supervisor.pid")" + new_pp="$(< "${LOG_DIR}/shim.pid")" + [ "$old_sp" != "$new_sp" ] + [ "$old_pp" != "$new_pp" ] + wait_for_dead "$old_sp" + wait_for_dead "$old_pp" + wait_for_ready + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 1 ] +} + +@test "restart starts a stopped daemon and passes strict readiness" { + run "$MANAGER" --restart + assert_manager_success + assert_output --partial "SHIM_RESTART_BEGIN status=begin prior=stopped" + assert_output --partial "SHIM_RESTART_RESULT status=ready" + wait_for_ready + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 1 ] +} + +@test "restart recovers stale state without signaling an unrelated process" { + mkdir -p "$LOG_DIR" + sleep 30 & + VICTIM_PID=$! + export VICTIM_PID + printf '%s\n' "$VICTIM_PID" > "${LOG_DIR}/shim.pid" + printf '%s\n' "$VICTIM_PID" > "${LOG_DIR}/supervisor.pid" + printf '%s\n' "$VICTIM_PID" > "${LOG_DIR}/pgid" + + run "$MANAGER" --restart + assert_manager_success + kill -0 "$VICTIM_PID" + wait_for_ready + [ "$(< "${LOG_DIR}/supervisor.pid")" != "$VICTIM_PID" ] + [ "$(< "${LOG_DIR}/shim.pid")" != "$VICTIM_PID" ] +} + +@test "restart stop failure is explicit and never emits false readiness" { + run "$MANAGER" --start + assert_manager_success + local old_sp old_pp + old_sp="$(< "${LOG_DIR}/supervisor.pid")" + old_pp="$(< "${LOG_DIR}/shim.pid")" + export DAAF_SHIM_TEST_STOP_FAILURE=1 + + run "$MANAGER" --restart + [ "$status" -eq 40 ] + assert_output --partial "SHIM_RESTART_RESULT status=failed stage=stop reason=termination_failed exit_code=40" + refute_output --partial "SHIM_RESTART_RESULT status=ready" + kill -0 "$old_sp" + kill -0 "$old_pp" + [ "$(grep -c 'MANAGER SHIM_RESTART_BEGIN status=begin' "$LOG_FILE")" -eq 1 ] + [ "$(grep -c 'MANAGER SHIM_RESTART_RESULT status=failed stage=stop' "$LOG_FILE")" -eq 1 ] + [ "$(grep -c 'MANAGER SHIM_RESTART_RESULT status=ready' "$LOG_FILE" || true)" -eq 0 ] +} + +@test "restart launch failure after stop is explicit and leaves no orphan" { + run "$MANAGER" --start + assert_manager_success + local old_sp old_pp + old_sp="$(< "${LOG_DIR}/supervisor.pid")" + old_pp="$(< "${LOG_DIR}/shim.pid")" + export FAKE_SHIM_BEHAVIOR=crash + export DAAF_SHIM_TEST_STORM_LIMIT=2 + export DAAF_SHIM_TEST_RESTART_DELAY=0.05 + export DAAF_SHIM_TEST_READINESS_WAIT=3 + + run "$MANAGER" --restart + [ "$status" -eq 41 ] + assert_output --partial "SHIM_RESTART_RESULT status=failed stage=launch reason=launch_failed exit_code=41" + refute_output --partial "SHIM_RESTART_RESULT status=ready" + wait_for_dead "$old_sp" + wait_for_dead "$old_pp" + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 0 ] + [ "$(count_exact_arg_processes "$MANAGER")" -eq 0 ] + [ ! -s "${LOG_DIR}/supervisor.pid" ] + [ ! -s "${LOG_DIR}/shim.pid" ] +} + +@test "restart readiness failure after stop is explicit and leaves no orphan" { + run "$MANAGER" --start + assert_manager_success + export FAKE_HEALTH_STATUS=starting + export DAAF_SHIM_TEST_READINESS_WAIT=3 + + run "$MANAGER" --restart + [ "$status" -eq 42 ] + assert_output --partial "SHIM_RESTART_RESULT status=failed stage=readiness reason=strict_readiness_failed exit_code=42" + refute_output --partial "SHIM_RESTART_RESULT status=ready" + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 0 ] + [ "$(count_exact_arg_processes "$MANAGER")" -eq 0 ] + [ ! -s "${LOG_DIR}/supervisor.pid" ] + [ ! -s "${LOG_DIR}/shim.pid" ] +} + +@test "restart result append failure exits 43 without a false persisted record" { + export DAAF_SHIM_TEST_RESULT_APPEND_FAILURE=1 + + run "$MANAGER" --restart + + [ "$status" -eq 43 ] + assert_output --partial "SHIM_RESTART_RESULT status=failed stage=record reason=manager_log_write exit_code=43" + assert_output --partial "record_persisted=no" + refute_output --partial "record_persisted=unknown" + [ "$(grep -c 'MANAGER SHIM_RESTART_BEGIN status=begin' "$LOG_FILE")" -eq 1 ] + [ "$(grep -c 'MANAGER SHIM_RESTART_RESULT' "$LOG_FILE" || true)" -eq 0 ] +} + +@test "signal during restart result persistence exits 130 without deadlock or orphan" { + export DAAF_SHIM_TEST_SIGNAL_RESULT_IN_PROGRESS=1 + + run timeout 15 "$MANAGER" --restart + + [ "$status" -eq 130 ] + assert_output --partial "SHIM_RESTART_RESULT status=failed stage=record reason=write_interrupted exit_code=130" + assert_output --partial "record_persisted=unknown" + refute_output --partial "record_persisted=no" + [ "$(grep -c 'MANAGER SHIM_RESTART_BEGIN status=begin' "$LOG_FILE")" -eq 1 ] + [ "$(grep -c 'MANAGER SHIM_RESTART_RESULT' "$LOG_FILE" || true)" -eq 0 ] + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 0 ] + [ "$(count_exact_arg_processes "$MANAGER")" -eq 0 ] + [ ! -s "${LOG_DIR}/supervisor.pid" ] + [ ! -s "${LOG_DIR}/shim.pid" ] +} + +@test "interrupted restart never emits success and leaves a truthful stopped state" { + run "$MANAGER" --start + assert_manager_success + local old_sp old_pp restart_pid restart_rc=0 + old_sp="$(< "${LOG_DIR}/supervisor.pid")" + old_pp="$(< "${LOG_DIR}/shim.pid")" + export FAKE_SHIM_START_DELAY=10 + export DAAF_SHIM_TEST_READINESS_WAIT=20 + + "$MANAGER" --restart > "${SHIM_TEST_ROOT}/restart-interrupt.out" 2>&1 & + restart_pid=$! + wait_for_log_text "MANAGER SHIM_RESTART_BEGIN status=begin" + wait_for_dead "$old_sp" + wait_for_dead "$old_pp" + kill -TERM "$restart_pid" + wait "$restart_pid" || restart_rc=$? + + [ "$restart_rc" -eq 130 ] + grep -F "SHIM_RESTART_RESULT status=failed stage=interrupt reason=signal_TERM exit_code=130" \ + "${SHIM_TEST_ROOT}/restart-interrupt.out" + ! grep -F "SHIM_RESTART_RESULT status=ready" "${SHIM_TEST_ROOT}/restart-interrupt.out" + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 0 ] + [ ! -s "${LOG_DIR}/supervisor.pid" ] + [ ! -s "${LOG_DIR}/shim.pid" ] +} + +@test "restart records are exactly once and supervisor records are not duplicated" { + run "$MANAGER" --start + assert_manager_success + run "$MANAGER" --restart + assert_manager_success + [ "$(grep -c 'MANAGER SHIM_RESTART_BEGIN status=begin' "$LOG_FILE")" -eq 1 ] + [ "$(grep -c 'MANAGER SHIM_RESTART_RESULT status=ready' "$LOG_FILE")" -eq 1 ] + [ "$(grep -c 'SUPERVISOR starting shim' "$LOG_FILE")" -eq 2 ] + [ "$(grep -c 'SUPERVISOR supervisor exiting' "$LOG_FILE")" -eq 1 ] +} + +@test "concurrent restart and start serialize without duplicate daemons" { + run "$MANAGER" --start + assert_manager_success + export FAKE_SHIM_START_DELAY=0.5 + local restart_pid start_pid + + "$MANAGER" --restart > "${SHIM_TEST_ROOT}/restart.out" 2>&1 & + restart_pid=$! + wait_for_log_text "MANAGER SHIM_RESTART_BEGIN status=begin" + "$MANAGER" --start > "${SHIM_TEST_ROOT}/start.out" 2>&1 & + start_pid=$! + wait "$restart_pid" + wait "$start_pid" + + wait_for_ready + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 1 ] + [ "$(grep -c 'MANAGER SHIM_RESTART_RESULT status=ready' "$LOG_FILE")" -eq 1 ] + grep -F "Shim already running and strictly ready" "${SHIM_TEST_ROOT}/start.out" +} + +@test "concurrent restarts serialize without duplicate daemons" { + run "$MANAGER" --start + assert_manager_success + export FAKE_SHIM_START_DELAY=0.5 + local first_pid second_pid + + "$MANAGER" --restart > "${SHIM_TEST_ROOT}/restart-first.out" 2>&1 & + first_pid=$! + wait_for_log_text "MANAGER SHIM_RESTART_BEGIN status=begin" + "$MANAGER" --restart > "${SHIM_TEST_ROOT}/restart-second.out" 2>&1 & + second_pid=$! + wait "$first_pid" + wait "$second_pid" + + wait_for_ready + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 1 ] + [ "$(grep -c 'MANAGER SHIM_RESTART_BEGIN status=begin' "$LOG_FILE")" -eq 2 ] + [ "$(grep -c 'MANAGER SHIM_RESTART_RESULT status=ready' "$LOG_FILE")" -eq 2 ] + [ "$(grep -c 'SUPERVISOR starting shim' "$LOG_FILE")" -eq 3 ] +} + +@test "stale PID and PGID files never signal an unrelated live process" { + mkdir -p "$LOG_DIR" + sleep 30 & + VICTIM_PID=$! + export VICTIM_PID + printf '%s\n' "$VICTIM_PID" > "${LOG_DIR}/shim.pid" + printf '%s\n' "$VICTIM_PID" > "${LOG_DIR}/supervisor.pid" + printf '%s\n' "$VICTIM_PID" > "${LOG_DIR}/pgid" + run "$MANAGER" --stop + assert_success + assert_output --partial "was not running" + kill -0 "$VICTIM_PID" + [ ! -e "${LOG_DIR}/shim.pid" ] + [ ! -e "${LOG_DIR}/supervisor.pid" ] + [ ! -e "${LOG_DIR}/pgid" ] +} + +@test "duplicate starts preserve one supervisor and one start record" { + run "$MANAGER" --start + assert_success + wait_for_file "${LOG_DIR}/supervisor.pid" + local first second + first="$(< "${LOG_DIR}/supervisor.pid")" + run "$MANAGER" --start + assert_success + second="$(< "${LOG_DIR}/supervisor.pid")" + [ "$first" = "$second" ] + [ "$(grep -c 'SUPERVISOR starting shim' "$LOG_FILE")" -eq 1 ] +} + +@test "supervisor records are single-write rather than tee-duplicated" { + run "$MANAGER" --start + assert_success + run "$MANAGER" --stop + assert_success + [ "$(grep -c 'SUPERVISOR starting shim' "$LOG_FILE")" -eq 1 ] + [ "$(grep -c 'SUPERVISOR supervisor exiting' "$LOG_FILE")" -eq 1 ] +} + +@test "crash storm stops after deterministic limit and leaves no live supervisor" { + export FAKE_SHIM_BEHAVIOR=crash + export FAKE_SHIM_CRASH_CODE=42 + export DAAF_SHIM_TEST_STORM_LIMIT=3 + export DAAF_SHIM_TEST_RESTART_DELAY=0.05 + export DAAF_SHIM_TEST_READINESS_WAIT=3 + run "$MANAGER" --start + assert_failure + wait_for_log_text "RESTART_STORM" + [ "$(grep -c 'shim exited rc=42' "$LOG_FILE")" -eq 3 ] + grep -F 'RESTART_STORM crashes=3' "$LOG_FILE" + [ ! -s "${LOG_DIR}/supervisor.pid" ] +} + +@test "stop interrupts supervisor restart sleep promptly" { + export FAKE_SHIM_BEHAVIOR=crash + export DAAF_SHIM_TEST_RESTART_DELAY=20 + mkdir -p "$LOG_DIR" + setsid "$MANAGER" __supervise >> "$LOG_FILE" 2>&1 & + DIRECT_SUPERVISOR_PID=$! + export DIRECT_SUPERVISOR_PID + wait_for_log_text "shim exited rc=42" + local before after + before="$(date +%s)" + run "$MANAGER" --stop + assert_success + after="$(date +%s)" + [ $((after - before)) -lt 5 ] + wait_for_dead "$DIRECT_SUPERVISOR_PID" + DIRECT_SUPERVISOR_PID="" +} + +@test "stop leaves no supervisor or shim child orphan" { + run "$MANAGER" --start + assert_success + wait_for_file "${LOG_DIR}/supervisor.pid" + wait_for_file "${LOG_DIR}/shim.pid" + local sp pp + sp="$(< "${LOG_DIR}/supervisor.pid")" + pp="$(< "${LOG_DIR}/shim.pid")" + run "$MANAGER" --stop + assert_success + wait_for_dead "$sp" + wait_for_dead "$pp" + [ ! -e "${LOG_DIR}/supervisor.pid" ] + [ ! -e "${LOG_DIR}/shim.pid" ] + [ ! -e "${LOG_DIR}/pgid" ] +} + +@test "no-setsid fallback restarts safely without a broad PGID" { + export DAAF_SHIM_TEST_NO_SETSID=1 + run "$MANAGER" --start + assert_success + wait_for_file "${LOG_DIR}/supervisor.pid" + wait_for_file "${LOG_DIR}/shim.pid" + local old_sp old_pp new_sp new_pp + old_sp="$(< "${LOG_DIR}/supervisor.pid")" + old_pp="$(< "${LOG_DIR}/shim.pid")" + [ ! -e "${LOG_DIR}/pgid" ] + + run "$MANAGER" --restart + assert_manager_success + new_sp="$(< "${LOG_DIR}/supervisor.pid")" + new_pp="$(< "${LOG_DIR}/shim.pid")" + [ "$old_sp" != "$new_sp" ] + [ "$old_pp" != "$new_pp" ] + [ ! -e "${LOG_DIR}/pgid" ] + wait_for_dead "$old_sp" + wait_for_dead "$old_pp" + + run "$MANAGER" --stop + assert_success + wait_for_dead "$new_sp" + wait_for_dead "$new_pp" + [ "$(grep -c 'process-group isolation unavailable' "$LOG_FILE")" -eq 2 ] +} diff --git a/tests/provider_shim/_loopback_harness.py b/tests/provider_shim/_loopback_harness.py index da05f970..4d165453 100644 --- a/tests/provider_shim/_loopback_harness.py +++ b/tests/provider_shim/_loopback_harness.py @@ -101,10 +101,14 @@ class Scenario: reject_nonstream: bool = False append_done: bool = True raw_stream_frames: Optional[list[bytes]] = None + preserve_raw_stream_chunks: bool = False abrupt_eof: bool = False stream_status: int = 200 stream_headers: dict[str, str] = field(default_factory=dict) stream_error_body: bytes = b'{"error":{"message":"fixture backend rejection"}}' + # Optional per-request JSON response sequence for stateful real-process tests. + # The final entry repeats when the request count exceeds the sequence length. + nonstream_responses: Optional[list[dict[str, Any]]] = None attempt_statuses: Optional[list[int | str]] = None attempt_headers: Optional[list[dict[str, str]]] = None disconnect_phase: Optional[str] = None @@ -653,8 +657,18 @@ def terminal_contract_scenario( ) -def raw_sse_scenario(name: str, frames: list[bytes]) -> Scenario: - """Build an exact byte-framing fixture for upstream SSE parser contracts.""" +def raw_sse_scenario( + name: str, + frames: list[bytes], + *, + preserve_chunks: bool = False, +) -> Scenario: + """Build an exact byte-framing fixture for upstream SSE parser contracts. + + ``preserve_chunks`` emits each supplied frame as its own HTTP transfer chunk so + tests can deterministically vary upstream segmentation without joining another + full copy of a large logical event in the harness. + """ return Scenario( name=name, @@ -662,6 +676,7 @@ def raw_sse_scenario(name: str, frames: list[bytes]) -> Scenario: nonstream_response=_nonstream_response(f"resp_{name}_nonstream", []), append_done=False, raw_stream_frames=list(frames), + preserve_raw_stream_chunks=preserve_chunks, ) @@ -1188,6 +1203,32 @@ def _send_abrupt_sse(self, frames: list[bytes]) -> None: self.wfile.flush() self.close_connection = True + def _send_chunked_sse( + self, + frames: list[bytes], + extra_headers: Optional[dict[str, str]] = None, + ) -> None: + # Preserve each non-empty fixture frame as one HTTP transfer chunk. + # This is a transport-segmentation control, not SSE framing: callers + # may split anywhere, including inside the literal "data: " prefix. + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.send_header("Transfer-Encoding", "chunked") + self.send_header("Connection", "close") + for name, value in (extra_headers or {}).items(): + self.send_header(name, value) + self.end_headers() + for frame in frames: + if not frame: + raise ValueError("preserved stream chunks must be non-empty") + self.wfile.write(f"{len(frame):X}\r\n".encode("ascii")) + self.wfile.write(frame) + self.wfile.write(b"\r\n") + self.wfile.flush() + self.wfile.write(b"0\r\n\r\n") + self.wfile.flush() + self.close_connection = True + def _stream_bytes(self, events: list[dict[str, Any]], append_done: bool) -> bytes: frames = [] for event in events: @@ -1380,6 +1421,8 @@ def do_POST(self) -> None: frames.append(b"data: [DONE]\n\n") if owner.scenario.abrupt_eof: self._send_abrupt_sse(frames) + elif owner.scenario.preserve_raw_stream_chunks: + self._send_chunked_sse(frames, attempt_headers) else: self._send_bytes( 200, @@ -1421,8 +1464,16 @@ def do_POST(self) -> None: attempt_headers, ) return + response_object = owner.scenario.nonstream_response + if owner.scenario.nonstream_responses: + response_object = owner.scenario.nonstream_responses[ + min( + request_number - 1, + len(owner.scenario.nonstream_responses) - 1, + ) + ] payload = json.dumps( - owner.scenario.nonstream_response, + response_object, separators=(",", ":"), ).encode("utf-8") self._send_bytes( @@ -1638,6 +1689,7 @@ def __call__(self) -> int: "request_start", "request_parsed", "upstream_attempt", + "transport_failure", "upstream_retry", "upstream_headers", "upstream_first_event", @@ -1791,6 +1843,12 @@ class RealShim(AbstractContextManager["RealShim"]): "LC_ALL": "C.UTF-8", "TZ": "UTC", } + _TEST_ENV_OVERRIDE_NAMES = frozenset({ + "SHIM_REASONING_EFFORT", + "SHIM_SANITIZE_TOOLS", + "SHIM_STRIP_MODEL_PREFIX", + "SHIM_TEXT_VERBOSITY", + }) _CONTROLLED_CHILD_ENV_NAMES = frozenset({ "SHIM_PORT", "SHIM_BACKEND_MODE", @@ -1817,12 +1875,27 @@ def __init__( mode: str, *, port_selector: Optional[Callable[[], int]] = None, + env_overrides: Optional[dict[str, str]] = None, ) -> None: if mode not in {"openai", "chatgpt"}: raise ValueError(f"unsupported shim mode: {mode}") + overrides = dict(env_overrides or {}) + rejected = set(overrides) - self._TEST_ENV_OVERRIDE_NAMES + if rejected: + raise ValueError( + f"test environment override is not allowlisted: {sorted(rejected)!r}" + ) + non_strings = sorted( + name for name, value in overrides.items() if not isinstance(value, str) + ) + if non_strings: + raise TypeError( + f"test environment override values must be strings: {non_strings!r}" + ) self.backend = backend self.mode = mode self._port_selector = port_selector or _reserve_dynamic_port + self._env_overrides = overrides self.port = 0 self.base_url = "" self.backend_base_url = "" @@ -1903,6 +1976,9 @@ def __enter__(self) -> "RealShim": "PYTHONUNBUFFERED": "1", } ) + # Apply only the narrow, constructor-validated behavior knobs. Backend, + # credential, proxy, port, and auth-store defaults remain fixed by the rig. + env.update(self._env_overrides) # The reserve/release/start sequence has an unavoidable test-only bind # race. Retry at most twice, and only when uvicorn's captured startup @@ -2093,6 +2169,7 @@ def post_messages( tools: Optional[list[dict[str, Any]]] = None, messages: Optional[list[dict[str, Any]]] = None, headers: Optional[dict[str, str]] = None, + timeout: float = 30.0, ) -> HTTPResult: body: dict[str, Any] = { "model": model, @@ -2109,6 +2186,7 @@ def post_messages( return self.post_raw_messages( json.dumps(body, separators=(",", ":")).encode("utf-8"), headers=headers, + timeout=timeout, ) def post_raw_messages( @@ -2128,6 +2206,21 @@ def post_raw_messages( ) return _direct_request(request, timeout=timeout) + def post_count_tokens( + self, + body: dict[str, Any], + *, + timeout: float = 2.0, + ) -> HTTPResult: + payload = json.dumps(body, separators=(",", ":")).encode("utf-8") + request = urllib.request.Request( + f"{self.base_url}/v1/messages/count_tokens", + data=payload, + headers={"Content-Type": "application/json"}, + method="POST", + ) + return _direct_request(request, timeout=timeout) + def remove_auth_store(self) -> None: """Remove only the fabricated isolated auth store for an auth-error probe.""" @@ -2157,7 +2250,11 @@ def assert_offline_contract(self) -> None: raise AssertionError("isolated credential directory is outside scripts/scratch") if self.child_env.get("OPENAI_API_KEY") != FAKE_OPENAI_KEY: raise AssertionError("child OpenAI key is not the fabricated test key") - allowed_names = set(self._CONTROLLED_BASE_ENV) | set(self._CONTROLLED_CHILD_ENV_NAMES) + allowed_names = ( + set(self._CONTROLLED_BASE_ENV) + | set(self._CONTROLLED_CHILD_ENV_NAMES) + | set(self._TEST_ENV_OVERRIDE_NAMES) + ) unexpected_names = set(self.child_env) - allowed_names if unexpected_names: raise AssertionError( @@ -2437,10 +2534,37 @@ def controlled_translation_failure(*args, **kwargs): selected_outcomes = list(attempt_outcomes or [response_status]) selected_close_failures = set(close_fail_attempts or set()) + class MidbodyReadTimeoutStream(module.httpx.AsyncByteStream): + async def __aiter__(self): + yield ( + b'event: response.created\n' + b'data: {"type":"response.created","response":' + b'{"id":"resp_midbody","status":"in_progress"}}\n\n' + ) + raise module.httpx.ReadTimeout("injected mid-body read timeout") + def response_for_attempt(attempt_number: int): outcome = selected_outcomes[min(attempt_number - 1, len(selected_outcomes) - 1)] - if outcome == "transport": - raise module.httpx.ConnectError("injected stream transport failure") + injected_transport_errors = { + "transport": module.httpx.ConnectError, + "connect_timeout": module.httpx.ConnectTimeout, + "read_timeout": module.httpx.ReadTimeout, + "write_timeout": module.httpx.WriteTimeout, + "pool_timeout": module.httpx.PoolTimeout, + } + if outcome in injected_transport_errors: + error_class = injected_transport_errors[outcome] + raise error_class("injected transport failure") + if outcome == "midbody_read_timeout": + return module.httpx.Response( + 200, + headers=headers, + stream=MidbodyReadTimeoutStream(), + request=module.httpx.Request( + "POST", "http://127.0.0.1:1/v1/responses" + ), + extensions={"http_version": http_version.encode("ascii", "replace")}, + ) status = int(outcome) content = ( stream_payload diff --git a/tests/provider_shim/test_historical_regressions.py b/tests/provider_shim/test_historical_regressions.py new file mode 100644 index 00000000..d4447957 --- /dev/null +++ b/tests/provider_shim/test_historical_regressions.py @@ -0,0 +1,622 @@ +"""Historical provider-shim contracts driven through the real subprocess path.""" + +from __future__ import annotations + +import copy +import json +import threading +import unittest + +from ._loopback_harness import ( + READ_TOOL, + USAGE, + MockResponsesServer, + RealShim, + events_scenario, + full_response_scenario, + lifecycle_for_response, +) + + +_HEALTH_KEYS = { + "service", + "status", + "backend", + "backend_mode", + "codex_home_present", + "version", + "sanitize_tools", + "reasoning_effort", + "text_verbosity", +} + + +def _response(response_id: str, output: list[dict[str, object]], usage: object = None): + body: dict[str, object] = { + "id": response_id, + "model": "gpt-fixture", + "status": "completed", + "output": copy.deepcopy(output), + "incomplete_details": None, + "error": None, + } + if usage is not None: + body["usage"] = copy.deepcopy(usage) + return body + + +def _message_payload( + *, + model: str = "gpt-fixture", + output_config: object = None, + thinking: object = None, + messages: list[dict[str, object]] | None = None, +) -> bytes: + body: dict[str, object] = { + "model": model, + "max_tokens": 256, + "stream": False, + "messages": messages or [{"role": "user", "content": "Historical rig."}], + } + if output_config is not None: + body["output_config"] = output_config + if thinking is not None: + body["thinking"] = thinking + return json.dumps(body, separators=(",", ":")).encode("utf-8") + + +def _tool_item( + item_id: str, + call_id: str, + name: str, + arguments: dict[str, object], +) -> dict[str, object]: + return { + "type": "function_call", + "id": item_id, + "call_id": call_id, + "name": name, + "arguments": json.dumps(arguments, separators=(",", ":")), + "status": "completed", + } + + +class ProviderShimHistoricalRegressionTests(unittest.TestCase): + maxDiff = 16000 + + def test_count_tokens_rejects_poison_then_real_request_calibrates(self) -> None: + scenario = full_response_scenario() + template = copy.deepcopy(scenario.nonstream_response) + invalid_responses: list[dict[str, object]] = [] + + missing_usage = copy.deepcopy(template) + missing_usage.pop("usage", None) + invalid_responses.append(missing_usage) + + nonobject_usage = copy.deepcopy(template) + nonobject_usage["usage"] = "malformed" + invalid_responses.append(nonobject_usage) + + missing_input = copy.deepcopy(template) + missing_input["usage"] = {"output_tokens": 17} + invalid_responses.append(missing_input) + + malformed_input = copy.deepcopy(template) + malformed_input["usage"] = {"input_tokens": "bad", "output_tokens": 17} + invalid_responses.append(malformed_input) + + zero_input = copy.deepcopy(template) + zero_input["usage"] = {"input_tokens": 0, "output_tokens": 17} + invalid_responses.append(zero_input) + + negative_input = copy.deepcopy(template) + negative_input["usage"] = {"input_tokens": -1, "output_tokens": 17} + invalid_responses.append(negative_input) + + outlier_input = copy.deepcopy(template) + outlier_input["usage"] = { + "input_tokens": 999_999_999, + "output_tokens": 17, + } + invalid_responses.append(outlier_input) + + valid_input = copy.deepcopy(template) + valid_input["usage"] = {"input_tokens": 50, "output_tokens": 17} + scenario.nonstream_responses = invalid_responses + [valid_input] + + count_body = { + "model": "gpt-fixture#low", + "messages": [{"role": "user", "content": "x" * 5000}], + "tools": [{"name": "Read", "input_schema": {"type": "object"}}], + } + request_payload = _message_payload() + + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "openai") as shim: + baseline_result = shim.post_count_tokens(count_body) + self.assertEqual(baseline_result.status, 200, baseline_result.text) + baseline = baseline_result.json()["input_tokens"] + self.assertGreater(baseline, 1) + + for expected_count in range(1, len(invalid_responses) + 1): + result = shim.post_raw_messages(request_payload) + self.assertEqual(result.status, 200, result.text) + unchanged = shim.post_count_tokens(count_body) + self.assertEqual(unchanged.status, 200, unchanged.text) + self.assertEqual(unchanged.json()["input_tokens"], baseline) + self.assertEqual(len(backend.responses_requests), expected_count) + + calibrated_request = shim.post_raw_messages(request_payload) + self.assertEqual(calibrated_request.status, 200, calibrated_request.text) + calibrated_result = shim.post_count_tokens(count_body) + self.assertEqual(calibrated_result.status, 200, calibrated_result.text) + calibrated = calibrated_result.json()["input_tokens"] + self.assertGreater(calibrated, 1) + self.assertNotEqual(calibrated, baseline) + + logs = shim.captured_stderr() + self.assertEqual(logs.count("count_tokens calibration rejected"), 1) + self.assertEqual(logs.count("count_tokens calibrated ratio="), 1) + backend.assert_request_counts(responses=8, oauth=0) + + def test_effort_suffix_and_precedence_matrix(self) -> None: + scenario = full_response_scenario() + default_cases = [ + ( + "gpt-fixture#low", + {"effort": "medium"}, + None, + "gpt-fixture", + "medium", + ), + ( + "gpt-fixture#low", + {"effort": "high"}, + None, + "gpt-fixture", + "low", + ), + ( + "gpt-fixture#medium", + {"effort": "invalid"}, + None, + "gpt-fixture", + "medium", + ), + ( + "gpt-fixture", + {"effort": " "}, + None, + "gpt-fixture", + "high", + ), + ( + "gpt-fixture#max", + None, + {"type": "disabled"}, + "gpt-fixture", + "none", + ), + ( + "gpt-fixture#max", + {"effort": "high"}, + {"type": "disabled"}, + "gpt-fixture", + "none", + ), + ( + "gpt-fixture#max", + {"effort": "low"}, + {"type": "disabled"}, + "gpt-fixture", + "low", + ), + ( + "gpt-fixture#future", + None, + None, + "gpt-fixture", + "high", + ), + ] + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "openai") as shim: + self.assertIsNone(shim.health["reasoning_effort"]) + for model, output_config, thinking, bare_model, effort in default_cases: + result = shim.post_raw_messages( + _message_payload( + model=model, + output_config=output_config, + thinking=thinking, + ) + ) + self.assertEqual(result.status, 200, result.text) + outgoing = backend.responses_requests[-1].body + self.assertEqual(outgoing["model"], bare_model) + self.assertEqual(outgoing["reasoning"]["effort"], effort) + backend.assert_request_counts(responses=len(default_cases), oauth=0) + + with MockResponsesServer(scenario) as backend: + with RealShim( + backend, + "openai", + env_overrides={"SHIM_REASONING_EFFORT": " xhigh "}, + ) as shim: + self.assertEqual(shim.health["reasoning_effort"], "xhigh") + env_cases = [ + ("gpt-fixture", None, "xhigh"), + ("gpt-fixture", {"effort": "high"}, "xhigh"), + ("gpt-fixture#", None, "xhigh"), + ("gpt-fixture#invalid", None, "xhigh"), + ("gpt-fixture#low", {"effort": "high"}, "low"), + ] + for model, output_config, effort in env_cases: + result = shim.post_raw_messages( + _message_payload(model=model, output_config=output_config) + ) + self.assertEqual(result.status, 200, result.text) + outgoing = backend.responses_requests[-1].body + self.assertEqual(outgoing["model"], "gpt-fixture") + self.assertEqual(outgoing["reasoning"]["effort"], effort) + + with MockResponsesServer(scenario) as backend: + with RealShim( + backend, + "openai", + env_overrides={"SHIM_REASONING_EFFORT": "invalid"}, + ) as shim: + result = shim.post_messages(stream=False) + self.assertEqual(result.status, 200, result.text) + self.assertEqual(shim.health["reasoning_effort"], "invalid") + self.assertEqual( + backend.responses_requests[0].body["reasoning"]["effort"], + "high", + ) + + def test_verbosity_startup_health_warning_and_payload_matrix(self) -> None: + cases = [ + ("unset", None, "high", 0), + ("low", "low", "low", 0), + ("medium", " MeDiUm ", "medium", 0), + ("high", "high", "high", 0), + ("blank", " ", "high", 1), + ("invalid", "verbose", "high", 1), + ] + scenario = full_response_scenario() + for name, raw, expected, warning_count in cases: + with self.subTest(case=name): + overrides = {} if raw is None else {"SHIM_TEXT_VERBOSITY": raw} + with MockResponsesServer(scenario) as backend: + with RealShim( + backend, + "openai", + env_overrides=overrides, + ) as shim: + result = shim.post_messages(stream=False) + self.assertEqual(result.status, 200, result.text) + self.assertEqual(shim.health["text_verbosity"], expected) + self.assertEqual( + backend.responses_requests[0].body["text"], + {"verbosity": expected}, + ) + logs = shim.captured_stderr() + self.assertEqual( + logs.count("SHIM_TEXT_VERBOSITY"), + warning_count, + logs, + ) + self.assertIn(f"text_verbosity={expected}", logs) + + def test_complete_stable_health_contract_both_lanes_and_overrides(self) -> None: + scenario = full_response_scenario() + cases = [ + ( + "openai", + { + "SHIM_SANITIZE_TOOLS": "false", + "SHIM_REASONING_EFFORT": "low", + "SHIM_TEXT_VERBOSITY": "medium", + "SHIM_STRIP_MODEL_PREFIX": "provider/", + }, + False, + "low", + "medium", + ), + ( + "chatgpt", + { + "SHIM_SANITIZE_TOOLS": "1", + "SHIM_REASONING_EFFORT": "max", + "SHIM_TEXT_VERBOSITY": "low", + }, + True, + "max", + "low", + ), + ] + for mode, overrides, sanitize, effort, verbosity in cases: + with self.subTest(mode=mode): + with MockResponsesServer(scenario) as backend: + with RealShim( + backend, + mode, + env_overrides=overrides, + ) as shim: + health_result = shim.get_health() + self.assertEqual(health_result.status, 200, health_result.text) + health = health_result.json() + self.assertEqual(set(health), _HEALTH_KEYS) + self.assertEqual(health["service"], "daaf-anthropic-openai-shim") + self.assertEqual(health["status"], "ok") + self.assertEqual(health["version"], "1.2.13") + self.assertEqual(health["backend_mode"], mode) + self.assertEqual(health["sanitize_tools"], sanitize) + self.assertEqual(health["reasoning_effort"], effort) + self.assertEqual(health["text_verbosity"], verbosity) + self.assertIs(health["codex_home_present"], True) + expected_backend = ( + f"{backend.base_url}/v1" + if mode == "openai" + else backend.base_url + ) + self.assertEqual(health["backend"], expected_backend) + for forbidden in ( + "port", + "pid", + "api_key", + "credential_path", + "codex_home", + ): + self.assertNotIn(forbidden, health) + + result = shim.post_messages( + stream=False, + model="provider/gpt-fixture", + ) + self.assertEqual(result.status, 200, result.text) + outgoing = backend.responses_requests[0].body + expected_model = ( + "gpt-fixture" if mode == "openai" else "provider/gpt-fixture" + ) + self.assertEqual(outgoing["model"], expected_model) + self.assertEqual(outgoing["reasoning"]["effort"], effort) + self.assertEqual(outgoing["text"], {"verbosity": verbosity}) + + def test_two_turn_reasoning_cache_replay_is_paired_and_conversation_local(self) -> None: + scenario = full_response_scenario() + first_response = copy.deepcopy(scenario.nonstream_response) + quiet_response = _response( + "resp_replay_followup", + [], + {"input_tokens": 31, "output_tokens": 1, "total_tokens": 32}, + ) + scenario.nonstream_responses = [first_response, quiet_response, quiet_response] + cached_reasoning = copy.deepcopy(first_response["output"][0]) + + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "openai") as shim: + first = shim.post_messages(stream=False, tools=[READ_TOOL]) + self.assertEqual(first.status, 200, first.text) + first_content = first.json()["content"] + paired_messages = [ + {"role": "assistant", "content": first_content}, + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": "call_full_fixture", + "content": "paired result", + } + ], + }, + ] + unrelated_messages = [ + { + "role": "assistant", + "content": [ + { + "type": "tool_use", + "id": "call_unrelated", + "name": "Read", + "input": {"file_path": "/daaf/CONTRIBUTING.md"}, + } + ], + }, + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": "call_unrelated", + "content": "unrelated result", + } + ], + }, + ] + results: dict[str, object] = {} + failures: list[BaseException] = [] + + def call(label: str, messages: list[dict[str, object]]) -> None: + try: + results[label] = shim.post_messages( + stream=False, + tools=[READ_TOOL], + messages=messages, + ) + except BaseException as error: + failures.append(error) + + threads = [ + threading.Thread( + target=call, + args=("paired", paired_messages), + daemon=True, + ), + threading.Thread( + target=call, + args=("unrelated", unrelated_messages), + daemon=True, + ), + ] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=20.0) + self.assertTrue(all(not thread.is_alive() for thread in threads)) + self.assertEqual(failures, []) + self.assertEqual(set(results), {"paired", "unrelated"}) + + requests_by_call: dict[str, dict[str, object]] = {} + for request in backend.responses_requests[1:]: + call_ids = [ + item.get("call_id") + for item in request.body["input"] + if item.get("type") == "function_call" + ] + if "call_full_fixture" in call_ids: + requests_by_call["paired"] = request.body + if "call_unrelated" in call_ids: + requests_by_call["unrelated"] = request.body + self.assertEqual(set(requests_by_call), {"paired", "unrelated"}) + + paired_input = requests_by_call["paired"]["input"] + function_at = next( + index + for index, item in enumerate(paired_input) + if item.get("type") == "function_call" + and item.get("call_id") == "call_full_fixture" + ) + self.assertGreater(function_at, 0) + self.assertEqual(paired_input[function_at - 1], cached_reasoning) + self.assertEqual( + paired_input[function_at - 1]["encrypted_content"], + "ENC_rs_full", + ) + self.assertFalse( + any( + item.get("type") == "reasoning" + for item in requests_by_call["unrelated"]["input"] + ) + ) + + paired_lifecycle = lifecycle_for_response(shim, results["paired"]) + unrelated_lifecycle = lifecycle_for_response(shim, results["unrelated"]) + paired_terminal = next( + line for line in paired_lifecycle if line.event == "terminal" + ) + unrelated_terminal = next( + line for line in unrelated_lifecycle if line.event == "terminal" + ) + self.assertEqual(paired_terminal.fields["reasoning_cache_miss"], "0") + self.assertEqual( + unrelated_terminal.fields["reasoning_cache_miss"], "1" + ) + backend.assert_request_counts(responses=3, oauth=0) + + def test_tool_argument_sanitization_matrix_and_disabled_passthrough(self) -> None: + originals = [ + ("call_read_empty", "Read", {"file_path": "/daaf/README.md", "pages": "", "note": ""}), + ("call_read_pages", "Read", {"file_path": "/daaf/a.pdf", "pages": "1-3"}), + ("call_agent", "Agent", {"prompt": "", "isolation": "remote", "model": "opus"}), + ("call_task", "Task", {"prompt": "task", "isolation": "", "model": "sonnet"}), + ("call_bash_false", "Bash", {"command": "pwd", "dangerouslyDisableSandbox": False, "description": ""}), + ("call_bash_true", "Bash", {"command": "pwd", "dangerouslyDisableSandbox": True}), + ] + output = [ + _tool_item(f"fc_{index}", call_id, name, arguments) + for index, (call_id, name, arguments) in enumerate(originals) + ] + response = _response("resp_sanitize", output, dict(USAGE)) + scenario = events_scenario( + "sanitize-matrix", + [], + nonstream_response=response, + ) + tools = [ + { + "name": name, + "description": "fixture", + "input_schema": {"type": "object", "properties": {}}, + } + for name in ("Read", "Agent", "Task", "Bash") + ] + expected = { + "call_read_empty": {"file_path": "/daaf/README.md", "note": ""}, + "call_read_pages": {"file_path": "/daaf/a.pdf", "pages": "1-3"}, + "call_agent": {"prompt": "", "model": "opus"}, + "call_task": {"prompt": "task", "model": "sonnet"}, + "call_bash_false": {"command": "pwd", "description": ""}, + "call_bash_true": {"command": "pwd", "dangerouslyDisableSandbox": True}, + } + + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "openai") as shim: + result = shim.post_messages(stream=False, tools=tools) + self.assertEqual(result.status, 200, result.text) + actual = { + block["id"]: block["input"] + for block in result.json()["content"] + if block.get("type") == "tool_use" + } + self.assertEqual(actual, expected) + self.assertEqual(shim.health["sanitize_tools"], True) + self.assertEqual( + [tool["name"] for tool in backend.responses_requests[0].body["tools"]], + ["Read", "Agent", "Task", "Bash"], + ) + replay = shim.post_messages( + stream=False, + tools=tools, + messages=[{"role": "assistant", "content": result.json()["content"]}], + ) + self.assertEqual(replay.status, 200, replay.text) + replayed_arguments = { + item["call_id"]: json.loads(item["arguments"]) + for item in backend.responses_requests[1].body["input"] + if item.get("type") == "function_call" + } + self.assertEqual(replayed_arguments, expected) + + with MockResponsesServer(scenario) as backend: + with RealShim( + backend, + "openai", + env_overrides={"SHIM_SANITIZE_TOOLS": "false"}, + ) as shim: + result = shim.post_messages(stream=False, tools=tools) + self.assertEqual(result.status, 200, result.text) + actual = { + block["id"]: block["input"] + for block in result.json()["content"] + if block.get("type") == "tool_use" + } + original_map = { + call_id: arguments for call_id, _name, arguments in originals + } + self.assertEqual(actual, original_map) + self.assertEqual(shim.health["sanitize_tools"], False) + replay = shim.post_messages( + stream=False, + tools=tools, + messages=[{"role": "assistant", "content": result.json()["content"]}], + ) + self.assertEqual(replay.status, 200, replay.text) + replayed_arguments = { + item["call_id"]: json.loads(item["arguments"]) + for item in backend.responses_requests[1].body["input"] + if item.get("type") == "function_call" + } + self.assertEqual(replayed_arguments, original_map) + + def test_realshim_env_override_seam_rejects_security_sensitive_keys(self) -> None: + scenario = full_response_scenario() + with MockResponsesServer(scenario) as backend: + for name in ("OPENAI_API_KEY", "SHIM_BACKEND_BASE_URL", "MY_SECRET"): + with self.subTest(name=name): + with self.assertRaisesRegex(ValueError, "not allowlisted"): + RealShim(backend, "openai", env_overrides={name: "forbidden"}) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/provider_shim/test_reasoning_formatting.py b/tests/provider_shim/test_reasoning_formatting.py index 1385e6a5..1a581086 100644 --- a/tests/provider_shim/test_reasoning_formatting.py +++ b/tests/provider_shim/test_reasoning_formatting.py @@ -71,8 +71,12 @@ def test_openai_streaming_legacy_bytes_unchanged(self) -> None: backend.responses_requests[0].path, "/v1/responses", ) + self.assertEqual( + shim.health.get("service"), "daaf-anthropic-openai-shim" + ) + self.assertEqual(shim.health.get("status"), "ok") self.assertEqual(shim.health.get("backend_mode"), "openai") - self.assertEqual(shim.health.get("version"), "1.2.11") + self.assertEqual(shim.health.get("version"), "1.2.13") def test_openai_nonstreaming_legacy_bytes_unchanged(self) -> None: scenario = central_multipart_scenario() diff --git a/tests/provider_shim/test_request_translation.py b/tests/provider_shim/test_request_translation.py new file mode 100644 index 00000000..78132646 --- /dev/null +++ b/tests/provider_shim/test_request_translation.py @@ -0,0 +1,629 @@ +"""Production-subprocess regressions for request translation and v1.2.12 diagnostics.""" + +from __future__ import annotations + +import base64 +import json +import threading +import unittest +import urllib.parse + +from ._loopback_harness import ( + USAGE, + MockResponsesServer, + RealShim, + controlled_asgi_probe, + events_scenario, + full_response_scenario, + lifecycle_for_response, + lifecycle_report, + parse_typed_sse, +) + + +_IMAGE_BYTES = b"fabricated-provider-shim-image-bytes" +_IMAGE_B64 = base64.b64encode(_IMAGE_BYTES).decode("ascii") +_IMAGE_URL = "https://images.example.invalid/visual.png?private=URL_SENTINEL_92Q" +_FILE_ID = "file_PROVIDER_SCOPED_SENTINEL_71K" + + +def _image(media_type: str = "image/png", data: str = _IMAGE_B64) -> dict[str, object]: + return { + "type": "image", + "source": {"type": "base64", "media_type": media_type, "data": data}, + } + + +def _missing_name_scenario(count: int = 50): + events: list[dict[str, object]] = [] + output: list[dict[str, object]] = [] + for index in range(count): + item_id = f"fc_aggregate_{index}" + call_id = f"call_aggregate_{index}" + arguments = "{}" + events.extend( + [ + { + "type": "response.output_item.added", + "item": { + "type": "function_call", + "id": item_id, + "call_id": call_id, + "name": "Noop", + }, + }, + { + "type": "response.function_call_arguments.done", + "item_id": item_id, + "arguments": arguments, + }, + { + "type": "response.output_item.done", + "item": { + "type": "function_call", + "id": item_id, + "call_id": call_id, + "name": "Noop", + "arguments": arguments, + }, + }, + ] + ) + output.append( + { + "type": "function_call", + "id": item_id, + "call_id": call_id, + "name": "Noop", + "arguments": arguments, + } + ) + events.append( + { + "type": "response.completed", + "response": { + "id": "resp_warning_aggregate", + "status": "completed", + "output": output, + "usage": dict(USAGE), + }, + } + ) + return events_scenario("warning-aggregate", events) + + +class ProviderShimRequestTranslationTests(unittest.TestCase): + maxDiff = 12000 + + def test_supported_base64_images_map_in_both_lanes(self) -> None: + for mode in ("openai", "chatgpt"): + with self.subTest(mode=mode): + scenario = full_response_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, mode) as shim: + for media_type in ( + "image/jpeg", + "image/png", + "image/gif", + "image/webp", + ): + result = shim.post_messages( + stream=False, + messages=[ + { + "role": "user", + "content": [_image(media_type)], + } + ], + ) + self.assertEqual(result.status, 200, media_type) + self.assertEqual(len(backend.responses_requests), 4) + for request, media_type in zip( + backend.responses_requests, + ("image/jpeg", "image/png", "image/gif", "image/webp"), + ): + content = request.body["input"][0]["content"] + self.assertEqual( + content, + [ + { + "type": "input_image", + "image_url": f"data:{media_type};base64,{_IMAGE_B64}", + "detail": "auto", + } + ], + ) + logs = shim.captured_stderr() + self.assertNotIn(_IMAGE_B64, logs) + self.assertNotIn(_IMAGE_BYTES.decode("ascii"), logs) + + def test_url_image_and_tool_result_preserve_content_order_both_lanes(self) -> None: + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "before"}, + {"type": "image", "source": {"type": "url", "url": _IMAGE_URL}}, + {"type": "text", "text": "after"}, + ], + }, + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": "call_image_result", + "content": [ + {"type": "text", "text": "tool-before"}, + _image(), + {"type": "text", "text": "tool-after"}, + ], + } + ], + }, + ] + expected_user = [ + {"type": "input_text", "text": "before"}, + {"type": "input_image", "image_url": _IMAGE_URL, "detail": "auto"}, + {"type": "input_text", "text": "after"}, + ] + expected_tool = [ + {"type": "input_text", "text": "tool-before"}, + { + "type": "input_image", + "image_url": f"data:image/png;base64,{_IMAGE_B64}", + "detail": "auto", + }, + {"type": "input_text", "text": "tool-after"}, + ] + for mode in ("openai", "chatgpt"): + with self.subTest(mode=mode): + scenario = full_response_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, mode) as shim: + result = shim.post_messages(stream=False, messages=messages) + self.assertEqual(result.status, 200, result.text) + request_input = backend.responses_requests[0].body["input"] + self.assertEqual(request_input[0]["content"], expected_user) + self.assertEqual( + request_input[1], + { + "type": "function_call_output", + "call_id": "call_image_result", + "output": expected_tool, + }, + ) + logs = shim.captured_stderr() + self.assertNotIn(_IMAGE_URL, logs) + self.assertNotIn(_IMAGE_B64, logs) + + def test_invalid_visual_and_unknown_blocks_are_local_400_without_backend(self) -> None: + cases = ( + ("invalid-base64", _image(data="abcd===")), + ("unsupported-mime", _image(media_type="image/svg+xml")), + ( + "unsupported-source", + {"type": "image", "source": {"type": "future", "data": "hidden"}}, + ), + ( + "file-source", + {"type": "image", "source": {"type": "file", "file_id": _FILE_ID}}, + ), + ( + "invalid-url-scheme", + {"type": "image", "source": {"type": "url", "url": "file:///secret"}}, + ), + ( + "invalid-url-credentials", + { + "type": "image", + "source": {"type": "url", "url": "https://user:pass@example.invalid/x"}, + }, + ), + ("unknown-block", {"type": "future_block", "payload": "PROMPT_SENTINEL_4X"}), + ) + for mode in ("openai", "chatgpt"): + with self.subTest(mode=mode): + scenario = full_response_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, mode) as shim: + for name, block in cases: + with self.subTest(case=name): + result = shim.post_messages( + stream=False, + messages=[{"role": "user", "content": [block]}], + ) + self.assertEqual(result.status, 400, (name, result.text)) + self.assertEqual( + result.json()["error"]["type"], + "invalid_request_error", + ) + assistant = shim.post_messages( + stream=False, + messages=[{"role": "assistant", "content": [_image()]}], + ) + self.assertEqual(assistant.status, 400, assistant.text) + self.assertEqual(len(backend.responses_requests), 0) + logs = shim.captured_stderr() + for forbidden in ( + _IMAGE_B64, + _IMAGE_URL, + _FILE_ID, + "PROMPT_SENTINEL_4X", + "https://user:pass@example.invalid/x", + ): + self.assertNotIn(forbidden, logs) + + def test_malformed_history_is_local_400_in_both_lanes(self) -> None: + sentinels = ( + "INVALID_ROLE_PROMPT_SENTINEL_7V", + "MISPLACED_TOOL_INPUT_SENTINEL_8W", + "MISPLACED_RESULT_SENTINEL_9X", + "INVALID_RESULT_CONTENT_SENTINEL_0Y", + "PRIVATE_TOOL_USE_ID_SENTINEL_1Z", + ) + cases = ( + ( + "invalid-role", + [ + { + "role": sentinels[0], + "content": "private prompt text", + } + ], + ), + ( + "missing-role", + [{"content": "private prompt text"}], + ), + ( + "tool-use-in-user-message", + [ + { + "role": "user", + "content": [ + { + "type": "tool_use", + "id": sentinels[4], + "name": "PrivateTool", + "input": {"secret": sentinels[1]}, + } + ], + } + ], + ), + ( + "tool-result-in-assistant-message", + [ + { + "role": "assistant", + "content": [ + { + "type": "tool_result", + "tool_use_id": sentinels[4], + "content": sentinels[2], + } + ], + } + ], + ), + ( + "tool-result-object-content", + [ + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": sentinels[4], + "content": {"private": sentinels[3]}, + } + ], + } + ], + ), + ( + "tool-result-null-content", + [ + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": sentinels[4], + "content": None, + } + ], + } + ], + ), + ) + for mode in ("openai", "chatgpt"): + with self.subTest(mode=mode): + scenario = full_response_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, mode) as shim: + response_texts = [] + for name, messages in cases: + with self.subTest(mode=mode, case=name): + result = shim.post_messages( + stream=False, + messages=messages, + ) + response_texts.append(result.text) + self.assertEqual(result.status, 400, result.text) + self.assertEqual( + result.json().get("error", {}).get("type"), + "invalid_request_error", + ) + backend.assert_request_counts(responses=0, oauth=0) + privacy_surfaces = "\n".join(response_texts) + shim.captured_stderr() + for sentinel in sentinels: + self.assertNotIn(sentinel, privacy_surfaces) + + control = full_response_scenario() + with MockResponsesServer(control) as backend: + with RealShim(backend, mode) as shim: + result = shim.post_messages( + stream=False, + messages=[ + {"role": "user", "content": "valid user text"}, + {"role": "assistant", "content": "valid assistant text"}, + { + "role": "assistant", + "content": [ + { + "type": "tool_use", + "id": "call_valid_history", + "name": "ValidTool", + "input": {"value": 1}, + } + ], + }, + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": "call_valid_history", + "content": "valid string result", + } + ], + }, + ], + ) + self.assertEqual(result.status, 200, result.text) + request_input = backend.responses_requests[0].body["input"] + self.assertEqual(request_input[-2]["type"], "function_call") + self.assertEqual( + request_input[-1], + { + "type": "function_call_output", + "call_id": "call_valid_history", + "output": "valid string result", + }, + ) + backend.assert_request_counts(responses=1, oauth=0) + + def test_text_only_tool_result_retains_string_compatibility(self) -> None: + scenario = full_response_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "openai") as shim: + result = shim.post_messages( + stream=False, + messages=[ + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": "call_text_only", + "content": [ + {"type": "text", "text": "one"}, + {"type": "text", "text": "two"}, + ], + } + ], + } + ], + ) + self.assertEqual(result.status, 200, result.text) + self.assertEqual( + backend.responses_requests[0].body["input"][0]["output"], + "one\ntwo", + ) + + def test_system_role_messages_fold_to_user_in_both_lanes(self) -> None: + # v1.2.13 regression (live outage 2026-07-19): current Claude Code appends + # a role:"system" message inside `messages` (alongside top-level + # context_management/output_config). The v1.2.12 user/assistant-only role + # checks rejected every real conversation turn with the static 400. + # System-role messages must pass validation and fold to user-role input. + for mode in ("openai", "chatgpt"): + with self.subTest(mode=mode): + scenario = full_response_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, mode) as shim: + body = { + "model": "gpt-fixture", + "max_tokens": 256, + "stream": False, + "metadata": {"user_id": "user_fixture_session_0001"}, + "context_management": {"edits": []}, + "output_config": {"verbosity": "high"}, + "system": [{"type": "text", "text": "system prompt"}], + "messages": [ + { + "role": "user", + "content": [{"type": "text", "text": "hello"}], + }, + { + "role": "system", + "content": "trailing system reminder", + }, + ], + } + result = shim.post_raw_messages( + json.dumps(body, separators=(",", ":")).encode("utf-8"), + ) + self.assertEqual(result.status, 200, result.text) + request = backend.responses_requests[-1] + input_items = request.body["input"] + folded = input_items[-1] + self.assertEqual(folded.get("role"), "user") + self.assertEqual( + folded.get("content"), + [{ + "type": "input_text", + "text": "trailing system reminder", + }], + ) + roles = { + item.get("role") + for item in input_items + if isinstance(item, dict) + } + self.assertNotIn("system", roles) + + def test_system_role_tool_blocks_stay_fail_closed(self) -> None: + # v1.2.13 companion: the validation gauntlet checks RAW roles, so a + # system-role message carrying tool_use/tool_result must still be + # rejected with the static 400 BEFORE any backend round-trip — the + # translator's fold-to-user must never launder tool blocks into a + # role placement the gauntlet would have refused. + for mode in ("openai", "chatgpt"): + for block in ( + {"type": "tool_use", "id": "toolu_x", "name": "Noop", "input": {}}, + {"type": "tool_result", "tool_use_id": "toolu_x", "content": "ok"}, + ): + with self.subTest(mode=mode, block=block["type"]): + scenario = full_response_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, mode) as shim: + body = { + "model": "gpt-fixture", + "max_tokens": 256, + "stream": False, + "messages": [ + {"role": "system", "content": [block]}, + ], + } + result = shim.post_raw_messages( + json.dumps( + body, separators=(",", ":"), + ).encode("utf-8"), + ) + self.assertEqual(result.status, 400, result.text) + self.assertEqual( + result.json()["error"]["message"], + "invalid request structure", + ) + self.assertEqual(len(backend.responses_requests), 0) + + def test_request_ids_and_transport_timeout_families_are_bounded(self) -> None: + scenario = full_response_scenario() + scenario.stream_headers = {"x-request-id": "provider-request-17"} + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "openai") as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + lifecycle_report(parse_typed_sse(result.body)) + lines = lifecycle_for_response(shim, result) + local_id = result.headers["x-daaf-request-id"] + self.assertEqual( + backend.responses_requests[0].headers.get("x-client-request-id"), + local_id, + ) + upstream = next(line for line in lines if line.event == "upstream_headers") + self.assertEqual(upstream.fields["upstream_req_id"], "provider-request-17") + terminal = next(line for line in lines if line.event == "terminal") + self.assertEqual(terminal.fields["client_req_id"], local_id) + self.assertEqual(terminal.fields["upstream_req_id"], "provider-request-17") + + expected = { + "connect_timeout": ("ConnectTimeout", "connect"), + "read_timeout": ("ReadTimeout", "header_wait_or_body_read"), + "write_timeout": ("WriteTimeout", "request_body_write"), + "pool_timeout": ("PoolTimeout", "connection_pool_wait"), + } + for outcome, (exception_class, phase) in expected.items(): + with self.subTest(timeout=outcome): + report = controlled_asgi_probe(attempt_outcomes=[outcome, 200]) + failures = [line for line in report.lifecycle if line.event == "transport_failure"] + self.assertEqual(len(failures), 1) + self.assertEqual(failures[0].fields["exception_class"], exception_class) + self.assertEqual(failures[0].fields["failure_phase"], phase) + self.assertEqual(failures[0].fields["retryable"], "y") + retry = next(line for line in report.lifecycle if line.event == "upstream_retry") + self.assertEqual(retry.fields["source"], "shim_policy") + self.assertEqual(retry.fields["reason"], "transport") + + midbody = controlled_asgi_probe(attempt_outcomes=["midbody_read_timeout"]) + midbody_failures = [ + line for line in midbody.lifecycle if line.event == "transport_failure" + ] + self.assertEqual(len(midbody_failures), 1) + self.assertEqual( + midbody_failures[0].fields["exception_class"], "ReadTimeout" + ) + self.assertEqual( + midbody_failures[0].fields["failure_phase"], + "post_stream_start_body_read", + ) + self.assertEqual(midbody_failures[0].fields["retryable"], "n") + midbody_terminal = next( + line for line in midbody.lifecycle if line.event == "terminal" + ) + self.assertEqual( + midbody_terminal.fields["failure_phase"], + "post_stream_start_body_read", + ) + self.assertEqual(midbody.upstream_calls, 1) + + def test_missing_name_warning_is_concurrency_safe_and_bounded(self) -> None: + scenario = _missing_name_scenario(150) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + results: list[object] = [] + failures: list[BaseException] = [] + + def call() -> None: + try: + results.append(shim.post_messages(stream=True)) + except BaseException as error: + failures.append(error) + + threads = [threading.Thread(target=call, daemon=True) for _ in range(2)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=20.0) + self.assertTrue(all(not thread.is_alive() for thread in threads)) + self.assertEqual(failures, []) + self.assertEqual(len(results), 2) + for result in results: + self.assertEqual(result.status, 200, result.text) + lifecycle_report(parse_typed_sse(result.body)) + warning_lines = [ + line + for line in shim.captured_stderr().splitlines() + if "arguments.done_missing_name" in line + ] + self.assertEqual(len(warning_lines), 4, warning_lines) + expected = ( + (1, None), + (100, 98), + (200, 99), + (300, 99), + ) + for line, (observed, suppressed) in zip(warning_lines, expected): + self.assertIn(f"observed={observed}", line) + if suppressed is None: + self.assertNotIn("suppressed=", line) + else: + self.assertIn(f"suppressed={suppressed}", line) + backend.assert_request_counts(responses=2, oauth=0) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/provider_shim/test_stream_hardening.py b/tests/provider_shim/test_stream_hardening.py index c172468c..e3988e78 100644 --- a/tests/provider_shim/test_stream_hardening.py +++ b/tests/provider_shim/test_stream_hardening.py @@ -739,6 +739,133 @@ def test_sse_partial_line_at_eof_fails(self) -> None: expected_kinds=[], ) + def test_sse_event_size_limit_is_transport_segmentation_invariant(self) -> None: + limit = 16 * 1024 * 1024 + template = { + "type": "response.completed", + "response": { + "id": "resp_size_boundary", + "status": "completed", + "output": [], + "usage": dict(USAGE), + "padding": "", + }, + } + empty_payload = json.dumps(template, separators=(",", ":")) + marker = '"padding":""' + self.assertIn(marker, empty_payload) + prefix, suffix = empty_payload.split(marker) + payload_overhead = len((prefix + '"padding":""' + suffix).encode("utf-8")) + + for label, target_size, should_succeed in ( + ("below", limit - 1, True), + ("at", limit, True), + ("above", limit + 1, False), + ): + padding_size = target_size - payload_overhead + self.assertGreaterEqual(padding_size, 0) + payload = ( + prefix + '"padding":"' + ("X" * padding_size) + '"' + suffix + ).encode("utf-8") + self.assertEqual(len(payload), target_size) + segmentations = ( + ( + "joined-by-harness", + [b"data: ", payload, b"\n\n"], + False, + ), + ( + "prefix-and-terminator-fragmented", + [b"d", b"a", b"t", b"a", b":", b" ", payload, b"\n", b"\n"], + True, + ), + ) + for segmentation, chunks, preserve_chunks in segmentations: + with self.subTest(boundary=label, segmentation=segmentation): + scenario = raw_sse_scenario( + f"event-size-{label}-{segmentation}", + chunks, + preserve_chunks=preserve_chunks, + ) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, timeout=60.0) + self.assertEqual(result.status, 200) + frames = parse_typed_sse(result.body) + if should_succeed: + lifecycle_report(frames) + else: + failure_lifecycle_report(frames) + backend.assert_request_counts(responses=1, oauth=0) + + def test_sse_event_size_counts_repeated_data_field_newline_join(self) -> None: + limit = 16 * 1024 * 1024 + template = { + "type": "response.completed", + "response": { + "id": "resp_multidata_size_boundary", + "status": "completed", + "output": [], + "usage": dict(USAGE), + "padding": "", + }, + } + empty_payload = json.dumps(template, separators=(",", ":")) + marker = '"padding":""' + prefix, suffix = empty_payload.split(marker) + payload_overhead = len((prefix + '"padding":""' + suffix).encode("utf-8")) + + for label, logical_size, should_succeed in ( + ("at", limit, True), + ("above", limit + 1, False), + ): + # Repeated SSE data fields insert one logical newline. Size the JSON bytes + # one byte below the target, then split after a comma where JSON permits + # that newline as insignificant whitespace. + padding_size = logical_size - 1 - payload_overhead + payload = ( + prefix + '"padding":"' + ("Y" * padding_size) + '"' + suffix + ).encode("utf-8") + split_at = payload.find(b',"response"') + 1 + self.assertGreater(split_at, 0) + self.assertEqual(len(payload) + 1, logical_size) + scenario = raw_sse_scenario( + f"multidata-event-size-{label}", + [ + b"data: ", + payload[:split_at], + b"\n", + b"data: ", + payload[split_at:], + b"\n", + b"\n", + ], + preserve_chunks=True, + ) + with self.subTest(boundary=label): + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, timeout=60.0) + self.assertEqual(result.status, 200) + frames = parse_typed_sse(result.body) + if should_succeed: + lifecycle_report(frames) + else: + failure_lifecycle_report(frames) + backend.assert_request_counts(responses=1, oauth=0) + + def test_sse_oversized_partial_line_eof_fails_without_dispatch(self) -> None: + limit = 16 * 1024 * 1024 + scenario = raw_sse_scenario( + "oversized-partial-line-eof", + [b"data: " + (b"Z" * (limit + 1))], + ) + self._assert_stream_failure( + scenario=scenario, + marker="SSE_OVERSIZED_PARTIAL_LINE_EOF", + expected_kinds=[], + ) + def test_sse_crlf_blank_line_framing_succeeds(self) -> None: terminal = { "type": "response.completed", @@ -1484,7 +1611,7 @@ def test_v1211_request_id_headers_and_lifecycle_correlation(self) -> None: startup_records = [ line for line in shim.captured_stderr().splitlines() - if "req_id=- phase=process shim v1.2.11 starting" in line + if "req_id=- phase=process shim v1.2.13 starting" in line ] self.assertEqual(len(startup_records), 1, startup_records) self.assertIn( diff --git a/tests/python/test_deploy_smoke.py b/tests/python/test_deploy_smoke.py index dec6fd2e..de282c79 100644 --- a/tests/python/test_deploy_smoke.py +++ b/tests/python/test_deploy_smoke.py @@ -13,6 +13,7 @@ python3 -m unittest discover -s /daaf/tests/python -p 'test_deploy_smoke.py' """ +import json import os import subprocess import sys @@ -48,6 +49,173 @@ def _scratch_dir(prefix): return d +class _HealthResponse: + def __init__(self, payload): + self.payload = payload + + def read(self): + if isinstance(self.payload, bytes): + return self.payload + return json.dumps(self.payload).encode("utf-8") + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, traceback): + return False + + +class ShimHealthProbeTests(unittest.TestCase): + def _payload(self, **overrides): + payload = { + "service": "daaf-anthropic-openai-shim", + "status": "ok", + "backend_mode": "openai", + "version": "1.2.12", + "sanitize_tools": True, + "codex_home_present": True, + } + payload.update(overrides) + return payload + + def _probe(self, route_name, payload): + route = route_detection.RouteInfo(route_name) + with mock.patch.object( + smoke_probes, "urlopen", return_value=_HealthResponse(payload) + ): + return smoke_probes.probe_shim_health(route) + + def _reported_text(self, result): + parts = [result.detail] + for evidence in result.evidence: + parts.extend((evidence.output or "", evidence.note or "")) + return "\n".join(parts) + + def test_exact_service_status_route_mode_and_safe_version_pass(self): + result = self._probe( + route_detection.ROUTE_OPENAI_API, + self._payload(version="1.2.12+build_7-x"), + ) + self.assertEqual(result.verdict, Verdict.PASS, result.detail) + self.assertIn("version=1.2.12+build_7-x", result.detail) + + def test_missing_version_fails_with_bounded_marker(self): + payload = self._payload() + payload.pop("version") + result = self._probe(route_detection.ROUTE_OPENAI_API, payload) + reported = self._reported_text(result) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("version must match", result.detail) + self.assertIn("version=", reported) + + def test_empty_version_fails_with_bounded_marker(self): + result = self._probe( + route_detection.ROUTE_OPENAI_API, + self._payload(version=""), + ) + reported = self._reported_text(result) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("version must match", result.detail) + self.assertIn("version=", reported) + + def test_unsafe_version_fails_without_reflecting_endpoint_text(self): + unsafe_version = "release/unsafe\nprivate-version-marker" + result = self._probe( + route_detection.ROUTE_OPENAI_API, + self._payload(version=unsafe_version), + ) + reported = self._reported_text(result) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("version must match", result.detail) + self.assertIn("version=", reported) + self.assertNotIn(unsafe_version, reported) + self.assertNotIn("private-version-marker", reported) + + def test_overlong_version_fails_without_reflecting_endpoint_text(self): + overlong_version = "A" * 65 + result = self._probe( + route_detection.ROUTE_OPENAI_API, + self._payload(version=overlong_version), + ) + reported = self._reported_text(result) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("version must match", result.detail) + self.assertIn("version=", reported) + self.assertNotIn(overlong_version, reported) + + def test_nonstring_versions_fail_with_bounded_marker(self): + for version in (None, 12, True, ["1.2.12"]): + with self.subTest(version=version): + result = self._probe( + route_detection.ROUTE_OPENAI_API, + self._payload(version=version), + ) + reported = self._reported_text(result) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("version must match", result.detail) + self.assertIn("version=", reported) + + def test_chatgpt_boolean_true_auth_presence_passes(self): + result = self._probe( + route_detection.ROUTE_CHATGPT, + self._payload(backend_mode="chatgpt", codex_home_present=True), + ) + self.assertEqual(result.verdict, Verdict.PASS, result.detail) + + def test_chatgpt_boolean_false_auth_presence_fails(self): + result = self._probe( + route_detection.ROUTE_CHATGPT, + self._payload(backend_mode="chatgpt", codex_home_present=False), + ) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("codex_home_present must be boolean true", result.detail) + + def test_chatgpt_missing_auth_presence_fails(self): + payload = self._payload(backend_mode="chatgpt") + payload.pop("codex_home_present") + result = self._probe(route_detection.ROUTE_CHATGPT, payload) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("codex_home_present must be boolean true", result.detail) + + def test_chatgpt_string_auth_presence_fails_with_bounded_marker(self): + result = self._probe( + route_detection.ROUTE_CHATGPT, + self._payload(backend_mode="chatgpt", codex_home_present="true"), + ) + reported = self._reported_text(result) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("codex_home_present must be boolean true", result.detail) + self.assertIn("codex_home_present=", reported) + + def test_chatgpt_truthy_nonboolean_auth_presence_fails(self): + result = self._probe( + route_detection.ROUTE_CHATGPT, + self._payload(backend_mode="chatgpt", codex_home_present=1), + ) + reported = self._reported_text(result) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("codex_home_present must be boolean true", result.detail) + self.assertIn("codex_home_present=", reported) + + def test_unrelated_service_degraded_status_and_wrong_mode_fail(self): + cases = ( + self._payload(service="unrelated-service"), + self._payload(status="degraded"), + self._payload(backend_mode="chatgpt"), + ) + for payload in cases: + with self.subTest(payload=payload): + result = self._probe(route_detection.ROUTE_OPENAI_API, payload) + self.assertEqual(result.verdict, Verdict.FAIL) + + def test_nonobject_and_malformed_health_json_fail_cleanly(self): + for payload in (["not", "an", "object"], b"{not-json"): + with self.subTest(payload=payload): + result = self._probe(route_detection.ROUTE_OPENAI_API, payload) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("invalid", result.detail) + + class TierDEnvSanitizationTests(unittest.TestCase): def test_removes_exactly_the_two_contaminants_and_preserves_the_rest(self): with mock.patch.dict(os.environ, { diff --git a/user_reference/01_installation_and_quickstart.md b/user_reference/01_installation_and_quickstart.md index fce70e75..49c182dc 100644 --- a/user_reference/01_installation_and_quickstart.md +++ b/user_reference/01_installation_and_quickstart.md @@ -824,8 +824,6 @@ Including R (with the full package set and Quarto) accounts for roughly **2 GB** **To use R**, just tell DAAF "set execution language to R" at the start of a session — no configuration files to edit. See the [R and Language Support FAQ](07_faq_technical.md#r-and-language-support) for details on switching between languages. -The smoke tests and their runner in `scripts/smoke_tests/` exercise each R library skill, plus a Python import smoke (`smoke_imports.py`) covering the pinned Python analysis stack; all run via `run_all_smoke_tests.sh` in any DAAF container. - #### Coming from Stata or R? Get code comments in your language If your background is in **Stata** or **R** rather than Python, you don't have to read unfamiliar code cold. Just tell DAAF your background once — for example, "I'm coming from Stata" or "my background is in R" — and it will offer to save that as a preference. From then on, every piece of analysis code DAAF writes carries inline comments showing the equivalent command in your language: Stata users see the `reghdfe`, `xtreg`, `esttab`, and `svy:` equivalents next to the Python (or R) that runs, and R users see their familiar `tidyverse`/`fixest` equivalents. This is powered by DAAF's built-in translation skills, so it stays accurate to the actual code. The preference persists across sessions, and turning it on is entirely conversational — there are no configuration files to edit and nothing to set up in advance. (Under the hood this sets the cross-language annotation preference described in `CLAUDE.md` § User Preferences; you never have to touch that file yourself.) @@ -957,7 +955,7 @@ The two-tier routing described above works identically: map `ANTHROPIC_DEFAULT_O **Who this is for:** researchers who have an **OpenAI API key** (pay-per-use) and would rather talk to OpenAI directly, with no OpenRouter middle-hop. -**What you'll end up with:** DAAF running on GPT models through a lightweight translation shim that ships built into DAAF. The shim presents an Anthropic-compatible endpoint on `localhost`, quietly forwards your requests to OpenAI, has **zero new dependencies**, and starts automatically inside the container — once it's set up, you won't think about it again. (Would you rather use a **ChatGPT subscription** than a pay-per-use API key? Skip ahead to the [ChatGPT subscription lane](#option-f-alternate-lane-chatgpt-subscription-codex-backend) below.) +**What you'll end up with:** DAAF running on GPT models through a lightweight translation shim that ships built into DAAF. The shim presents an Anthropic-compatible endpoint on `localhost`, forwards your requests to OpenAI, and starts automatically inside the container — once it's set up, you normally won't think about it again. (Would you rather use a **ChatGPT subscription** than a pay-per-use API key? Skip ahead to the [ChatGPT subscription lane](#option-f-alternate-lane-chatgpt-subscription-codex-backend) below.) > **Billing prerequisite — API credits are separate from ChatGPT.** An OpenAI API key only works if its platform.openai.com project has **prepaid credits purchased**: adding a payment card alone is not enough (the $5+ credit purchase is a separate step), and new API accounts receive no free credits. A ChatGPT Plus/Pro subscription does **not** include API access — ChatGPT and the API are separate billing systems, and there is **no OpenAI-sanctioned** way to run a third-party tool like DAAF on a ChatGPT subscription (subscription usage is scoped to OpenAI's official apps; verified against OpenAI's terms and Codex documentation, 2026-07-11). An unfunded key fails with an instant `429 insufficient_quota` on every request — see the [technical FAQ entry on instant 429s](07_faq_technical.md#q-my-gpt-session-fails-instantly-with-429-errors-on-every-request-option-f). DAAF *does* ship a supported alternative that reuses your Codex (ChatGPT) OAuth login to route through your ChatGPT subscription — see the ["ChatGPT subscription lane"](#option-f-alternate-lane-chatgpt-subscription-codex-backend) below. It's a functional, carefully built route, but it works through a backend interface OpenAI doesn't officially offer for third-party tools, so it may change if OpenAI changes the backend, and you are responsible for compliance with OpenAI's terms. The API-key lane described here is the alternative if you'd rather stay on an officially offered interface. @@ -986,10 +984,13 @@ CLAUDE_CODE_MAX_CONTEXT_TOKENS=1050000 # see "Context window on GP **Step 4 — Confirm it's working (optional).** If a GPT session ever misbehaves, a small manager script and a health endpoint let you check the shim at a glance: ```bash -bash /daaf/scripts/provider_shim/start_shim.sh --status # is it running? -curl -s http://127.0.0.1:4141/health # health check +bash /daaf/scripts/provider_shim/start_shim.sh --status # is it running? +bash /daaf/scripts/provider_shim/start_shim.sh --restart # replace it atomically +curl -s http://127.0.0.1:4141/health # health check ``` +`--status` reports "strictly ready" only when `/health` returns DAAF's exact shim service identity, `status: "ok"`, the version derived from the local shim source, and the normalized configured backend mode (`openai` or `chatgpt`). This prevents an old shim—or an unrelated service that happens to answer on port 4141—from looking healthy. Its exit status deliberately distinguishes states: `--status` exits zero for both a cleanly ready shim and a cleanly stopped shim, while it exits nonzero for missing prerequisites or source, a running supervisor whose shim is not strictly ready, or a reachable endpoint whose health contract does not match. Use the single-command `--restart` action after changing shim code or startup settings: it owns stop, wait, replacement launch, and strict readiness under one lifecycle lock. **If the active Claude Code session itself routes through this shim, never ask it to run `--stop` and then `--start` in separate assistant turns**—the stop can sever the route before the later turn exists. Container boot uses the manager's nonfatal `--auto` path: it records one machine-readable `SHIM_AUTO_READY` or `SHIM_AUTO_FAILURE` line and continues boot even if the opt-in shim cannot become ready. Explicit `--start` and `--restart` exit nonzero whenever their requested operation is unsuccessful, so an operator can act. + That's the whole setup. The paragraphs that follow are optional tuning you can safely skip at first, and deeper mechanics and diagnostics live in **Under the hood / troubleshooting reference** at the very end of this lane. **Optional tuning (safe to skip at first).** @@ -1010,11 +1011,19 @@ See the [technical FAQ entry on controlling GPT reasoning effort](07_faq_technic *You can safely skip this subsection — it's here for when you want to look under the hood or diagnose a problem. These mechanics are shared by both shim lanes (this one and the ChatGPT lane below).* -**Shim controls, logs, and tuning variables.** The manager script (`start_shim.sh`, shown in Step 4) also accepts `--start`, `--stop`, and `--auto`. The shim's log lives at `/daaf/scripts/provider_shim/logs/shim.log`. Backend failures are logged as structural metadata — status, exact structured type/code, mapped Anthropic type, retry/attempt information, phase, and safe allowlisted headers/request IDs. Free-form backend response bodies and messages are not retained, because provider prose can reflect request content; downstream client errors use bounded, classification-derived text instead. The [technical FAQ](07_faq_technical.md#q-my-gpt-session-fails-instantly-with-429-errors-on-every-request-option-f) walks through status/type/code triage such as `insufficient_quota` (billing) versus a true rate limit. Defaults are fine for almost everyone; the tuning variables (`SHIM_PORT`, `SHIM_BACKEND_BASE_URL`, `SHIM_BACKEND_API_KEY`, `SHIM_STRIP_MODEL_PREFIX`, `SHIM_SANITIZE_TOOLS`, `SHIM_REASONING_EFFORT`, `SHIM_TEXT_VERBOSITY`) are documented in `environment_settings_example.txt`. +**Shim controls, logs, and tuning variables.** The manager script (`start_shim.sh`, shown in Step 4) accepts `--start`, `--restart`, `--stop`, `--status`, and `--auto`. The shim's log lives at `/daaf/scripts/provider_shim/logs/shim.log`. Manager, supervisor, and Python records are serialized through one append path, so concurrent writers cannot interleave or duplicate stored records. Rotation happens before a new supervisor starts and only when the active log is **larger than 25 MiB**; DAAF retains five generations. The log directory and lock directories are mode `0700`, active and retained log files are mode `0600`, and startup refuses symlink or non-regular log/state targets instead of following them. The manager also verifies saved PID/PGID identity before signaling; when `setsid` isolation is unavailable it falls back to exact per-process identity checks rather than killing a shared process group. Restart emits one bounded `SHIM_RESTART_BEGIN` record and one `SHIM_RESTART_RESULT` record, with distinct stop, launch, and readiness failure outcomes. + +Backend failures are logged as structural metadata — status, exact structured type/code, mapped Anthropic type, retry/attempt information, phase, and safe allowlisted request IDs. Free-form backend response bodies and messages are not retained, because provider prose can reflect request content; downstream client errors use bounded, classification-derived text instead. The [technical FAQ](07_faq_technical.md#q-my-gpt-session-fails-instantly-with-429-errors-on-every-request-option-f) walks through status/type/code triage such as `insufficient_quota` (billing) versus a true rate limit. Defaults are fine for almost everyone; the tuning variables (`SHIM_PORT`, `SHIM_BACKEND_BASE_URL`, `SHIM_BACKEND_API_KEY`, `SHIM_STRIP_MODEL_PREFIX`, `SHIM_SANITIZE_TOOLS`, `SHIM_REASONING_EFFORT`, `SHIM_TEXT_VERBOSITY`) are documented in `environment_settings_example.txt`. + +**Images.** The shim accepts image blocks in user messages and image-bearing tool results, translating them to the public OpenAI Responses `input_image` shape while preserving the original text/image order. Sources may be valid Base64 with an exact MIME type of JPEG, PNG, GIF, or WebP, or a fully qualified `http://` or `https://` URL. Text-only tool results keep the established string-output behavior; the list form is used only when a result contains an image. The shim validates structure without inspecting or logging image bytes, and it does **not** claim to detect animated GIFs. Unsupported or malformed forms fail locally with `400 invalid_request_error` before any backend call; the [image troubleshooting entry](07_faq_technical.md#q-what-image-inputs-does-the-provider-shim-support) lists the exact rejection cases. + +This translation is grounded in OpenAI's public `/v1/responses` contract. On 2026-07-18, one privacy-safe live request sent a synthetic one-pixel Base64 PNG through the ChatGPT/Codex lane and received HTTP 200, an Anthropic-shaped response containing text, and a DAAF request ID; unique prompt and Base64 sentinels were absent from all retained shim logs checked by the gate. That is evidence that this specific Base64-PNG shape was compatible on that date, not an official OpenAI guarantee. URL images and other MIME/source variants were not part of the live gate. **Tool-call sanitization.** This is **on by default** (`SHIM_SANITIZE_TOOLS`): the shim silently strips known GPT tool-call quirks (empty `pages` parameters, `isolation` fills on subagent dispatches, redundant sandbox flags) that would otherwise each cost a wasted error round-trip; every strip is recorded in the shim log. Set `SHIM_SANITIZE_TOOLS=0` (and restart the shim — the flag is read at startup) when running DAAFBench against shim-routed models, which must observe raw model behavior; confirm via the `/health` endpoint's `sanitize_tools` field. -**Correlated request diagnostics.** Every JSON or SSE response from `POST /v1/messages` carries an `x-daaf-request-id` response header containing an internally generated 32-character hexadecimal ID. The same value appears as `req_id=<32-hex-id>` on that request's shim and HTTP-client log records, so saving the header lets you reconstruct one request without matching unrelated concurrent traffic. The stable lifecycle event names are `request_start`, `request_parsed`, `upstream_attempt`, `upstream_retry`, `upstream_headers`, `upstream_first_event`, `downstream_first_content`, `backend_error`, `disconnect`, `terminal`, and `cleanup`; retry events can repeat and some milestones apply only to streaming or failure paths, but every `/v1/messages` lifecycle has exactly one `event=terminal` summary and one `event=cleanup` record. These records expose diagnostic metadata rather than content (`phase`, monotonic `elapsed_ms`/`dur_ms`, attempt and retry counts, allowlisted upstream request-ID metadata such as `upstream_req_id_header` and `upstream_req_id`, normalized `http_version`, failure phase and structured error type/code, disconnect state, `terminal_frame_send`, `body_close_send`, and cleanup status/error); prompts, request message bodies, translated payloads, tool schemas and inputs, raw SSE content, free-form backend response bodies/messages, credentials, and arbitrary headers are never logged, and upstream IDs never replace the local `x-daaf-request-id` correlation value. For a safe request-ID grep workflow and a field-by-field mid-response diagnostic, see [How do I diagnose a provider-shim server error mid-response?](07_faq_technical.md#q-how-do-i-diagnose-a-provider-shim-server-error-mid-response). +**Correlated request diagnostics.** Every JSON or SSE response from `POST /v1/messages` carries an `x-daaf-request-id` response header containing an internally generated 32-character hexadecimal ID. The same value appears as `req_id=<32-hex-id>` on that request's shim and HTTP-client log records, so saving the header lets you reconstruct one request without matching unrelated concurrent traffic. Where available, the lifecycle also distinguishes the client request ID sent upstream from the provider request ID returned in response headers; neither replaces the local correlation ID. The stable lifecycle event names are `request_start`, `request_parsed`, `upstream_attempt`, `upstream_retry`, `upstream_headers`, `upstream_first_event`, `downstream_first_content`, `backend_error`, `disconnect`, `terminal`, and `cleanup`; retry events can repeat and some milestones apply only to streaming or failure paths, but every `/v1/messages` lifecycle has exactly one `event=terminal` summary and one `event=cleanup` record. + +These records expose bounded diagnostic metadata rather than content: transport exception class and failure phase, monotonic `elapsed_ms`/`dur_ms` timings, attempt and retry counts, retry source/reason, allowlisted client/provider request IDs, normalized `http_version`, structured error type/code, disconnect state, terminal/body-close send state, and cleanup status/error. They **never log** prompts or text, tool inputs or arguments, Base64 or decoded image bytes, image URLs or query strings, file IDs or names, credentials or cookies, raw SSE, complete request/response bodies, or free-form backend prose. For a safe request-ID grep workflow and a field-by-field mid-response diagnostic, see [How do I diagnose a provider-shim server error mid-response?](07_faq_technical.md#q-how-do-i-diagnose-a-provider-shim-server-error-mid-response). **Reading the evidence carefully.** The terminal send fields are deliberately narrow evidence. `send_completed` means only that the awaited ASGI `send` call returned; it does **not** prove that the client received or acknowledged the bytes. Likewise, `event=disconnect` records that the shim observed an ASGI `http.disconnect` event at `observed_phase`; it is not proof of what the user received or displayed, does not establish that the disconnect caused an upstream operation to be cancelled, and does not imply that an upstream operation existed. Generic ASGI task cancellation is not itself recorded as client-disconnect evidence. Read the correlated events in order before drawing a causal conclusion: for example, a structured backend `server_error` followed by `disconnect` is evidence that the shim observed the backend-declared error first, not proof that the later disconnect caused it or that the terminal error reached the client. diff --git a/user_reference/07_faq_technical.md b/user_reference/07_faq_technical.md index f5a65a91..79002578 100644 --- a/user_reference/07_faq_technical.md +++ b/user_reference/07_faq_technical.md @@ -283,11 +283,31 @@ Restart the container after editing `environment_settings.txt`. The shim auto-starts from the container entrypoint and is kept alive by a supervisor, so it should already be running. To diagnose: ```bash -bash /daaf/scripts/provider_shim/start_shim.sh --status # is it running? -curl -s http://127.0.0.1:4141/health # health check +bash /daaf/scripts/provider_shim/start_shim.sh --status # is it running? +bash /daaf/scripts/provider_shim/start_shim.sh --restart # atomic repair/reload +curl -s http://127.0.0.1:4141/health # health check ``` -Its log is at `/daaf/scripts/provider_shim/logs/shim.log` — check there first. The manager script also accepts `--start`, `--stop`, and `--auto`. Remember that Option F requires the image to have been **rebuilt** after you set `DAAF_PROVIDER_SHIM=openai` (the auto-launch is baked into the entrypoint), so if nothing is running, confirm you rebuilt. If the shim *is* running but every request fails instantly with a 429, see the next entry. +Its log is at `/daaf/scripts/provider_shim/logs/shim.log` — check there first. `--status` calls the shim strictly ready only when `/health` matches the exact DAAF service identity, `status: "ok"`, the version derived from the installed source, and the normalized configured backend mode. Its exit status deliberately distinguishes states: a cleanly ready shim and a cleanly stopped shim both exit zero, while missing prerequisites or source, a running supervisor whose shim is not strictly ready, and a reachable endpoint whose health contract does not match all exit nonzero. A machine-readable `SHIM_READINESS_FAILURE` line identifies a contract mismatch; container boot's deliberately nonfatal `--auto` path records `SHIM_AUTO_READY` or `SHIM_AUTO_FAILURE` and continues boot. Explicit `--start` and `--restart` exit nonzero whenever their requested operation is unsuccessful, so an operator can act. + +For an in-place repair or reload, use `--restart`, which keeps stop → wait → launch → strict readiness inside one locked manager invocation and records one `SHIM_RESTART_BEGIN` plus one `SHIM_RESTART_RESULT`. This is required when the active Claude Code session itself uses the shim route. Do **not** split `--stop` and `--start` across separate assistant turns: after the stop, the session may no longer have a working provider route with which to issue the start. + +The manager serializes concurrent `--restart`, `--start`, and `--stop` actions and sends signals only after exact process-identity checks. It rotates only when `shim.log` is larger than 25 MiB and retains five generations; active and retained logs are private (`0600`), while the log and lock directories are `0700`. It refuses symlink/non-regular targets. If `setsid` process-group isolation is unavailable, teardown uses verified per-PID signaling instead of risking the caller's shared group. Restart failures identify the stop, launch, or readiness stage and never report success before the complete strict health contract passes. + +Remember that Option F requires the image to have been **rebuilt** after you set `DAAF_PROVIDER_SHIM=openai` (the auto-launch is baked into the entrypoint), so if nothing is running, confirm you rebuilt. If the shim *is* running but every request fails instantly with a 429, see the relevant entry below. + +### Q: What image inputs does the provider shim support? + +For the direct OpenAI API lane, the shim supports images in **user messages** and in **tool results**. It accepts two source forms: + +- valid Base64 paired with exactly one of `image/jpeg`, `image/png`, `image/gif`, or `image/webp`; or +- a fully qualified `http://` or `https://` URL with no embedded credentials. + +The shim emits the public OpenAI Responses `input_image` shape and preserves the original order of text and image blocks. Image-bearing tool results use the Responses item-list form; text-only tool results retain the existing string-output behavior. The shim validates Base64 syntax and structure but deliberately does not decode or inspect image content, so it does **not** claim to identify animated GIFs. + +A malformed or fidelity-unsafe form fails locally, before any backend request, as HTTP `400` with `invalid_request_error`. That includes malformed Base64; an unsupported MIME type or source type; an unsafe/non-HTTP(S) URL; a provider-scoped `file` source; an image in assistant history; an unknown history or tool-result block; and a malformed message/content block. Errors identify structural coordinates and the failure class, not the image, URL, file identifier, or surrounding text. + +The public `/v1/responses` image contract is the documented compatibility basis. On 2026-07-18, one privacy-safe live request sent a synthetic one-pixel Base64 PNG through the ChatGPT/Codex lane and received HTTP 200, an Anthropic-shaped response containing text, and a DAAF request ID. The gate also checked retained shim logs and found neither its unique prompt sentinel nor its unique Base64 sentinel. This is compatibility evidence for that specific Base64-PNG shape on that date, **not** an official OpenAI guarantee. URL images and other MIME/source variants were not part of the live gate. ### Q: How do I diagnose a provider-shim server error mid-response? @@ -307,12 +327,12 @@ The first grep reconstructs the complete correlated lifecycle. Stable event name 1. **Inspect `event=backend_error`.** `backend_type`, `backend_code`, `anthropic_type`, and `failure_phase` identify the structured or transport failure the shim recorded. Error mapping follows an explicit precedence: a real non-2xx backend HTTP status maps first and takes precedence; otherwise backend `code` is checked before backend `type`. At either structured field, `context_length_exceeded` maps to Anthropic `invalid_request_error`, `server_error` maps to `api_error`, and an unknown value remains `api_error`. Free-form backend response bodies/messages are not retained or reflected because provider prose can contain request-derived material; the user-facing message is bounded, classification-derived text selected from status/type/code families or a fixed unknown fallback. Do not infer that every failure is backend-originated merely because the user saw an `api_error`: local translation, send, disconnect, and cleanup failures have distinct evidence. 2. **Inspect `event=disconnect`, if present.** This event is narrow evidence that the shim observed an ASGI `http.disconnect`; `observed_phase` records where it was observed. It does not prove what the user received or displayed, that the observation caused an upstream operation to be cancelled, or even that an upstream operation existed. Generic ASGI task cancellation is not itself evidence of client disconnect and does not create this event. If a backend-declared error appears earlier, a later disconnect is a subsequent observed condition, not evidence that the disconnect caused the backend error. If no backend failure was known first, the terminal outcome can instead be `disconnect`. -3. **Inspect the one `event=terminal` summary.** Check `outcome`, `attempts`, `retries`, `failure_phase`, `terminal_frame_send`, `body_close_send`, `disconnect`, and `disconnect_phase`. The send states are `not_attempted`, `attempted`, `send_completed`, `skipped_disconnect`, or `write_failed`. Critically, `send_completed` means only that the awaited ASGI send returned; it does **not** establish client receipt, acknowledgment, or guaranteed delivery. A `write_failed` state is direct evidence of a downstream send failure; other conclusions about what the client displayed remain inference. -4. **Inspect the one `event=cleanup` record.** `status=completed` records clean shim-owned resource cleanup; `status=failed` plus the bounded `error` type records a cleanup failure. Cleanup evidence does not overwrite an earlier success, backend error, or disconnect outcome. +3. **Inspect the one `event=terminal` summary.** Check `outcome`, `attempts`, `retries`, `retry_reason`, `retry_source`, `transport_exception`, `failure_phase`, `terminal_frame_send`, `body_close_send`, `disconnect`, and `disconnect_phase`. The send states are `not_attempted`, `attempted`, `send_completed`, `skipped_disconnect`, or `write_failed`. Critically, `send_completed` means only that the awaited ASGI send returned; it does **not** establish client receipt, acknowledgment, or guaranteed delivery. A `write_failed` state is direct evidence of a downstream send failure; other conclusions about what the client displayed remain inference. +4. **Inspect the one `event=cleanup` record.** `status=completed` records clean shim-owned resource cleanup; `status=failed` plus the bounded `error` type records a cleanup failure. Cleanup evidence does not overwrite an earlier success, backend error, or disconnect outcome. Exactly-once guards make terminal and cleanup records stable accounting boundaries even when an exception or cancellation crosses finalization. -Additional safe fields can help locate the failing phase without revealing request content: `phase`, monotonic `elapsed_ms`/`dur_ms`, attempts/retries, the allowlisted upstream request-ID fields `upstream_req_id_header` and `upstream_req_id`, normalized `http_version`, disconnect state, terminal semantic-frame state, body-close state, and cleanup status. The local `req_id` remains authoritative for joining shim records; an upstream request ID is separate metadata and never replaces it. +Additional safe fields can help locate the failing phase without revealing request content: transport exception class and phase, monotonic `elapsed_ms`/`dur_ms`, attempt/retry counts and retry source/reason, the sent `client_req_id` and returned allowlisted provider ID (`upstream_req_id_header` / `upstream_req_id`) where available, normalized `http_version`, disconnect state, terminal semantic-frame state, body-close state, and cleanup status. The local `req_id` remains authoritative for joining shim records; client/provider IDs are separate metadata and never replace it. -The lifecycle and diagnostic logs intentionally omit prompts, message bodies, translated payloads, tool schemas and inputs, raw SSE content, free-form backend response bodies/messages, credentials, and arbitrary headers. Do not add those values to bug-report grep commands. Retry policy, timeouts, backoff, connection pooling, and heartbeat behavior are independent of this logging, so a new correlated failure should be diagnosed from the recorded sequence before proposing transport-policy changes. +The lifecycle and diagnostic logs intentionally **never include** prompts or text, tool inputs or arguments, Base64 or decoded image bytes, image URLs or query strings, file IDs or names, credentials or cookies, raw SSE, complete request/response bodies, or free-form backend prose. Do not add those values to bug-report grep commands. Retry policy, timeouts, backoff, connection pooling, and heartbeat behavior are independent of this logging, so a new correlated failure should be diagnosed from the recorded sequence before proposing transport-policy changes. ### Q: My GPT session fails instantly with 429 errors on every request (Option F) @@ -341,7 +361,7 @@ The shim reads the OAuth token from `auth.json` and **refreshes it automatically **Parallel use.** Running more than one container is the clean way to parallelize: each container does its own `codex login` (an independent token grant — no collision), and they share only your ChatGPT usage pool, so running them in parallel just draws that pool down faster. For several codex tools inside a *single* container, give each its own login under a separate `CODEX_HOME` to avoid a rare token-rotation race. -**Under the hood (safe to skip).** Request translation, tools, and the content-block lifecycle are shared across both lanes; only authentication and endpoint differ. On this lane the shim always requests upstream SSE — including for inbound non-stream calls, whose terminal response is accumulated into JSON — and malformed/missing terminals or invalid block ordering fail explicitly. Tool-call events missing optional identity fields (which the undocumented Codex backend may omit) degrade gracefully with grep-stable `wire-divergence:` warnings in the shim log instead of failing the turn, and an invalid usage token counter is dropped rather than discarding a completed response. One response-formatting rule is ChatGPT-only: reliably identified reasoning-summary part boundaries are preserved as blank lines (`\n\n`), while OpenAI/API-key mode retains exact concatenation and its real upstream non-stream JSON path. +**Under the hood (safe to skip).** Request translation, tools, and the content-block lifecycle are shared across both lanes; only authentication and endpoint differ. On this lane the shim always requests upstream SSE — including for inbound non-stream calls, whose terminal response is accumulated into JSON — and malformed/missing terminals or invalid block ordering fail explicitly. The tolerated live `arguments.done` event that omits `name` no longer produces one warning per event: the shim logs the first occurrence and then periodic aggregate summaries with counts. Hard or conflicting protocol errors remain visible and still fail explicitly; an invalid usage token counter is dropped rather than discarding a completed response. One response-formatting rule is ChatGPT-only: reliably identified reasoning-summary part boundaries are preserved as blank lines (`\n\n`), while OpenAI/API-key mode retains exact concatenation and its real upstream non-stream JSON path. See the full walkthrough in [**01. Installation & Quick Start — Option F, alternate lane: ChatGPT subscription**](01_installation_and_quickstart.md#option-f-alternate-lane-chatgpt-subscription-codex-backend). @@ -463,7 +483,7 @@ Cost remains a meaningful barrier to entry for DAAF, but it's shrinking. As open ### Q: How much disk space does DAAF use? -The Docker image is roughly **8.6 GB** after building (the exact size varies with your Docker version and platform). It includes an Ubuntu 24.04 (Noble) base image, Python 3.12, 46 pinned Python packages (data science, geospatial, econometrics, visualization, ML), geospatial system libraries (GDAL/GEOS/PROJ), Claude Code, R, 60+ pinned R packages (tidyverse, fixest, survey, sf, and more), and the Quarto CLI. The R runtime, packages, and Quarto account for roughly **2 GB** of that total (approximately 8.6 GB with R versus 6.4 GB without). Docker also keeps build cache layers, so total Docker disk usage may be somewhat higher. +The Docker image is roughly **8.6 GB** after building (the exact size varies with your Docker version and platform). It includes an Ubuntu 24.04 (Noble) base image, Python 3.12, **52 explicitly pinned Python distributions** — 50 with import targets plus metadata-only `svy-rs==0.10.0` and `svy-io==0.1.1` (alongside the provider shim's exact `httpx==0.28.1` and `uvicorn==0.51.0` runtime pins) — covering data science, geospatial, econometrics, visualization, ML, and shim infrastructure; geospatial system libraries (GDAL/GEOS/PROJ); Claude Code; R; 60+ pinned R packages (tidyverse, fixest, survey, sf, and more); and the Quarto CLI. The R runtime, packages, and Quarto account for roughly **2 GB** of that total (approximately 8.6 GB with R versus 6.4 GB without). Docker also keeps build cache layers, so total Docker disk usage may be somewhat higher. Beyond the image, your Docker volume will grow as you create research projects. Each project accumulates scripts, parquet data files, session logs, and notebooks. A typical full-pipeline project might add 50-500 MB depending on how many datasets you fetch and how large they are. From 0e5c71fb34efcba03c8cdc422f6c140ea93e40fb Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 11:57:24 +0000 Subject: [PATCH 06/66] Minor adjustment of dockerfile for shim dependencies --- Dockerfile | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/Dockerfile b/Dockerfile index cc565f35..8a593e3f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -486,6 +486,17 @@ RUN uv pip install --system \ RUN uv pip install --system \ faker==40.31.0 +# Install provider-shim direct runtime dependencies after all other framework +# Python packages. Both arrive transitively today (httpx via svy; uvicorn via +# marimo), but the persistent shim imports them directly and must not depend on +# unrelated packages retaining compatible versions. Keeping these direct pins +# last in the Python stack also prevents a later framework Python block from +# replacing them. The build is still root here; the single runtime transition to +# appuser remains at the existing security boundary below. +RUN uv pip install --system \ + httpx==0.28.1 \ + uvicorn==0.51.0 + # ============================================ # Install R Data Science Packages + Quarto # ============================================ @@ -789,18 +800,6 @@ ARG CLAUDE_CODE_VERSION=2.1.202 RUN curl -fsSL https://claude.ai/install.sh | bash -s ${CLAUDE_CODE_VERSION} ENV PATH="/home/appuser/.local/bin:${PATH}" -# Provider-shim direct runtime dependencies. These were already installed -# transitively (httpx via svy; uvicorn via marimo), but the persistent shim imports -# them directly, so its contract must not depend on unrelated packages retaining -# those dependencies. Keep this framework-owned pin layer late: neither package is -# needed to build the Python/R stacks above, and placing it here preserves Docker's -# expensive-layer cache when only the shim runtime pins change. -USER root -RUN uv pip install --system \ - httpx==0.28.1 \ - uvicorn==0.51.0 -USER appuser - # ============================================ # USER ADDITIONS — add your own packages and tools here # ============================================ From 3c58840f1e3ba456fe5bad71dfea72efc88051c7 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 14:26:51 +0000 Subject: [PATCH 07/66] docs: restore approachable user voice in shim and threshold documentation Compress implementation-spec detail that recent provider-shim commits imported into user-facing docs, while preserving all load-bearing facts (shim commands and the stop/start session-safety caution, context ceiling values 1050000/370000, image formats and dated probe evidence, ~52 package count, reproducibility verdicts, evidence-grading guidance). Setup steps and troubleshooting entries now lead with actions; "safe to skip" blocks are reframed as bold "Optional reading" leads; the duplicated threshold profile-membership table is dropped from 02_understanding_daaf.md; Quarto-viewing passages are trimmed to parity with their marimo equivalents; CONTRIBUTING release-gate invariants are restructured into a scannable checklist. Co-Authored-By: Claude Fable 5 --- CONTRIBUTING.md | 8 ++- .../01_installation_and_quickstart.md | 41 ++++++------- user_reference/02_understanding_daaf.md | 22 +++---- user_reference/03_best_practices.md | 9 ++- user_reference/07_faq_technical.md | 58 ++++++------------- 5 files changed, 58 insertions(+), 80 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a7ed7f16..b429a7e3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -436,7 +436,13 @@ If your change touches how DAAF talks to a model provider -- a new or changed AP **Provider-shim changes have a stricter release gate.** A reviewable shim PR should show all of the following, with commands and observed results rather than a remembered pass claim: 1. **Production-path deterministic coverage.** Exercise the real translator/ASGI/subprocess paths in `tests/provider_shim/`, not a duplicate implementation inside a test. Run the provider-shim suite, the lifecycle suite at `tests/bash/start_shim.bats`, the DAAFBench route/provenance tests under `benchmarks/tests/`, and the deploy-smoke harness self-tests at `tests/python/test_deploy_smoke.py`. -2. **Lifecycle and privacy invariants.** Preserve strict readiness (exact service identity, `ok` status, source-derived version, normalized backend mode), the nonfatal machine-readable `--auto` boot contract, serialized `--restart`/`--start`/`--stop` ownership, exactly-once restart begin/result records with distinct stop/launch/readiness failures, serialized single-write logging, private bounded retention, safe PID/PGID identity checks, and exactly-once terminal/cleanup accounting. Diagnostics may record local/client/provider request IDs where available, transport class/phase, bounded timings, and retry attribution—but never prompts/text, tool inputs/arguments, image data or URLs/query strings, file IDs/names, credentials/cookies, raw SSE, complete bodies, or free-form backend prose. +2. **Lifecycle and privacy invariants.** A reviewer should be able to confirm each of these still holds — each is stated with why it matters: + - **Strict readiness** (exact service identity, `ok` status, source-derived version, normalized backend mode) — so a stale shim, or an unrelated service answering on the port, can't masquerade as healthy. + - **Nonfatal `--auto` boot contract** — container boot records one machine-readable result and continues even when the opt-in shim can't become ready, so a shim problem never blocks the container. + - **Serialized lifecycle ownership** — `--restart`/`--start`/`--stop` run under one lock, and a restart emits exactly-once begin/result records with distinct stop, launch, and readiness-failure outcomes, so concurrent invocations can't collide and every restart is auditable. + - **Safe process signaling** — PID/PGID identity checks ensure the manager only signals the process it actually started, never a shared or unrelated one. + - **Serialized, bounded, private logging** — a single append path prevents interleaved or duplicated records, retention is size-bounded, and every request lifecycle has exactly-once terminal and cleanup accounting. + - **The privacy never-list** — diagnostics may record request IDs (local/client/provider), transport class/phase, bounded timings, and retry attribution, but **never** prompts or text, tool inputs/arguments, image data or URLs/query strings, file IDs/names, credentials/cookies, raw SSE, complete bodies, or free-form backend prose. 3. **Image-contract evidence.** User-message and tool-result images must preserve source ordering and fail locally on unsupported or malformed content rather than silently dropping it. The public OpenAI Responses schema is the documented basis. The private ChatGPT/Codex wire is not an official image guarantee: fixtures for it must come from observed live evidence, never a guessed documentation shape, and the capture process must not log payloads. Run a privacy-safe live image probe on that lane before claiming ChatGPT image compatibility. 4. **Image and deployment validation.** If Docker dependencies change, run `scripts/smoke_tests/run_all_smoke_tests.sh`; its Python gate verifies 50 import targets plus two metadata-only distributions — 52 explicit pinned distributions in total. It preserves exact-version checks for `httpx==0.28.1` and `uvicorn==0.51.0` and uses metadata-only presence/exact-version checks for `svy-rs==0.10.0` and `svy-io==0.1.1` rather than inventing import names. After deploying the changed shim, restart it with the single atomic command `bash /daaf/scripts/provider_shim/start_shim.sh --restart`, confirm the source version through strict `/health`, then run deployment-smoke Tier 0 and Tier 1 against the active route. When the current Claude Code session itself routes through the shim, never schedule `--stop` and `--start` as separate assistant turns—the first turn can remove the transport needed for the second. Deterministic success is necessary, but it does not substitute for this deployed round-trip. diff --git a/user_reference/01_installation_and_quickstart.md b/user_reference/01_installation_and_quickstart.md index 49c182dc..fed6f5bb 100644 --- a/user_reference/01_installation_and_quickstart.md +++ b/user_reference/01_installation_and_quickstart.md @@ -947,7 +947,7 @@ The two-tier routing described above works identically: map `ANTHROPIC_DEFAULT_O > **GPT-5.6 tier naming:** the GPT-5.6 family uses OpenAI's tier-analog names — *Sol* (strong; Opus-analog), *Terra* (mid; Sonnet-analog), and *Luna* (economy; Haiku-analog) — all with 1,050,000-token windows. DAAF's recommended defaults are Sol and Terra, validated live through the full agentic stack (including subagent dispatch on the tier remaps) on 2026-07-09, the day of the OpenRouter release. > -> **Avoid the `-pro` slugs via OpenRouter (Option C):** despite listing the same 1,050,000-token windows and identical pricing, `gpt-5.6-sol-pro` / `-terra-pro` / `-luna-pro` fail with hard "Prompt is too long" API errors once a session's real context reaches roughly 50k tokens — through the Anthropic-compatible endpoint their token accounting runs ~4x the non-pro count against an enforced ~200k ceiling, and the inflation also makes them bill ~2-4x more for identical work. Empirically verified across all three -pro variants in the 2026-07-10 DAAFBench smoke battery (see `benchmarks/README.md` § 1). The non-pro slugs are unaffected. +> **Avoid the `-pro` slugs via OpenRouter (Option C) — they fail and cost more.** `gpt-5.6-sol-pro` / `-terra-pro` / `-luna-pro` error out with hard "Prompt is too long" failures once a session grows past roughly 50k tokens, and they bill ~2-4x more for identical work (their token accounting runs ~4x the non-pro count against an enforced ~200k ceiling). Empirically verified across all three -pro variants in the 2026-07-10 DAAFBench smoke battery (see `benchmarks/README.md` § 1); the non-pro slugs are unaffected. **Restart the container** (`docker compose down`, then `bash run_daaf.sh`) to pick up the changes. No rebuild is needed for OpenRouter — it is config-only. @@ -989,11 +989,11 @@ bash /daaf/scripts/provider_shim/start_shim.sh --restart # replace it atomically curl -s http://127.0.0.1:4141/health # health check ``` -`--status` reports "strictly ready" only when `/health` returns DAAF's exact shim service identity, `status: "ok"`, the version derived from the local shim source, and the normalized configured backend mode (`openai` or `chatgpt`). This prevents an old shim—or an unrelated service that happens to answer on port 4141—from looking healthy. Its exit status deliberately distinguishes states: `--status` exits zero for both a cleanly ready shim and a cleanly stopped shim, while it exits nonzero for missing prerequisites or source, a running supervisor whose shim is not strictly ready, or a reachable endpoint whose health contract does not match. Use the single-command `--restart` action after changing shim code or startup settings: it owns stop, wait, replacement launch, and strict readiness under one lifecycle lock. **If the active Claude Code session itself routes through this shim, never ask it to run `--stop` and then `--start` in separate assistant turns**—the stop can sever the route before the later turn exists. Container boot uses the manager's nonfatal `--auto` path: it records one machine-readable `SHIM_AUTO_READY` or `SHIM_AUTO_FAILURE` line and continues boot even if the opt-in shim cannot become ready. Explicit `--start` and `--restart` exit nonzero whenever their requested operation is unsuccessful, so an operator can act. +In short: `--status` tells you whether the shim is healthy, and `--restart` safely replaces it in one step (it stops the old shim, waits, launches a fresh one, and confirms it's ready). Reach for `--restart` whenever you change shim settings or a session misbehaves. **One important caution: if the active Claude Code session itself routes through this shim, never ask it to run `--stop` and then `--start` in separate turns**—the stop severs your own session's route before the later turn can run. Always use `--restart`, which does both under a single lock. -That's the whole setup. The paragraphs that follow are optional tuning you can safely skip at first, and deeper mechanics and diagnostics live in **Under the hood / troubleshooting reference** at the very end of this lane. +That's the whole setup. The paragraphs that follow are optional reading you can come back to when you need it — and deeper mechanics and diagnostics live in **Under the hood / troubleshooting reference** at the very end of this lane. -**Optional tuning (safe to skip at first).** +**Optional tuning.** **Reasoning effort.** The shim always sets the OpenAI request's reasoning effort, resolved by a four-tier precedence chain — first present wins: (1) a per-request signal from Claude Code, (2) a `#` suffix you append to a model slug (e.g. `ANTHROPIC_DEFAULT_SONNET_MODEL=gpt-5.6-terra[1m]#medium`; it works alongside the `[1m]` window hint and is stripped before the request reaches OpenAI), (3) the `SHIM_REASONING_EFFORT` env var, and (4) the default `high` (parity with DAAF Claude sessions). Valid values are `none`, `low`, `medium`, `high`, `xhigh`, and `max` (`max` is gpt-5.6-only). Leave everything unset to get `high` everywhere; set the env var or a slug suffix only if you want a different level. @@ -1003,31 +1003,25 @@ See the [technical FAQ entry on controlling GPT reasoning effort](07_faq_technic **Response verbosity.** Separately from reasoning effort, the shim sends OpenAI's `text.verbosity` control on every request, defaulting to `high` for parity with DAAF's warm, educational posture (`high` adds warmth and volume; `low` is terse); set `SHIM_TEXT_VERBOSITY=low` or `medium` in `environment_settings.txt` if GPT responses feel too long, and see the [technical FAQ entry on terse GPT responses](07_faq_technical.md#q-gpt-responses-feel-terse-compared-to-claude-option-f) if they feel too brief. -**Context window on GPT sessions.** Claude Code enforces its context-window budget *locally*, and for a model slug it does not recognize it assumes a small (~200K) window — far below the 1,050,000-token physical window of the gpt-5.6 family on API/OpenRouter routes. On the direct OpenAI-API shim route, **append `[1m]` to your bare GPT slugs** in the model variables (e.g. `ANTHROPIC_DEFAULT_OPUS_MODEL=gpt-5.6-sol[1m]`, as shown above) and declare `CLAUDE_CODE_MAX_CONTEXT_TOKENS=1050000`: Claude Code reads `[1m]` as a 1M-window hint and strips the suffix before sending, while the explicit declaration aligns Claude Code and DAAF's accounting denominator. On OpenRouter, keep the provider-prefixed **bare** slug (e.g. `ANTHROPIC_MODEL=openai/gpt-5.6-sol`) and declare `CLAUDE_CODE_MAX_CONTEXT_TOKENS=1050000` explicitly; `[1m]` is not established for those provider-prefixed slugs and must not be generalized across routes. DAAF does **not** use `CLAUDE_CODE_AUTO_COMPACT_WINDOW` as an alternative — automatic compaction remains disabled because it can disrupt orchestration. Separately, **the shim calibrates its token-count estimates against the real backend counts**: it learns the true tokens-per-byte ratio from each backend response and estimates slightly *under* the real count, so if a limit is ever reached the failure mode is a loud, recoverable backend error rather than a silent client-side stop ("Prompt is too long") well below the real window. If a GPT session still reports "Context limit reached" / "Prompt is too long" at low utilization, see the [technical FAQ entry on low-utilization context errors](07_faq_technical.md#q-my-gpt-session-says-context-limit-reached--prompt-is-too-long-at-low-utilization). **The [ChatGPT-subscription (Codex) lane below](#option-f-alternate-lane-chatgpt-subscription-codex-backend) is different:** its backend enforces a much lower effective input ceiling — measured at ~370,000 tokens for `gpt-5.6-sol` (2026-07-16) — that no client hint can raise. On that lane, set `CLAUDE_CODE_MAX_CONTEXT_TOKENS=370000` so Claude Code and DAAF account against the measured backend ceiling. DAAF uses that denominator for utilization, statusline rendering, severity, and behavioral stop/restart guidance; it does not enable automatic compaction or implement a transport-level request blocker, and the backend remains the ultimate hard ceiling. +**Context window on GPT sessions.** Two small settings tell Claude Code how big your GPT model's context window really is — without them it assumes a small (~200K) window for slugs it doesn't recognize, far below the 1,050,000-token window of the gpt-5.6 family. -**Optional 64K Claude Code output budget.** Claude Code defaults `CLAUDE_CODE_MAX_OUTPUT_TOKENS` to `32000` and supports a maximum of `64000`. If unusually long specialist work is being cut off, you may opt in by adding the plain decimal `CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000` (the form supported by the pinned Claude Code 2.1.202) to the **host** `daaf-docker/environment_settings.txt` before Claude Code/the container starts, then recreate the container through the normal host flow. This is not free context: thinking tokens count toward the output budget, and raising the reservation leaves less context for conversation history and tool results, which can cause DAAF's context-pressure stop/restart guidance to fire earlier. Your provider or selected model may enforce a lower limit regardless. Keep specialist final returns bounded even with 64K enabled; prefer the default `32000` unless you have a concrete truncation problem. See the [technical FAQ entry](07_faq_technical.md#q-how-do-i-opt-in-to-a-64k-claude-code-output-budget) for the tradeoff summary. - -##### Under the hood / troubleshooting reference (Option F) - -*You can safely skip this subsection — it's here for when you want to look under the hood or diagnose a problem. These mechanics are shared by both shim lanes (this one and the ChatGPT lane below).* +- **On the direct OpenAI-API shim route,** append `[1m]` to your bare GPT slugs (e.g. `ANTHROPIC_DEFAULT_OPUS_MODEL=gpt-5.6-sol[1m]`, as shown above) and set `CLAUDE_CODE_MAX_CONTEXT_TOKENS=1050000`. Claude Code reads `[1m]` as a 1M-window hint and strips it before sending. +- **On OpenRouter,** keep the provider-prefixed bare slug (e.g. `ANTHROPIC_MODEL=openai/gpt-5.6-sol`) and set `CLAUDE_CODE_MAX_CONTEXT_TOKENS=1050000`. Don't add `[1m]` to provider-prefixed slugs — it isn't established for that form. +- **On the [ChatGPT-subscription (Codex) lane below](#option-f-alternate-lane-chatgpt-subscription-codex-backend),** set `CLAUDE_CODE_MAX_CONTEXT_TOKENS=370000` instead — that backend caps input around 370,000 tokens (measured for `gpt-5.6-sol`, 2026-07-16), and no client hint can raise it. -**Shim controls, logs, and tuning variables.** The manager script (`start_shim.sh`, shown in Step 4) accepts `--start`, `--restart`, `--stop`, `--status`, and `--auto`. The shim's log lives at `/daaf/scripts/provider_shim/logs/shim.log`. Manager, supervisor, and Python records are serialized through one append path, so concurrent writers cannot interleave or duplicate stored records. Rotation happens before a new supervisor starts and only when the active log is **larger than 25 MiB**; DAAF retains five generations. The log directory and lock directories are mode `0700`, active and retained log files are mode `0600`, and startup refuses symlink or non-regular log/state targets instead of following them. The manager also verifies saved PID/PGID identity before signaling; when `setsid` isolation is unavailable it falls back to exact per-process identity checks rather than killing a shared process group. Restart emits one bounded `SHIM_RESTART_BEGIN` record and one `SHIM_RESTART_RESULT` record, with distinct stop, launch, and readiness failure outcomes. +*Why it matters:* the declaration keeps Claude Code and DAAF measuring utilization against the same real window. DAAF does **not** use `CLAUDE_CODE_AUTO_COMPACT_WINDOW` — automatic compaction stays off because it can disrupt orchestration. If a GPT session reports "Context limit reached" / "Prompt is too long" at low utilization, see the [technical FAQ entry on low-utilization context errors](07_faq_technical.md#q-my-gpt-session-says-context-limit-reached--prompt-is-too-long-at-low-utilization). -Backend failures are logged as structural metadata — status, exact structured type/code, mapped Anthropic type, retry/attempt information, phase, and safe allowlisted request IDs. Free-form backend response bodies and messages are not retained, because provider prose can reflect request content; downstream client errors use bounded, classification-derived text instead. The [technical FAQ](07_faq_technical.md#q-my-gpt-session-fails-instantly-with-429-errors-on-every-request-option-f) walks through status/type/code triage such as `insufficient_quota` (billing) versus a true rate limit. Defaults are fine for almost everyone; the tuning variables (`SHIM_PORT`, `SHIM_BACKEND_BASE_URL`, `SHIM_BACKEND_API_KEY`, `SHIM_STRIP_MODEL_PREFIX`, `SHIM_SANITIZE_TOOLS`, `SHIM_REASONING_EFFORT`, `SHIM_TEXT_VERBOSITY`) are documented in `environment_settings_example.txt`. - -**Images.** The shim accepts image blocks in user messages and image-bearing tool results, translating them to the public OpenAI Responses `input_image` shape while preserving the original text/image order. Sources may be valid Base64 with an exact MIME type of JPEG, PNG, GIF, or WebP, or a fully qualified `http://` or `https://` URL. Text-only tool results keep the established string-output behavior; the list form is used only when a result contains an image. The shim validates structure without inspecting or logging image bytes, and it does **not** claim to detect animated GIFs. Unsupported or malformed forms fail locally with `400 invalid_request_error` before any backend call; the [image troubleshooting entry](07_faq_technical.md#q-what-image-inputs-does-the-provider-shim-support) lists the exact rejection cases. - -This translation is grounded in OpenAI's public `/v1/responses` contract. On 2026-07-18, one privacy-safe live request sent a synthetic one-pixel Base64 PNG through the ChatGPT/Codex lane and received HTTP 200, an Anthropic-shaped response containing text, and a DAAF request ID; unique prompt and Base64 sentinels were absent from all retained shim logs checked by the gate. That is evidence that this specific Base64-PNG shape was compatible on that date, not an official OpenAI guarantee. URL images and other MIME/source variants were not part of the live gate. +**Optional 64K Claude Code output budget.** Claude Code defaults `CLAUDE_CODE_MAX_OUTPUT_TOKENS` to `32000` and supports a maximum of `64000`. If unusually long specialist work is being cut off, you may opt in by adding the plain decimal `CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000` (the form supported by the pinned Claude Code 2.1.202) to the **host** `daaf-docker/environment_settings.txt` before Claude Code/the container starts, then recreate the container through the normal host flow. This is not free context: thinking tokens count toward the output budget, and raising the reservation leaves less context for conversation history and tool results, which can cause DAAF's context-pressure stop/restart guidance to fire earlier. Your provider or selected model may enforce a lower limit regardless. Keep specialist final returns bounded even with 64K enabled; prefer the default `32000` unless you have a concrete truncation problem. See the [technical FAQ entry](07_faq_technical.md#q-how-do-i-opt-in-to-a-64k-claude-code-output-budget) for the tradeoff summary. -**Tool-call sanitization.** This is **on by default** (`SHIM_SANITIZE_TOOLS`): the shim silently strips known GPT tool-call quirks (empty `pages` parameters, `isolation` fills on subagent dispatches, redundant sandbox flags) that would otherwise each cost a wasted error round-trip; every strip is recorded in the shim log. Set `SHIM_SANITIZE_TOOLS=0` (and restart the shim — the flag is read at startup) when running DAAFBench against shim-routed models, which must observe raw model behavior; confirm via the `/health` endpoint's `sanitize_tools` field. +##### Under the hood / troubleshooting reference (Option F) -**Correlated request diagnostics.** Every JSON or SSE response from `POST /v1/messages` carries an `x-daaf-request-id` response header containing an internally generated 32-character hexadecimal ID. The same value appears as `req_id=<32-hex-id>` on that request's shim and HTTP-client log records, so saving the header lets you reconstruct one request without matching unrelated concurrent traffic. Where available, the lifecycle also distinguishes the client request ID sent upstream from the provider request ID returned in response headers; neither replaces the local correlation ID. The stable lifecycle event names are `request_start`, `request_parsed`, `upstream_attempt`, `upstream_retry`, `upstream_headers`, `upstream_first_event`, `downstream_first_content`, `backend_error`, `disconnect`, `terminal`, and `cleanup`; retry events can repeat and some milestones apply only to streaming or failure paths, but every `/v1/messages` lifecycle has exactly one `event=terminal` summary and one `event=cleanup` record. +**Optional reading.** Here for when you want to look under the hood or diagnose a problem. These mechanics are shared by both shim lanes (this one and the ChatGPT lane below). -These records expose bounded diagnostic metadata rather than content: transport exception class and failure phase, monotonic `elapsed_ms`/`dur_ms` timings, attempt and retry counts, retry source/reason, allowlisted client/provider request IDs, normalized `http_version`, structured error type/code, disconnect state, terminal/body-close send state, and cleanup status/error. They **never log** prompts or text, tool inputs or arguments, Base64 or decoded image bytes, image URLs or query strings, file IDs or names, credentials or cookies, raw SSE, complete request/response bodies, or free-form backend prose. For a safe request-ID grep workflow and a field-by-field mid-response diagnostic, see [How do I diagnose a provider-shim server error mid-response?](07_faq_technical.md#q-how-do-i-diagnose-a-provider-shim-server-error-mid-response). +**Logs and diagnostics.** The shim keeps its own diagnostic log at `/daaf/scripts/provider_shim/logs/shim.log`. It records only technical metadata — timings, status codes, error types, retry counts, and request-correlation IDs — and **never** your prompts or text, tool inputs, image bytes or URLs, credentials, raw response streams, or full request/response bodies. If a GPT session misbehaves, run `--restart` and check that log; the [technical FAQ](07_faq_technical.md#q-my-gpt-session-fails-instantly-with-429-errors-on-every-request-option-f) walks through common status/type/code triage such as `insufficient_quota` (billing) versus a true rate limit. -**Reading the evidence carefully.** The terminal send fields are deliberately narrow evidence. `send_completed` means only that the awaited ASGI `send` call returned; it does **not** prove that the client received or acknowledged the bytes. Likewise, `event=disconnect` records that the shim observed an ASGI `http.disconnect` event at `observed_phase`; it is not proof of what the user received or displayed, does not establish that the disconnect caused an upstream operation to be cancelled, and does not imply that an upstream operation existed. Generic ASGI task cancellation is not itself recorded as client-disconnect evidence. Read the correlated events in order before drawing a causal conclusion: for example, a structured backend `server_error` followed by `disconnect` is evidence that the shim observed the backend-declared error first, not proof that the later disconnect caused it or that the terminal error reached the client. +**Images.** The shim accepts images in your messages and tool results and forwards them to OpenAI without inspecting or logging the bytes; the [image troubleshooting entry](07_faq_technical.md#q-what-image-inputs-does-the-provider-shim-support) lists exactly what's accepted. Image support was verified with a dated privacy-safe probe (a 2026-07-18 Base64-PNG request) — that's evidence the shape worked on that date, not an official OpenAI guarantee. -**Structured-error normalization.** The shim's structured-error normalizer gives a real non-2xx backend HTTP status first precedence. When there is no such status, it checks backend `code` before backend `type`: `context_length_exceeded` maps to Anthropic `invalid_request_error`, `server_error` maps to `api_error`, and unknown structured failures remain `api_error`. Users receive bounded, classification-derived messages selected from status/type/code families and a fixed unknown fallback; arbitrary provider prose is neither retained nor reflected. Only observability and error presentation are shaped this way — retry policy, timeout values, backoff, connection pooling, and heartbeat behavior are independent of it. +**Advanced tuning.** Tool-call sanitization is on by default (`SHIM_SANITIZE_TOOLS`); the shim quietly strips known GPT tool-call quirks that would otherwise waste an error round-trip. Turn it off (`SHIM_SANITIZE_TOOLS=0`, then restart) only when running DAAFBench, which must observe raw model behavior. The remaining tuning variables are documented in `environment_settings_example.txt`. #### Option F, alternate lane: ChatGPT subscription (Codex backend) @@ -1065,7 +1059,7 @@ If that reports `codex: command not found`, your image predates the universal in ANTHROPIC_DEFAULT_SONNET_MODEL=gpt-5.6-terra[1m] CLAUDE_CODE_MAX_CONTEXT_TOKENS=370000 # Codex lane: backend-capped ~370k (measured), NOT 1,050,000 — see below ``` - In this mode `OPENAI_API_KEY` / `SHIM_BACKEND_API_KEY` are **ignored** — the OAuth token is the credential. Codex availability alone does not select this lane; both `DAAF_PROVIDER_SHIM=openai` and `SHIM_BACKEND_MODE=chatgpt` are explicit opt-ins. **Note the `CLAUDE_CODE_MAX_CONTEXT_TOKENS=370000`** above: unlike the API-key lane's full `1050000`, the ChatGPT/Codex backend enforces a much lower effective *input* ceiling (~370,000 tokens, measured for `gpt-5.6-sol` on 2026-07-16 — accepted at 369,941, rejected at 372,905), independent of the model's true 1M window on the API route. Declaring `370000` aligns Claude Code and DAAF's physical-window accounting with that ceiling. DAAF uses it for utilization, statusline rendering, severity, and behavioral stop/restart guidance. DAAF keeps automatic compaction disabled and does not implement a transport-level request blocker; the Codex backend remains the ultimate hard ceiling. Keeping `[1m]` on the configured slugs does not raise that ceiling because DAAF's matched runtime consumers apply `min(resolved_window, 370000)` as a final accounting constraint. A breach surfaces as a clean `invalid_request_error` (`context_length_exceeded`) rather than a retryable-looking 502. Re-measure with `scripts/provider_shim/probe_context_ceiling.py` if OpenAI changes the backend. + In this mode `OPENAI_API_KEY` / `SHIM_BACKEND_API_KEY` are **ignored** — the OAuth token is the credential. Codex availability alone does not select this lane; both `DAAF_PROVIDER_SHIM=openai` and `SHIM_BACKEND_MODE=chatgpt` are explicit opt-ins. **Note the `CLAUDE_CODE_MAX_CONTEXT_TOKENS=370000`** above: unlike the API-key lane's full `1050000`, the ChatGPT/Codex backend caps input around 370,000 tokens (measured for `gpt-5.6-sol`, 2026-07-16), independent of the model's true 1M window on the API route — and no `[1m]` hint or client setting can raise it. Declaring `370000` keeps Claude Code and DAAF's utilization accounting aligned with that ceiling; a breach surfaces as a clean `invalid_request_error` (`context_length_exceeded`). Re-measure with `scripts/provider_shim/probe_context_ceiling.py` if OpenAI changes the backend. 5. **Recreate the container** to apply (`docker compose down`, then `bash run_daaf.sh` / `.\run_daaf.ps1`). The shim reads `SHIM_BACKEND_MODE` at startup. **Confirm it's working.** After the container comes back up, check the shim log: @@ -1095,8 +1089,7 @@ See the [technical FAQ entry on the ChatGPT subscription lane](07_faq_technical. GPT support is a **supported capability we've engineered carefully** — actively tested, including through DAAF's own benchmark runs, and still gathering wider community experience across different setups. A few specifics are worth knowing: - **Context utilization is estimated, not exact.** OpenRouter's Anthropic-compatible endpoint does not implement token counting, so Claude Code falls back to estimation on GPT sessions (the shim does the same). The context bar and utilization warnings are close approximations, not precise counts. -- **Set `CLAUDE_CODE_MAX_CONTEXT_TOKENS` to the route's real physical accounting window.** Claude Code assumes a 200k window for models it doesn't recognize, which is wrong for the 400k and 1,050,000-token GPT models. Setting this variable (as shown in the blocks above) aligns Claude Code and DAAF's utilization denominator. **The two shim lanes differ here:** the API-key lane (and OpenRouter) serve the model's full window (`1050000`), but the ChatGPT-subscription (Codex) lane's backend enforces a much lower effective *input* ceiling — measured at ~370,000 tokens for `gpt-5.6-sol` (2026-07-16) — so on that lane declare `CLAUDE_CODE_MAX_CONTEXT_TOKENS=370000` instead. This is a backend-enforced limit on the Codex route, independent of the model's true 1M window on the API route; no `[1m]` hint or client setting can raise it. DAAF's three runtime consumers use the exact `DAAF_PROVIDER_SHIM=openai` + `SHIM_BACKEND_MODE=chatgpt` gate and apply `min(resolved_window, 370000)` to the mapped GPT 5.4/5.5/5.6 flagship family, including stale-cache and same-model paths. DAAF uses the result for utilization, statusline rendering, severity, and behavioral stop/restart guidance; it keeps automatic compaction disabled and does not implement a transport-level request blocker. The backend remains the ultimate hard ceiling. Re-measure with `scripts/provider_shim/probe_context_ceiling.py` if OpenAI changes it. Sol keeps its exact-Sol quality-threshold profile on both lanes — physical-window mapping and quality-threshold profile selection are separate lookups. -- **Exact GPT 5.6 Sol is the validated exception; other GPT variants stay conservative.** The terminal model slug must be exactly `gpt-5.6-sol` or `gpt-5.6-sol[1m]`; the identifier may be bare or may contain one or more provider path prefixes ending in `/`. These exact terminal GPT 5.6 Sol model slugs use the standard percentage gates — ELEVATED at 40%, HIGH at 60%, and CRITICAL at 75% — while retaining Sol's higher validated absolute gates of 300,000, 400,000, and 500,000 tokens, respectively; each status fires when either its percentage or absolute gate is reached. Claude Fable/Mythos remains on 30%/40%/50% with the same 300,000/400,000/500,000 absolute gates. On the ChatGPT-subscription lane's 370,000-token physical window, Sol's percentage gates correspond to 148,000, 222,000, and 277,500 tokens, so they fire before the retained absolute gates. Malformed left-boundary strings such as `xgpt-5.6-sol`, `foo-gpt-5.6-sol`, and `vendor/notgpt-5.6-sol` remain conservative, as do Terra, Luna, Pro, mini, chat, date snapshots, future variants, and right-side suffix or trailing variants. Those use the conservative-default gates (40%/150,000, 60%/200,000, 75%/250,000) unless separately validated and registered. Threshold-profile selection is version-specific and independent of physical context-window mapping: the wider GPT 5.6 family can still map to a 1,050,000-token window even when a particular variant uses conservative quality thresholds. GPT is not part of the Claude Fable/Mythos model family; exact Sol now shares the standard 40%/60%/75% percentages but retains the higher validated Sol absolute gates. +- **Set `CLAUDE_CODE_MAX_CONTEXT_TOKENS` to match your route.** Claude Code assumes a small (~200k) window for models it doesn't recognize, which is wrong for the large GPT windows. Set it to `1050000` on the API-key and OpenRouter routes, or to `370000` on the ChatGPT-subscription (Codex) lane, whose backend caps input around 370k tokens regardless of the model's true 1M window. DAAF detects your model and applies the matching quality thresholds automatically. Re-measure the Codex ceiling with `scripts/provider_shim/probe_context_ceiling.py` if OpenAI changes it. - **OpenRouter's Anthropic endpoint is officially scoped to Anthropic models.** GPT works through it (proven live), but OpenRouter documents this endpoint for Claude models — it is effectively unsupported territory the vendor could change at any time. - **Anthropic does not officially support routing Claude Code to non-Claude models** through any gateway. DAAF offers and tests these lanes as a supported capability, not something either vendor guarantees. diff --git a/user_reference/02_understanding_daaf.md b/user_reference/02_understanding_daaf.md index 43673778..ead338fa 100644 --- a/user_reference/02_understanding_daaf.md +++ b/user_reference/02_understanding_daaf.md @@ -388,9 +388,9 @@ And here's what each of them do in more detail, and how the workflow works so yo **What it is:** You have an existing completed DAAF analysis and want an independent mechanical check from its delivered audit notebook. Intake requires one unambiguous original Report and one unambiguous DAAF Stage 9 archive: Marimo `.py` for Python or canonical Quarto `.qmd` for R. Generic Marimo apps and arbitrary Quarto projects are not supported. If there are multiple candidates or both formats, DAAF stops and asks you to choose. Reproduction always uses a new folder and a new extraction root; it never merges into or overwrites an earlier attempt. -After extraction, DAAF normalizes the project root and runs a mandatory containment audit before execution. A passing audit gives bounded assurance: the known original-root string is absent and each supported script has one correct static `PROJECT_DIR` assignment. It does not prove every dynamically constructed path safe. A script with a nonpassing audit cannot run in RV-2 unless it is corrected and re-audited or you approve that exact script as an exclusion before RV-2. +Before re-running anything, DAAF checks each script to make sure it's safe to re-run — that it points at the new reproduction folder and won't reach back into the original project's files. A script that fails this check isn't run until it's either corrected or you explicitly approve running it as-is. This is a safety check on where scripts read and write, not a guarantee that every possible path in the code is safe. -Evidence is graded rather than guessed. Log comparison covers only metrics actually printed in both logs. Supported Parquet artifacts receive direct schema/content comparison; selected files can receive exact-byte comparison, which proves byte identity only. Figures are reviewed side by side, and saved tables/models are compared only when an inspectable representation exists. Anything missing, unsupported, or unproven is labeled **NOT DIRECTLY VERIFIED**, never silently treated as a match. +Evidence is graded rather than guessed. Log comparison covers only metrics actually printed in both logs. Supported Parquet data files get a direct schema and content comparison; selected files can also be compared byte-for-byte, which proves they are identical. Figures are reviewed side by side, and saved tables/models are compared only when an inspectable representation exists. Anything missing, unsupported, or unproven is labeled **NOT DIRECTLY VERIFIED**, never silently treated as a match. **What a "reproduced" verdict does and doesn't promise.** When DAAF reports an analysis as reproduced, it means DAAF re-ran the work and the results matched the original within reasonable tolerances -- not that every number came out bit-for-bit identical. Small, legitimate differences can arise: data that was re-downloaded may have been updated at the source since the original run, the software environment may have moved on, and some statistical methods involve a degree of randomness. DAAF's comparison uses purpose-built checks for the different kinds of evidence -- data files, figures, execution logs, and the quantitative claims in the report -- and those checks are built to account for reasonable tolerances rather than to demand exact equality. The goal is a trustworthy, well-documented answer to "does this hold up when someone re-runs it?", not a guarantee of identical digits. @@ -403,7 +403,7 @@ Evidence is graded rather than guessed. Log comparison covers only metrics actua - **Whether to re-fetch data** (default: yes). Re-fetch mode executes Stage 5 and tests current acquisition behavior. Frozen-input mode copies and hashes the original raw files, does **not** run Stage 5, starts with downstream scripts, and verifies those raw hashes remain unchanged. In frozen mode, acquisition reproducibility and mirrors are explicitly out of scope. - **Methodological review depth** (default: light). Light review focuses on mechanical reproduction and notable concerns; full review also scrutinizes methodology, assumptions, and interpretation. -Exact exclusions only leave the verdict denominator when you approve them before re-execution begins. A script skipped later remains an in-scope gap. Environment comparison is language-aware: Python inventories Python, Marimo, and imported packages; R inventories R, Quarto, repository/snapshot metadata, and installed imported packages. Unpinned original versions remain UNKNOWN / NOT DIRECTLY VERIFIED rather than being inferred. When DAAF needs a diagnostic script for that inventory, it keeps the script and its execution record under the reproduction project's `scripts/repro_checks/` folder; it never installs packages into the running environment. +Something only drops out of the reproduction verdict if you approve excluding it up front — a script you skip partway through still counts as an in-scope gap. DAAF also compares the software environments the two runs used (which language, notebook, and package versions), labeling anything it can't pin down as UNKNOWN / NOT DIRECTLY VERIFIED rather than guessing. Any helper scripts it writes for that comparison are kept with the rest of the reproduction project's files, and it never installs packages into the running environment. **Expected time investment:** Depends on the complexity of the original analysis. A straightforward single-source analysis might take 15-30 minutes; a multi-source analysis with extensive transformations could take longer. You'll review the Reproduction Report at the end. @@ -720,7 +720,7 @@ marimo run 'research/YYYY-MM-DD_Title/YYYY-MM-DD_Notebook.py' --host 0.0.0.0 --p ``` You can also open the `.py` file in any text editor -- marimo notebooks are just Python. -**How to view Quarto notebooks (R):** The easiest way is the **DAAF Control Panel** (`bash daaf.sh` / `.\daaf.ps1` from your `daaf-docker` folder → **5) View Quarto Notebooks (R)**), which runs `view_quarto.sh` for you. (You can also run `bash view_quarto.sh` / `.\view_quarto.ps1` directly.) With no argument, the viewer recursively finds every `.qmd` anywhere below `research/`, presents a sorted numbered picker, and renders the notebook you select; `0`, blank input, `q`/`Q`, or end-of-file cancels without rendering. You can bypass the picker with a direct `.qmd` path, or pass a project folder when that project contains exactly one recursive match (multiple matches are printed rather than guessed). The selected notebook is rendered to a single self-contained HTML file inside the container, copied to a `quarto_html/` folder on your host, and opened in your browser (the R-notebook counterpart to `view_notebooks.sh` for Python). The host filename is the flat notebook basename, so equal basenames overwrite; set `QUARTO_HTML_DIR` differently when both outputs must be retained. In a Full Pipeline Stage 9 notebook, rendering evaluates only explicitly enabled preview chunks; archived analysis-script chunks remain disabled, so render success validates the audit document rather than reproducing the full analysis. Alternatively, from inside the container you can render by hand with `quarto render 'research/YYYY-MM-DD_Title/YYYY-MM-DD_Notebook.qmd'`, then view the resulting HTML file in the browser-based code editor. You can also open the `.qmd` file directly in any text editor -- Quarto notebooks are plain text with YAML frontmatter and code chunks. +**How to view Quarto notebooks (R):** The easiest way is the **DAAF Control Panel** (`bash daaf.sh` / `.\daaf.ps1` from your `daaf-docker` folder → **5) View Quarto Notebooks (R)**), which runs `view_quarto.sh` for you — it lists every `.qmd` under `research/`, lets you pick one, renders it to a self-contained HTML file, and opens it in your browser. (You can also run `bash view_quarto.sh` / `.\view_quarto.ps1` directly, or pass it a specific `.qmd` path.) Alternatively, from inside the container you can render by hand with `quarto render 'research/YYYY-MM-DD_Title/YYYY-MM-DD_Notebook.qmd'`, then view the resulting HTML file in the browser-based code editor. You can also open the `.qmd` file directly in any text editor -- Quarto notebooks are plain text with YAML frontmatter and code chunks. ### The Report @@ -856,17 +856,9 @@ Real research analyses take time -- often more time than a single Claude Code se Claude Code operates within a context window that can be up to 1M tokens. However, quality can degrade well before the window is full, so DAAF enforces dual thresholds — both percentage-based and absolute token counts, whichever fires first. As DAAF works through a Full Pipeline analysis, delegating tasks to agents, receiving results, and coordinating the workflow, it gradually fills up this context. When it gets too full, Claude's performance degrades, and it becomes increasingly susceptible to erratic behavior due to **context rot**. -The exact points at which DAAF acts depend on the **context-quality threshold profile** assigned to the model you are running. DAAF detects the exact model identifier automatically and applies the right profile — you do not need to configure anything. Claude Fable/Mythos models retain the validated extended-horizon profile. Exact GPT 5.6 Sol identifiers use the standard percentage gates but retain Sol's higher validated absolute gates. Opus, Sonnet, unknown model IDs, every other GPT variant, GLM models, and all other alternative-provider models use the conservative-default profile unless that exact model has been separately validated and registered. +DAAF watches context usage for every agent it runs and applies model-appropriate thresholds automatically — you configure nothing. It detects which model you're running and picks the right profile: models DAAF has specifically validated (Claude Fable/Mythos, and exact GPT 5.6 Sol) get more generous thresholds, while everything else uses conservative defaults. The important part isn't the exact numbers — it's that quality can start to degrade before a window is technically full, which is why DAAF acts early rather than waiting for it to fill. -Threshold-profile selection is version-specific and separate from the model's physical context-window size. For GPT 5.6 Sol, the terminal model slug must be exactly `gpt-5.6-sol` or `gpt-5.6-sol[1m]`; the identifier may be bare or may contain one or more provider path prefixes ending in `/`. Malformed left-boundary strings such as `xgpt-5.6-sol`, `foo-gpt-5.6-sol`, and `vendor/notgpt-5.6-sol` remain conservative, as do right-side suffix or trailing variants. GPT is not part of Claude's Fable/Mythos family: exact Sol now shares the standard 40%/60%/75% percentages but retains the higher validated Sol absolute gates of 300k/400k/500k. Terra, Luna, Pro, mini, chat, date snapshots, future variants, and trailing modifiers remain conservative unless separately validated. This means a wider GPT 5.6 variant can still have a 1,050,000-token physical window while DAAF applies the conservative quality thresholds. That physical window is itself route-conditional: the 1,050,000 figure holds on the API-key and OpenRouter routes, while the ChatGPT-subscription (Codex) shim lane is backend-capped near ~370,000 tokens (measured for Sol, 2026-07-16), which DAAF's hooks and statuslines lane-gate automatically. On that 370k physical window, exact Sol's percentage boundaries are 148k, 222k, and 277.5k tokens, before its retained 300k, 400k, and 500k absolute gates. - -To prevent context rot, DAAF monitors its own context utilization continuously and manages this proactively. The thresholds below trigger the same set of protective actions for every model — only the trigger points differ by threshold profile (percentage OR token count, whichever comes first): - -| Threshold Profile | Membership | ELEVATED at | HIGH at | CRITICAL at | -|-------------------|------------|-------------|---------|-------------| -| **Validated extended-horizon** | Claude Fable/Mythos models | ≥ 30% or ≥ 300k tokens | ≥ 40% or ≥ 400k tokens | ≥ 50% or ≥ 500k tokens | -| **Exact GPT 5.6 Sol** | Exact terminal model slugs, bare or provider-prefixed: `gpt-5.6-sol` or `gpt-5.6-sol[1m]` | ≥ 40% or ≥ 300k tokens | ≥ 60% or ≥ 400k tokens | ≥ 75% or ≥ 500k tokens | -| **Conservative-default** | Opus, Sonnet, unknown model IDs, every other GPT variant, GLM models, and all other alternative-provider models unless individually validated and registered | ≥ 40% or ≥ 150k tokens | ≥ 60% or ≥ 200k tokens | ≥ 75% or ≥ 250k tokens | +DAAF monitors its own context utilization continuously and takes escalating protective action as it climbs. The four levels below, and what DAAF does at each, are the same for every model — only the exact trigger points differ: | Status | What DAAF Does | |--------|----------------| @@ -944,7 +936,7 @@ DAAF includes convenience scripts for viewing your files, notebooks, and session - **Browse and edit project files:** `bash run_vscode.sh` (macOS/Linux) or `.\run_vscode.ps1` (Windows) -- opens a browser-based VS Code editor. You can preview Markdown reports and plans with `Shift+Ctrl+V`, read Python and R scripts with syntax highlighting, and inspect the Git history to see what changed and when. See [**03. Best Practices — Using the Browser-Based Code Editor**](03_best_practices.md#using-the-browser-based-code-editor) for more. - **View interactive notebooks (Python):** `bash view_notebooks.sh` (macOS/Linux) or `.\view_notebooks.ps1` (Windows) -- opens marimo's notebook browser at [http://localhost:2718](http://localhost:2718), where you can browse and open any project notebook. -- **View Quarto documents (R):** `bash view_quarto.sh` (macOS/Linux) or `.\view_quarto.ps1` (Windows) -- recursively discovers every `.qmd` below `research/` and presents a sorted numbered picker, then renders the selection to self-contained HTML, copies it to `quarto_html/` on your host, and opens it in your browser. Cancel with `0`, blank, `q`/`Q`, or end-of-file; pass a direct `.qmd` path to bypass selection, or pass a project folder only when it has one recursive notebook match. Equal basenames share the same flat output path and overwrite; set `QUARTO_HTML_DIR` differently to retain both. +- **View Quarto documents (R):** `bash view_quarto.sh` (macOS/Linux) or `.\view_quarto.ps1` (Windows) -- finds every `.qmd` below `research/`, lets you pick one from a numbered list, renders it to a self-contained HTML file on your host, and opens it in your browser. Pass a direct `.qmd` path to skip the picker. - **View session logs:** `bash view_logs.sh` (macOS/Linux) or `.\view_logs.ps1` (Windows) -- opens the **DAAF Log Explorer**, an interactive timeline that shows orchestrator actions, subagent dispatches, and tool calls in your browser at [http://localhost:2719](http://localhost:2719). See the [Installation Guide — Viewing Session Logs](01_installation_and_quickstart.md#viewing-session-logs-in-your-browser) for details. --- diff --git a/user_reference/03_best_practices.md b/user_reference/03_best_practices.md index b2c778b0..c1caf28e 100644 --- a/user_reference/03_best_practices.md +++ b/user_reference/03_best_practices.md @@ -245,7 +245,14 @@ The report follows a standard structure (Executive Summary, Key Findings, Data & ### Reading DAAF's Claims: Observed Facts vs. Inference -One habit worth building as you read any DAAF output is to notice how it grades its own claims by evidence. When DAAF reports that something *ran* -- a fetch returned so many rows, a validation passed -- the actual command and its output are on record, quoted in the script's execution log for you to check; statements without that kind of record are inferences, and DAAF is instructed to phrase them so they read that way rather than as established fact. Pay especially close attention when DAAF says something is *impossible* or *unavailable* -- that a source doesn't offer a variable, or that an operation can't be done -- because a wrong "no" fails silently and can quietly harden into accepted fact, so those negative claims deserve the same "show me the check" scrutiny you'd give any surprising result. And when DAAF tells you how much it did -- files changed, outcomes addressed -- that accounting should trace back to actual tool output, not to memory. Reading with this lens tells you which parts of a report you can take at face value and which warrant a second look. +One habit worth building as you read any DAAF output is to notice how it grades its own claims by evidence. A quick lens to read with: + +- **Ran-and-quoted means observed fact.** When DAAF reports that something *ran* -- a fetch returned so many rows, a validation passed -- the actual command and its output are on record in the script's execution log for you to check. +- **Everything else is inference.** Statements without that kind of record are inferences, and DAAF is instructed to phrase them so they read that way rather than as established fact. +- **Watch negative claims most closely.** When DAAF says something is *impossible* or *unavailable* -- that a source doesn't offer a variable, or that an operation can't be done -- give it extra scrutiny. A wrong "no" fails silently and can quietly harden into accepted fact, so those claims deserve the same "show me the check" treatment you'd give any surprising result. +- **Completion counts should trace to output.** When DAAF tells you how much it did -- files changed, outcomes addressed -- that accounting should trace back to actual tool output, not to memory. + +Reading with this lens tells you which parts of a report you can take at face value and which warrant a second look. ### Reading the Notebook diff --git a/user_reference/07_faq_technical.md b/user_reference/07_faq_technical.md index 79002578..43f4b7d2 100644 --- a/user_reference/07_faq_technical.md +++ b/user_reference/07_faq_technical.md @@ -251,13 +251,13 @@ This is expected, not a bug: DAAF ships with Claude as the default model, so a G Not exactly — treat it as a close estimate. OpenRouter's Anthropic-compatible endpoint (and the provider shim) do not implement precise token counting, so Claude Code falls back to *estimating* context usage on GPT sessions. The context bar and the ELEVATED/HIGH/CRITICAL utilization warnings still work and are a good guide, but the percentages are approximations rather than exact counts. -**Under the hood (safe to skip).** DAAF's quality thresholds depend on the exact model identifier, not on whether the provider is GPT in general. Claude Fable/Mythos models retain the validated extended-horizon thresholds of 30%/300k, 40%/400k, and 50%/500k. For GPT 5.6 Sol, the terminal model slug must be exactly `gpt-5.6-sol` or `gpt-5.6-sol[1m]`; the identifier may be bare or may contain one or more provider path prefixes ending in `/`. These exact terminal GPT 5.6 Sol model slugs use the standard percentage gates while retaining Sol's higher validated absolute gates: 40% or 300k for ELEVATED, 60% or 400k for HIGH, and 75% or 500k for CRITICAL, whichever trigger fires first. Malformed left-boundary strings such as `xgpt-5.6-sol`, `foo-gpt-5.6-sol`, and `vendor/notgpt-5.6-sol` remain conservative, as do Terra, Luna, Pro, mini, chat, date snapshots, future GPT variants, and right-side suffix or trailing variants. Those use the conservative-default thresholds of 40%/150k, 60%/200k, and 75%/250k unless separately validated and registered. Threshold-profile selection is independent of physical context-window mapping: another GPT 5.6 variant can still map to a 1,050,000-token window while using conservative quality thresholds. GPT is not part of the Claude Fable/Mythos model family; exact Sol now shares the standard percentages but retains the higher validated Sol absolute gates. Separately, on the ChatGPT-subscription shim lane the physical window itself is backend-capped at ~370k; there, exact Sol's percentage boundaries are 148k, 222k, and 277.5k tokens, before the retained 300k, 400k, and 500k absolute gates (see the next entry). +**Optional reading — under the hood.** DAAF's quality thresholds key off the exact model identifier, not just "is this GPT." Models DAAF has specifically validated (Claude Fable/Mythos, and exact GPT 5.6 Sol) get extended thresholds; every other model falls back to conservative defaults until it's validated too. Selection is automatic from the model ID — you don't configure it. One subtlety: which threshold profile a model gets is a separate question from how big its context window is, so a model can have a large window but still use conservative thresholds. And on the ChatGPT-subscription lane the window itself is capped lower, which is why thresholds fire sooner there (see the next entry). ### Q: The statusline shows the wrong context window on a GPT session (e.g. 200k when my model has more) Nothing is broken here — the bar is just showing a wrong guess, and it's a one-line fix. Claude Code assumes a 200k context window for any model it doesn't recognize, which is wrong for the 400k `gpt-5.2` / `gpt-5.4-mini` and the 1,050,000-token `gpt-5.4` / `gpt-5.5` / `gpt-5.6` family (Sol/Terra/Luna) on API/OpenRouter routes. Fix it by setting `CLAUDE_CODE_MAX_CONTEXT_TOKENS` in `environment_settings.txt` to the route's real physical accounting window (for example, `1050000` for `gpt-5.6-sol` over OpenRouter or the OpenAI API), then recreate the container and restart the Claude Code session. -The **ChatGPT-subscription (Codex) shim lane** is different: its backend enforces a much lower effective *input* ceiling, measured at ~370,000 tokens for `gpt-5.6-sol` (2026-07-16), independent of the model's true 1M window on the API route. On that lane set `CLAUDE_CODE_MAX_CONTEXT_TOKENS=370000`. This aligns Claude Code and DAAF's accounting with the backend ceiling. DAAF's statusline, context reporter, and subagent statusline all apply `min(resolved_window, 370000)` as a final constraint for the matched GPT 5.4/5.5/5.6 flagship family when both exact lane signals are present, so stale caches, incoming `[1m]` windows, and higher explicit declarations cannot preserve an unsafe denominator. DAAF uses the result for utilization, statusline rendering, severity, and behavioral stop/restart guidance. It keeps automatic compaction disabled and does not implement a transport-level request blocker; the backend remains the ultimate hard ceiling. +The **ChatGPT-subscription (Codex) lane** is different: its backend caps the usable input window much lower — measured at ~370,000 tokens for `gpt-5.6-sol` (2026-07-16), regardless of the model's true 1M window on the API route. On that lane, set `CLAUDE_CODE_MAX_CONTEXT_TOKENS=370000` instead, so Claude Code and DAAF measure usage against the real ceiling. DAAF applies this automatically wherever it can, and keeps automatic compaction off; the backend itself remains the ultimate hard limit. ### Q: How do I opt in to a 64K Claude Code output budget? @@ -288,32 +288,23 @@ bash /daaf/scripts/provider_shim/start_shim.sh --restart # atomic repair/reload curl -s http://127.0.0.1:4141/health # health check ``` -Its log is at `/daaf/scripts/provider_shim/logs/shim.log` — check there first. `--status` calls the shim strictly ready only when `/health` matches the exact DAAF service identity, `status: "ok"`, the version derived from the installed source, and the normalized configured backend mode. Its exit status deliberately distinguishes states: a cleanly ready shim and a cleanly stopped shim both exit zero, while missing prerequisites or source, a running supervisor whose shim is not strictly ready, and a reachable endpoint whose health contract does not match all exit nonzero. A machine-readable `SHIM_READINESS_FAILURE` line identifies a contract mismatch; container boot's deliberately nonfatal `--auto` path records `SHIM_AUTO_READY` or `SHIM_AUTO_FAILURE` and continues boot. Explicit `--start` and `--restart` exit nonzero whenever their requested operation is unsuccessful, so an operator can act. +Its log is at `/daaf/scripts/provider_shim/logs/shim.log` — check there first. `--status` tells you whether the shim is cleanly running (or cleanly stopped); anything else means it isn't healthy, and the log will usually say why. -For an in-place repair or reload, use `--restart`, which keeps stop → wait → launch → strict readiness inside one locked manager invocation and records one `SHIM_RESTART_BEGIN` plus one `SHIM_RESTART_RESULT`. This is required when the active Claude Code session itself uses the shim route. Do **not** split `--stop` and `--start` across separate assistant turns: after the stop, the session may no longer have a working provider route with which to issue the start. - -The manager serializes concurrent `--restart`, `--start`, and `--stop` actions and sends signals only after exact process-identity checks. It rotates only when `shim.log` is larger than 25 MiB and retains five generations; active and retained logs are private (`0600`), while the log and lock directories are `0700`. It refuses symlink/non-regular targets. If `setsid` process-group isolation is unavailable, teardown uses verified per-PID signaling instead of risking the caller's shared group. Restart failures identify the stop, launch, or readiness stage and never report success before the complete strict health contract passes. +For an in-place repair or reload, use `--restart` — it stops and relaunches the shim in a single step. This matters when the active Claude Code session is itself running through the shim: do **not** split `--stop` and `--start` across separate assistant turns, because once the shim is stopped your session no longer has a working provider route to issue the `--start` with. `--restart` avoids that trap by never leaving the route down between turns. Remember that Option F requires the image to have been **rebuilt** after you set `DAAF_PROVIDER_SHIM=openai` (the auto-launch is baked into the entrypoint), so if nothing is running, confirm you rebuilt. If the shim *is* running but every request fails instantly with a 429, see the relevant entry below. ### Q: What image inputs does the provider shim support? -For the direct OpenAI API lane, the shim supports images in **user messages** and in **tool results**. It accepts two source forms: - -- valid Base64 paired with exactly one of `image/jpeg`, `image/png`, `image/gif`, or `image/webp`; or -- a fully qualified `http://` or `https://` URL with no embedded credentials. - -The shim emits the public OpenAI Responses `input_image` shape and preserves the original order of text and image blocks. Image-bearing tool results use the Responses item-list form; text-only tool results retain the existing string-output behavior. The shim validates Base64 syntax and structure but deliberately does not decode or inspect image content, so it does **not** claim to identify animated GIFs. - -A malformed or fidelity-unsafe form fails locally, before any backend request, as HTTP `400` with `invalid_request_error`. That includes malformed Base64; an unsupported MIME type or source type; an unsafe/non-HTTP(S) URL; a provider-scoped `file` source; an image in assistant history; an unknown history or tool-result block; and a malformed message/content block. Errors identify structural coordinates and the failure class, not the image, URL, file identifier, or surrounding text. +For the direct OpenAI API lane, the shim supports images in **user messages** and in **tool results**. You can supply an image two ways: as Base64 data paired with one of `image/jpeg`, `image/png`, `image/gif`, or `image/webp`, or as a plain `http://`/`https://` URL with no credentials embedded in it. A malformed or unsupported image fails right away with a local error, before anything is sent to the backend. One limitation: the shim doesn't decode image contents, so it can't detect animated GIFs. -The public `/v1/responses` image contract is the documented compatibility basis. On 2026-07-18, one privacy-safe live request sent a synthetic one-pixel Base64 PNG through the ChatGPT/Codex lane and received HTTP 200, an Anthropic-shaped response containing text, and a DAAF request ID. The gate also checked retained shim logs and found neither its unique prompt sentinel nor its unique Base64 sentinel. This is compatibility evidence for that specific Base64-PNG shape on that date, **not** an official OpenAI guarantee. URL images and other MIME/source variants were not part of the live gate. +This was verified with a privacy-safe probe on 2026-07-18 (a synthetic one-pixel PNG round-tripped successfully) — that's evidence that shape worked on that date, not an official OpenAI guarantee, and the URL variants weren't part of that test. ### Q: How do I diagnose a provider-shim server error mid-response? -**You almost certainly don't need this entry.** It's a deep-dive for the rare case where a GPT request fails partway through a response and you want to file a precise bug report or trace exactly what the shim did. If that's not you, it's completely safe to skip — the shim keeps its own logs and the earlier troubleshooting entries cover the common failures. If you *do* want to trace a specific failed request, here's how. +**Optional reading — you almost certainly don't need this entry.** It's a deep-dive for the rare case where a GPT request fails partway through a response and you want to file a precise bug report or trace exactly what the shim did. If that's not you, feel free to skip it — the shim keeps its own logs, and the earlier troubleshooting entries cover the common failures. If you *do* want to trace a specific failed request, here's how. -The provider shim gives each `POST /v1/messages` request an internal 32-character hexadecimal correlation ID. The shim returns it as the `x-daaf-request-id` response header on both JSON and SSE responses and writes the same value as `req_id=<32-hex-id>` on correlated records in `/daaf/scripts/provider_shim/logs/shim.log`. If the failing client or its network diagnostics expose response headers, copy that header value immediately. If they do not, use the failure timestamp to find the nearest `event=terminal` record and copy its `req_id`; treat a timestamp match as a lead to verify against the full lifecycle, not proof that it is the same request. +Every request through the shim gets a unique 32-character ID. The shim returns it as the `x-daaf-request-id` response header and writes the same value as `req_id=` on that request's records in `/daaf/scripts/provider_shim/logs/shim.log`. If your client can show you response headers, copy that value; otherwise, find the log record nearest the failure time and copy its `req_id`. Use a safe placeholder rather than pasting prompts, credentials, or response content into a command. Replace the example ID with the 32-hex value you captured: @@ -323,16 +314,9 @@ grep -F "req_id=${REQUEST_ID} " /daaf/scripts/provider_shim/logs/shim.log grep -E "req_id=${REQUEST_ID} .*event=(backend_error|disconnect|terminal|cleanup)" /daaf/scripts/provider_shim/logs/shim.log ``` -The first grep reconstructs the complete correlated lifecycle. Stable event names are `request_start`, `request_parsed`, `upstream_attempt`, `upstream_retry`, `upstream_headers`, `upstream_first_event`, `downstream_first_content`, `backend_error`, `disconnect`, `terminal`, and `cleanup`. Some milestones are conditional and attempts/retries may repeat, but the shim emits `event=terminal` exactly once and `event=cleanup` exactly once for each `/v1/messages` request. Read the focused records in causal order: +These two commands pull out the full record of what the shim did for that one request, in order: when it started, what the backend returned, and how it ended. That sequence is usually enough to see where a request went wrong. -1. **Inspect `event=backend_error`.** `backend_type`, `backend_code`, `anthropic_type`, and `failure_phase` identify the structured or transport failure the shim recorded. Error mapping follows an explicit precedence: a real non-2xx backend HTTP status maps first and takes precedence; otherwise backend `code` is checked before backend `type`. At either structured field, `context_length_exceeded` maps to Anthropic `invalid_request_error`, `server_error` maps to `api_error`, and an unknown value remains `api_error`. Free-form backend response bodies/messages are not retained or reflected because provider prose can contain request-derived material; the user-facing message is bounded, classification-derived text selected from status/type/code families or a fixed unknown fallback. Do not infer that every failure is backend-originated merely because the user saw an `api_error`: local translation, send, disconnect, and cleanup failures have distinct evidence. -2. **Inspect `event=disconnect`, if present.** This event is narrow evidence that the shim observed an ASGI `http.disconnect`; `observed_phase` records where it was observed. It does not prove what the user received or displayed, that the observation caused an upstream operation to be cancelled, or even that an upstream operation existed. Generic ASGI task cancellation is not itself evidence of client disconnect and does not create this event. If a backend-declared error appears earlier, a later disconnect is a subsequent observed condition, not evidence that the disconnect caused the backend error. If no backend failure was known first, the terminal outcome can instead be `disconnect`. -3. **Inspect the one `event=terminal` summary.** Check `outcome`, `attempts`, `retries`, `retry_reason`, `retry_source`, `transport_exception`, `failure_phase`, `terminal_frame_send`, `body_close_send`, `disconnect`, and `disconnect_phase`. The send states are `not_attempted`, `attempted`, `send_completed`, `skipped_disconnect`, or `write_failed`. Critically, `send_completed` means only that the awaited ASGI send returned; it does **not** establish client receipt, acknowledgment, or guaranteed delivery. A `write_failed` state is direct evidence of a downstream send failure; other conclusions about what the client displayed remain inference. -4. **Inspect the one `event=cleanup` record.** `status=completed` records clean shim-owned resource cleanup; `status=failed` plus the bounded `error` type records a cleanup failure. Cleanup evidence does not overwrite an earlier success, backend error, or disconnect outcome. Exactly-once guards make terminal and cleanup records stable accounting boundaries even when an exception or cancellation crosses finalization. - -Additional safe fields can help locate the failing phase without revealing request content: transport exception class and phase, monotonic `elapsed_ms`/`dur_ms`, attempt/retry counts and retry source/reason, the sent `client_req_id` and returned allowlisted provider ID (`upstream_req_id_header` / `upstream_req_id`) where available, normalized `http_version`, disconnect state, terminal semantic-frame state, body-close state, and cleanup status. The local `req_id` remains authoritative for joining shim records; client/provider IDs are separate metadata and never replace it. - -The lifecycle and diagnostic logs intentionally **never include** prompts or text, tool inputs or arguments, Base64 or decoded image bytes, image URLs or query strings, file IDs or names, credentials or cookies, raw SSE, complete request/response bodies, or free-form backend prose. Do not add those values to bug-report grep commands. Retry policy, timeouts, backoff, connection pooling, and heartbeat behavior are independent of this logging, so a new correlated failure should be diagnosed from the recorded sequence before proposing transport-policy changes. +Importantly, the log records only this lifecycle metadata — it **never** contains your prompts, the model's text, tool inputs, image bytes or URLs, file names, or credentials. That's what makes a `req_id` record safe to paste into a bug report. ### Q: My GPT session fails instantly with 429 errors on every request (Option F) @@ -357,11 +341,11 @@ Yes — through a supported route we've built carefully; because it's a newer ro The shim reads the OAuth token from `auth.json` and **refreshes it automatically** (the token lasts ~10 days and is never logged); if a refresh fails permanently, the shim log tells you to re-run `codex login --device-auth`. -**Context ceiling on this lane.** The Codex subscription backend enforces a much lower effective *input* ceiling than the model's true 1M window — measured at ~370,000 tokens for `gpt-5.6-sol` (2026-07-16: accepted at 369,941, rejected at 372,905). This is a backend-enforced limit on the Codex route, not a client setting: no `[1m]` hint or local declaration can raise it. Set `CLAUDE_CODE_MAX_CONTEXT_TOKENS=370000` on this lane (not the API-key lane's `1050000`) to align Claude Code and DAAF's physical-window accounting with the measured backend ceiling. DAAF uses this denominator for utilization, statusline rendering, severity, and behavioral stop/restart guidance. It keeps automatic compaction disabled and does not implement a transport-level request blocker; the backend remains the ultimate hard ceiling. The matched runtime consumers defensively cap stale or higher resolved values at 370,000 while preserving lower positive declarations. A breach surfaces as a clean `invalid_request_error` (`context_length_exceeded`) rather than a retryable-looking 502. Re-measure with `scripts/provider_shim/probe_context_ceiling.py` if OpenAI changes the backend. +**Context ceiling on this lane.** The Codex subscription backend caps the usable input window much lower than the model's true 1M window — measured at ~370,000 tokens for `gpt-5.6-sol` (2026-07-16). This is a hard backend limit on the Codex route, not a client setting: no `[1m]` hint or local declaration can raise it. Set `CLAUDE_CODE_MAX_CONTEXT_TOKENS=370000` on this lane (not the API-key lane's `1050000`), so Claude Code and DAAF measure usage against the real ceiling. DAAF applies this automatically and keeps automatic compaction off; the backend itself remains the ultimate hard limit. If you exceed it, you'll get a clean "context length exceeded" error rather than a confusing retry loop. **Parallel use.** Running more than one container is the clean way to parallelize: each container does its own `codex login` (an independent token grant — no collision), and they share only your ChatGPT usage pool, so running them in parallel just draws that pool down faster. For several codex tools inside a *single* container, give each its own login under a separate `CODEX_HOME` to avoid a rare token-rotation race. -**Under the hood (safe to skip).** Request translation, tools, and the content-block lifecycle are shared across both lanes; only authentication and endpoint differ. On this lane the shim always requests upstream SSE — including for inbound non-stream calls, whose terminal response is accumulated into JSON — and malformed/missing terminals or invalid block ordering fail explicitly. The tolerated live `arguments.done` event that omits `name` no longer produces one warning per event: the shim logs the first occurrence and then periodic aggregate summaries with counts. Hard or conflicting protocol errors remain visible and still fail explicitly; an invalid usage token counter is dropped rather than discarding a completed response. One response-formatting rule is ChatGPT-only: reliably identified reasoning-summary part boundaries are preserved as blank lines (`\n\n`), while OpenAI/API-key mode retains exact concatenation and its real upstream non-stream JSON path. +**Optional reading — under the hood.** Both lanes handle tools and message content the same way; only the login method and endpoint differ. If the backend ever sends something malformed or out of order, the shim fails loudly rather than quietly passing along corrupted output. A few small formatting and logging details differ between the two lanes, but nothing you need to configure or worry about. See the full walkthrough in [**01. Installation & Quick Start — Option F, alternate lane: ChatGPT subscription**](01_installation_and_quickstart.md#option-f-alternate-lane-chatgpt-subscription-codex-backend). @@ -373,14 +357,12 @@ Occasionally a GPT turn ends with a reasoning-only block and no visible text, wh **Short answer: you probably don't need to touch this.** By default GPT already runs at the highest reasoning effort (`high`), matching how DAAF runs Claude — so for most people, leaving everything unset is exactly right. Reach for the controls below only if you want to dial effort *down* (for speed or cost) or pin it explicitly. One gotcha to know first: the reasoning-effort selector in the Claude Code `/model` menu does **not** work for GPT — use the environment-variable methods below instead. -Here's how it works under the hood. On the direct-OpenAI shim lane, every request to OpenAI carries a `reasoning.effort` value, and the shim resolves it from a **four-tier precedence chain** — the first tier present wins: +You have two ways to set it, both in `environment_settings.txt`: -1. **Per-request signal from Claude Code** — when Claude Code specifies an effort for a turn, that takes precedence over the tiers below. **Important caveat:** the `/model` reasoning-effort selector in the Claude Code UI does **not** work for GPT slugs. Claude Code gates that selector by model-ID pattern and, for an unrecognized (GPT) slug, pins the per-request signal to `high` on *every* request regardless of what you select — so the selector is inert. To keep it from masking your real preference, the shim treats an inbound `high` as that pin (i.e. as unset) and falls through to the tiers below; any *other* inbound value (`low`, `medium`, etc.) is still honored. The practical consequence: **steer GPT effort with tier 2 or tier 3 below, not the `/model` selector.** -2. **A `#` suffix on the model slug** — append it in `environment_settings.txt`, e.g. `ANTHROPIC_DEFAULT_SONNET_MODEL=gpt-5.6-terra[1m]#medium`. This works *alongside* the `[1m]` window hint (Claude Code consumes `[1m]` locally and passes `#medium` through), and the shim strips the suffix before the request reaches OpenAI — the backend only ever sees the bare `gpt-5.6-terra`. -3. **The `SHIM_REASONING_EFFORT` env var** — a single default for the whole shim, applied when there is no usable per-request signal and no slug suffix. -4. **The built-in default, `high`** — used when none of the above is set, for posture parity with DAAF's Claude sessions. +1. **A `#` suffix on the model slug** — e.g. `ANTHROPIC_DEFAULT_SONNET_MODEL=gpt-5.6-terra[1m]#medium`. This works alongside the `[1m]` window hint, and the shim strips the suffix before the request reaches OpenAI (the backend only ever sees the bare `gpt-5.6-terra`). +2. **The `SHIM_REASONING_EFFORT` env var** — a single default applied to the whole shim. -Valid values are `none`, `low`, `medium`, `high`, `xhigh`, and `max` (`max` is gpt-5.6-only; `none` disables reasoning). An unrecognized value at any tier is ignored with a log warning and the next tier applies. You can confirm what the shim resolved by checking the shim log (`/daaf/scripts/provider_shim/logs/shim.log`): each request line ends with `effort=:`, where source is one of `inbound`, `slug`, `env`, or `default`. (Because of that demotion, an inbound `high` with no slug/env override logs `effort=high:default`, not `effort=high:inbound`.) Most users need to set nothing — leaving everything unset gives `high` everywhere. Both the env var and slug suffixes are read at shim startup / request time; changes to `environment_settings.txt` require the usual container recreate. See also the Option F [reasoning-effort paragraph](01_installation_and_quickstart.md#option-f-openai-api-directly-daaf-provider-shim) in the installation guide. If GPT replies also feel *shorter* or *terser* than you expect, that is a separate knob — see [GPT responses feel terse compared to Claude](#q-gpt-responses-feel-terse-compared-to-claude-option-f) below. +Valid values are `none`, `low`, `medium`, `high`, `xhigh`, and `max` (`max` is gpt-5.6-only; `none` disables reasoning). An explicit setting always wins over the built-in `high` default, and a per-slug suffix wins over the shim-wide env var; the shim logs which source it used on each request line in `/daaf/scripts/provider_shim/logs/shim.log` (as `effort=:`). Most people set nothing and get `high` everywhere. Changes to `environment_settings.txt` take effect after the usual container recreate. See also the Option F [reasoning-effort paragraph](01_installation_and_quickstart.md#option-f-openai-api-directly-daaf-provider-shim) in the installation guide. If GPT replies also feel *shorter* or *terser* than you expect, that is a separate knob — see [GPT responses feel terse compared to Claude](#q-gpt-responses-feel-terse-compared-to-claude-option-f) below. ### Q: GPT responses feel terse compared to Claude (Option F) @@ -483,7 +465,7 @@ Cost remains a meaningful barrier to entry for DAAF, but it's shrinking. As open ### Q: How much disk space does DAAF use? -The Docker image is roughly **8.6 GB** after building (the exact size varies with your Docker version and platform). It includes an Ubuntu 24.04 (Noble) base image, Python 3.12, **52 explicitly pinned Python distributions** — 50 with import targets plus metadata-only `svy-rs==0.10.0` and `svy-io==0.1.1` (alongside the provider shim's exact `httpx==0.28.1` and `uvicorn==0.51.0` runtime pins) — covering data science, geospatial, econometrics, visualization, ML, and shim infrastructure; geospatial system libraries (GDAL/GEOS/PROJ); Claude Code; R; 60+ pinned R packages (tidyverse, fixest, survey, sf, and more); and the Quarto CLI. The R runtime, packages, and Quarto account for roughly **2 GB** of that total (approximately 8.6 GB with R versus 6.4 GB without). Docker also keeps build cache layers, so total Docker disk usage may be somewhat higher. +The Docker image is roughly **8.6 GB** after building (the exact size varies with your Docker version and platform). It includes an Ubuntu 24.04 (Noble) base image, Python 3.12, **~52 pinned Python packages** covering data science, geospatial, econometrics, visualization, ML, and shim infrastructure; geospatial system libraries (GDAL/GEOS/PROJ); Claude Code; R; 60+ pinned R packages (tidyverse, fixest, survey, sf, and more); and the Quarto CLI. The R runtime, packages, and Quarto account for roughly **2 GB** of that total (approximately 8.6 GB with R versus 6.4 GB without). Docker also keeps build cache layers, so total Docker disk usage may be somewhat higher. Beyond the image, your Docker volume will grow as you create research projects. Each project accumulates scripts, parquet data files, session logs, and notebooks. A typical full-pipeline project might add 50-500 MB depending on how many datasets you fetch and how large they are. @@ -907,7 +889,7 @@ bash view_quarto.sh research/2026-01-24_Your_Project/output/analysis/notebook.qm .\view_quarto.ps1 research/2026-01-24_Your_Project/output/analysis/notebook.qmd ``` -The script renders the selected notebook to a single self-contained HTML file inside the container, copies it out to a `quarto_html/` folder next to your `docker-compose.yml`, and opens it in your browser -- handling container startup automatically. For a Full Pipeline Stage 9 notebook, this render validates the document and explicitly enabled previews; archived Stage 5-8 script chunks remain disabled, so render success is not a full analysis reproduction. Output uses the notebook's flat basename, so rendering two notebooks named `report.qmd` targets the same `quarto_html/report.html` and the later render overwrites the earlier one. Set `QUARTO_HTML_DIR` to a different host directory when both must be retained. +The script renders the selected notebook to a single self-contained HTML file, copies it out to a `quarto_html/` folder next to your `docker-compose.yml`, and opens it in your browser -- starting the container automatically if it isn't already running. If you'd rather do it by hand, render from inside the container and copy the result out yourself: @@ -938,9 +920,7 @@ This isn't an error -- it's DAAF being responsible about Claude's working memory Claude has a finite context window. As a session progresses and Claude processes more information, that window fills up. Even with large context windows (up to 1M tokens), quality can degrade well before the window is full, so DAAF enforces both percentage-based and absolute token thresholds — whichever fires first. -The exact trigger points depend on the model's **context-quality threshold profile**. DAAF detects the exact model identifier automatically — nothing to configure. Claude Fable/Mythos models retain the validated extended-horizon profile. Exact terminal GPT 5.6 Sol model slugs, bare or provider-prefixed, use the standard percentage gates while retaining Sol's higher validated absolute gates. Opus, Sonnet, unknown model IDs, every other GPT variant, GLM models, and all other alternative-provider models use the conservative-default profile unless individually validated and registered. - -For GPT 5.6 Sol, the terminal model slug must be exactly `gpt-5.6-sol` or `gpt-5.6-sol[1m]`; the identifier may be bare or may contain one or more provider path prefixes ending in `/`. Malformed left-boundary strings such as `xgpt-5.6-sol`, `foo-gpt-5.6-sol`, and `vendor/notgpt-5.6-sol` remain conservative, as do right-side suffix or trailing variants. Threshold-profile selection is version-specific and independent of physical context-window mapping. Terra, Luna, Pro, mini, chat, date snapshots, future variants, and trailing modifiers remain conservative unless separately validated, even when a wider GPT 5.6 variant maps to a 1,050,000-token physical window. That 1,050,000-token physical window is itself route-conditional: it holds on the OpenAI-API-key and OpenRouter routes, while the ChatGPT-subscription (Codex) shim lane is backend-capped near ~370,000 tokens (measured for Sol, 2026-07-16), which DAAF's hooks and statuslines lane-gate automatically. On that 370k physical window, exact Sol's standard percentage boundaries are 148k, 222k, and 277.5k tokens, before its retained 300k, 400k, and 500k absolute gates. Physical-window mapping and quality-threshold profile selection remain separate. GPT is not part of Claude's Fable/Mythos family: exact Sol now shares standard percentages but retains the higher validated Sol absolute gates. The four status levels and what DAAF does at each are the same for every model; only the trigger points differ: +The exact trigger points vary by model — DAAF detects which model you're running automatically and applies the matching profile, so there's nothing to configure. Broadly, models DAAF has specifically validated (Claude Fable/Mythos, and exact GPT 5.6 Sol) get more generous thresholds, while everything else — Opus, Sonnet, other GPT variants, GLM, and any unrecognized model — uses conservative defaults. One wrinkle worth knowing: on the ChatGPT-subscription (Codex) lane the usable context window itself is capped lower (~370k tokens), so the thresholds there fire earlier than on the full-window API route. The four status levels and what DAAF does at each are the same for every model; only the trigger points differ: | Threshold Profile | Membership | ELEVATED at | HIGH at | CRITICAL at | |-------------------|------------|-------------|---------|-------------| From aec170fb2bc5ea08bfe27f3bbfe3eeaa33cd6ce1 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 15:51:21 +0000 Subject: [PATCH 08/66] feat(host+docs): bind-mount file access, shared-workspace multi-install support, and Codex plugin guidance Two user-guidance strands with supporting infrastructure: File management (bind mounts): commented read-only bind-mount opt-in block in docker-compose.yml (long syntax, /host_data target); install-guide section with platform permission matrix (UID-1000 pass-through on Linux/WSL-fs, translated ownership on macOS//mnt/c), write-enabled advanced path, and an explicit reproducibility trade-off callout (copy inputs into project data/raw); FAQ troubleshooting entry; CLAUDE.md provenance bullet (/host_data read-only, outside backup/audit boundary); cross-links from best-practices and extending guides; reproducibility-mode acknowledgment of externally-guaranteed inputs. Simultaneous Claude+ChatGPT (shared workspace): DAAF_DATA_VOLUME_NAME override (unset = legacy _daaf-data; set = literal volume name) centralized in daaf_lib resolvers (bash + PowerShell), propagated to all 17 whitelist copies, backup/restore/install derivations in both twins, and install-time seeding; commented external:true + name: opt-in block under the daaf-data volume; 5-step conversion procedure in the install guide (edit-in-container-then- rebuild); shared-workspace FAQ; migrate_daaf legacy-scope comments. Codex plugin (codex-plugin-cc): setup subsection framed around the two race-free configurations (dual-container; single Anthropic-route container), one warning against combining with the ChatGPT shim lane (refresh-token rotation race), device-code auth default; explicit nodejs install + build-time Node >= 18.18 floor assertion at the end of the Dockerfile root section. Tests: resolver precedence cases (bats + new Pester twin), install seeding case, env-example key lint 6->7, whitelist parity pin 6->7 keys, test_migration Phase 8 override sub-scenario in both harness twins. All suites green (parity 9/9, daaf_lib 32/32, install 42/42, env-example 10/10, backup 47/47, restore 50/50, test_migration bats 49/49 / Pester 56/56, conventions lint 0 failures). Co-Authored-By: Claude Fable 5 --- .../reproducibility-verification-mode.md | 2 + .../references/user-support-mode.md | 2 + CLAUDE.md | 1 + Dockerfile | 17 ++ docker-compose.yml | 22 +++ scripts/host/backup_daaf.ps1 | 7 +- scripts/host/backup_daaf.sh | 8 +- scripts/host/daaf_lib.ps1 | 54 +++++- scripts/host/daaf_lib.sh | 53 +++++- scripts/host/environment_settings_example.txt | 11 ++ scripts/host/install.ps1 | 8 +- scripts/host/install.sh | 9 +- scripts/host/migrate_daaf.ps1 | 6 + scripts/host/migrate_daaf.sh | 6 + scripts/host/rebuild_daaf.ps1 | 2 +- scripts/host/rebuild_daaf.sh | 2 +- scripts/host/restore_from_backup.ps1 | 7 +- scripts/host/restore_from_backup.sh | 8 +- scripts/host/run_daaf.ps1 | 2 +- scripts/host/run_daaf.sh | 2 +- scripts/host/run_vscode.ps1 | 2 +- scripts/host/run_vscode.sh | 2 +- scripts/host/test_migration.ps1 | 30 ++++ scripts/host/test_migration.sh | 33 ++++ scripts/host/update_daaf.ps1 | 2 +- scripts/host/update_daaf.sh | 2 +- scripts/host/view_logs.ps1 | 2 +- scripts/host/view_logs.sh | 2 +- scripts/host/view_notebooks.ps1 | 2 +- scripts/host/view_notebooks.sh | 2 +- scripts/host/view_quarto.ps1 | 2 +- scripts/host/view_quarto.sh | 2 +- tests/bash/daaf_lib.bats | 57 +++++++ tests/bash/environment_settings_example.bats | 6 +- tests/bash/install.bats | 16 ++ tests/bash/settings_whitelist_parity.bats | 35 ++-- tests/powershell/daaf_lib.Tests.ps1 | 65 ++++++++ .../01_installation_and_quickstart.md | 157 +++++++++++++++++- user_reference/02_understanding_daaf.md | 2 + user_reference/03_best_practices.md | 2 + user_reference/04_extending_daaf.md | 2 + user_reference/07_faq_technical.md | 18 ++ 42 files changed, 610 insertions(+), 62 deletions(-) create mode 100644 tests/powershell/daaf_lib.Tests.ps1 diff --git a/.claude/skills/daaf-orchestrator/references/reproducibility-verification-mode.md b/.claude/skills/daaf-orchestrator/references/reproducibility-verification-mode.md index 612bda28..64127ab4 100644 --- a/.claude/skills/daaf-orchestrator/references/reproducibility-verification-mode.md +++ b/.claude/skills/daaf-orchestrator/references/reproducibility-verification-mode.md @@ -28,6 +28,8 @@ Two decisions must be confirmed at **mode confirmation** AND reconfirmed **after At mode confirmation: present defaults, ask user to confirm or adjust. After RV-1: present the script inventory, original-artifact inventory, and evidence gaps, then ask the user to reconfirm both decisions with the scope now concrete. If the user chose frozen data but `original_files/data/raw/` has no copied raw files, stop and require a new choice before RV-2; do not silently fall back to re-fetching or to processed data. +> **Bind-mounted inputs depend on externally-guaranteed files.** If the original analysis read inputs from a user-configured host bind mount (`/host_data`) rather than from data archived inside the project, those files live outside DAAF's hash/backup boundary — verification then depends on the user supplying the *same* mount contents. Confirm the frozen `data/raw/` copy (or an equivalent user-provided snapshot) is present before treating such a reproduction as complete; DAAF cannot hash-verify data it never archived. + --- ## Reproducibility Verification Workflow diff --git a/.claude/skills/daaf-orchestrator/references/user-support-mode.md b/.claude/skills/daaf-orchestrator/references/user-support-mode.md index 12c2b1d5..8b9a0157 100644 --- a/.claude/skills/daaf-orchestrator/references/user-support-mode.md +++ b/.claude/skills/daaf-orchestrator/references/user-support-mode.md @@ -128,6 +128,8 @@ Users frequently have questions about the tools DAAF runs on, not just DAAF itse | Question About | Where to Look (local) | Authoritative External Docs | |----------------|----------------------|----------------------------| | Docker setup, container management, resource allocation, security | `user_reference/01_installation_and_quickstart.md`, `user_reference/07_faq_technical.md` | https://docs.docker.com/reference/ | +| Bind mounts / sharing host folders into the container | `user_reference/01_installation_and_quickstart.md`, `user_reference/07_faq_technical.md` | https://docs.docker.com/engine/storage/bind-mounts/ | +| Running Claude and ChatGPT installs in parallel / sharing one research workspace | `user_reference/01_installation_and_quickstart.md`, `user_reference/07_faq_technical.md` | https://docs.docker.com/reference/compose-file/volumes/ | | Git usage, version control, diffs, commits | `user_reference/01_installation_and_quickstart.md`, `user_reference/03_best_practices.md` | https://git-scm.com/docs | | Claude Code features, configuration, model selection, IDE integration | `user_reference/07_faq_technical.md`, `user_reference/01_installation_and_quickstart.md` | https://code.claude.com/docs/en/overview | | Running without Docker, alternative AI providers | `user_reference/07_faq_technical.md` | — | diff --git a/CLAUDE.md b/CLAUDE.md index c898e1bf..61488c2d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -276,6 +276,7 @@ Context management is NEVER about reducing the quality or completeness of work. - You MUST NEVER write working files to `/tmp` (redirects, `cp`/`mv`/`tee`/`mkdir`/`touch`, downloads, `sed -i`, archive extraction, or `git clone` targeting `/tmp`). `/tmp` is outside the Docker-volume backup boundary and the audit trail, and the session log viewer marks `/tmp` paths as ephemeral, unarchived references — files written there are lost silently. Temporary and intermediate files belong inside the project (see § Project Conventions > Scratch Files). - **Exception — reads are fine:** DAAF's own hooks and statuslines legitimately cache coordination state in `/tmp` (e.g. `/tmp/claude-ctx-window-*`, `/tmp/claude-model-*`). *Reading* those caches via Bash is permitted; only *writes* to `/tmp` are blocked. Reading a `/tmp` cache and redirecting the output into the project is the sanctioned rescue pattern. +- **Read-only external data via bind mount (`/host_data`):** when a user has configured a bind mount, `/host_data` is a **read-only** external data source that sits outside the backup and audit boundary. Read from it freely, but never write to it, and copy any inputs you actually use into the project's own data directory (with the usual raw-data conventions) so provenance is preserved and the analysis stays reproducible from the archived working copy. ### Safety-System Integrity diff --git a/Dockerfile b/Dockerfile index 8a593e3f..d31519d7 100644 --- a/Dockerfile +++ b/Dockerfile @@ -706,6 +706,23 @@ RUN mkdir -p /home/appuser/.local/share/code-server/User \ && chown -R appuser:appuser /home/appuser/.local \ /home/appuser/.config +# ============================================ +# Node.js runtime for codex-plugin-cc +# ============================================ +# codex-plugin-cc (the Codex plugin for Claude Code) needs a Node.js runtime in +# the container. Node already arrives transitively via libnode-dev (the R gt/V8 +# stack), but that presence is incidental -- install `nodejs` explicitly so a +# future R-stack change cannot silently drop it (no exact version pin, per DAAF +# convention: the distro's packaged Node tracks the base image release). The +# build-time floor assertion FAILS the build loudly if the packaged Node is +# older than 18.18 (the minimum codex-plugin-cc supports), rather than letting +# a too-old runtime surface as an obscure plugin failure later. +RUN apt-get update && apt-get install -y --no-install-recommends \ + nodejs \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* \ + && node -e 'const v=process.versions.node, [maj,min]=v.split(".").map(Number); if (maj<18 || (maj===18 && min<18)) { console.error("ERROR: Node "+v+" is older than the 18.18 floor required by codex-plugin-cc"); process.exit(1); } console.log("Node "+v+" satisfies the codex-plugin-cc >= 18.18 floor");' + # ============================================ # Set up working directory # ============================================ diff --git a/docker-compose.yml b/docker-compose.yml index 793b7257..c38f740a 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -50,6 +50,18 @@ services: # recreation, and image rebuilds. Only `docker compose down -v` or an # explicit `docker volume rm` deletes it. - daaf-claude-config:/home/appuser/.claude + # OPTIONAL host-folder bind mount (commented out by default). Uncomment the + # four lines below and set `source` to an ABSOLUTE path on your host to + # expose a host folder to the container at /host_data. Read-only + # (`read_only: true`) is strongly recommended so the container cannot alter + # your host files. IMPORTANT: a bind mount is OUTSIDE the DAAF backup/audit + # boundary -- files under /host_data are NOT captured by backup_daaf.sh and + # are not part of the reproducibility trail. See the Installation Guide, + # section "Linking Host Folders into the Container (Bind Mounts)". + # - type: bind + # source: /absolute/path/on/host + # target: /host_data + # read_only: true environment: # Relocate ~/.claude.json and credentials into the persisted .claude # directory (above) so they survive rebuilds too. Set in compose (not the @@ -84,4 +96,14 @@ services: volumes: daaf-data: # Docker manages this storage + # OPTIONAL shared-workspace mode (commented out by default). Uncomment BOTH + # lines below to point this install at an EXISTING data volume shared with + # another install, instead of letting Docker manage a per-project volume. Set + # `name:` to the full volume name and set the matching DAAF_DATA_VOLUME_NAME in + # environment_settings.txt so the host tools (backup_daaf.sh / restore) resolve + # the same volume. The named volume must ALREADY EXIST -- Docker will not + # create an `external: true` volume. See the Installation Guide, section + # "Sharing One Research Workspace Across Two Installs (Advanced)". + # external: true + # name: daaf_daaf-data daaf-claude-config: # Docker manages this storage (Claude Code state) diff --git a/scripts/host/backup_daaf.ps1 b/scripts/host/backup_daaf.ps1 index bdfaf168..c4bb0a61 100644 --- a/scripts/host/backup_daaf.ps1 +++ b/scripts/host/backup_daaf.ps1 @@ -74,7 +74,7 @@ if ($env:DAAF_DRY_RUN -eq "1") { function Import-DaafSettingsInline { param([string]$SettingsFile = "./environment_settings.txt") if (-not (Test-Path -LiteralPath $SettingsFile)) { return } - $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH') + $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH', 'DAAF_DATA_VOLUME_NAME') # -Encoding UTF8: PS 5.1's bare Get-Content misreads BOM-less UTF-8 as ANSI # (cp1252); the settings writer is BOM-less UTF-8, so reads are pinned to match. foreach ($rawLine in (Get-Content -LiteralPath $SettingsFile -Encoding UTF8)) { @@ -120,9 +120,14 @@ Set-StrictMode -Version 3.0 # derives the prefix from the project name (default "daaf"), so a second instance # with DAAF_PROJECT_NAME=daaf2 owns the volume "daaf2_daaf-data". Default unset => # "daaf_daaf-data" (byte-for-byte identical to the previous hardcoded value). +# DAAF_DATA_VOLUME_NAME, when set, overrides the whole derivation with a verbatim +# full volume name (the shared-workspace escape hatch); unset => the derived +# default. Matches Resolve-DaafDataVolumeName in daaf_lib.ps1 (inlined here because +# this standalone script does not dot-source the library). $projectName = "daaf" if ($env:DAAF_PROJECT_NAME) { $projectName = $env:DAAF_PROJECT_NAME } $VolumeName = "${projectName}_daaf-data" +if ($env:DAAF_DATA_VOLUME_NAME) { $VolumeName = $env:DAAF_DATA_VOLUME_NAME } # Second volume: Claude Code state (auth/credentials, session history and # transcripts, plugins, ~/.claude.json). Backed up into a dedicated hidden # subfolder of the backup so it does not contaminate the data-volume file counts diff --git a/scripts/host/backup_daaf.sh b/scripts/host/backup_daaf.sh index 80251da1..ac8bc1f3 100755 --- a/scripts/host/backup_daaf.sh +++ b/scripts/host/backup_daaf.sh @@ -43,7 +43,7 @@ _daaf_load_settings() { line="$(printf '%s' "${line}" | tr -d '\r')" case "${line}" in ''|'#'*) continue ;; esac case "${line}" in - DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*) + DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*|DAAF_DATA_VOLUME_NAME=*) key="${line%%=*}"; val="${line#*=}" case "${val}" in \"*\") val="${val#\"}"; val="${val%\"}" ;; @@ -64,7 +64,11 @@ _daaf_load_settings # derives the prefix from the project name (default "daaf"), so a second instance # with DAAF_PROJECT_NAME=daaf2 owns the volume "daaf2_daaf-data". Default unset => # "daaf_daaf-data" (byte-for-byte identical to the previous hardcoded value). -VOLUME_NAME="${DAAF_PROJECT_NAME:-daaf}_daaf-data" +# DAAF_DATA_VOLUME_NAME, when set, overrides the whole derivation with a verbatim +# full volume name (the shared-workspace escape hatch); unset => the derived +# default. Matches resolve_data_volume_name in daaf_lib.sh (inlined here because +# this standalone script does not source the library). +VOLUME_NAME="${DAAF_DATA_VOLUME_NAME:-${DAAF_PROJECT_NAME:-daaf}_daaf-data}" # Second volume: Claude Code state (auth/credentials, session history and # transcripts, plugins, ~/.claude.json). Backed up into a dedicated hidden # subfolder of the backup so it does not contaminate the data-volume file diff --git a/scripts/host/daaf_lib.ps1 b/scripts/host/daaf_lib.ps1 index faef00c2..e0320806 100644 --- a/scripts/host/daaf_lib.ps1 +++ b/scripts/host/daaf_lib.ps1 @@ -11,7 +11,10 @@ # helpers the Bash Control Panel relies on, adapted to PowerShell idiom: # Import-DaafSettingsFile -- export the whitelisted DAAF_* vars (four multi-instance # keys + the DAAF_DEV build flag + the DAAF_BRANCH -# updater ref) from environment_settings.txt +# updater ref + the DAAF_DATA_VOLUME_NAME data-volume +# override) from environment_settings.txt +# Resolve-DaafDataVolumeName -- echo the resolved data-volume name, honoring the +# optional DAAF_DATA_VOLUME_NAME full-name override # Set-DaafSettingsKey -- insert/update a single KEY=value line in a settings # file (write counterpart to Import-DaafSettingsFile) # Read-DaafLine -- read one input line, working under redirected stdin (CI) @@ -67,13 +70,16 @@ if (Get-Command Read-DaafLine -ErrorAction SilentlyContinue) { return } # --- Multi-Instance / Build-Flag Settings Loader --- # PowerShell counterpart to daaf_lib.sh load_daaf_settings. Bridges -# environment_settings.txt -> process environment for the six whitelisted +# environment_settings.txt -> process environment for the seven whitelisted # DAAF_* variables: the four multi-instance keys so `docker compose` # interpolation in docker-compose.yml (${DAAF_PROJECT_NAME:-daaf}, # ${DAAF_PORT_*:-27xx}) resolves them, plus DAAF_DEV, the opt-in # BUILD flag consumed as `--build-arg DAAF_DEV=${DAAF_DEV:-0}`, plus DAAF_BRANCH, # the updater's target ref (read env-only today; whitelisting it here lets a -# value persisted in environment_settings.txt reach update_daaf.ps1). +# value persisted in environment_settings.txt reach update_daaf.ps1), plus +# DAAF_DATA_VOLUME_NAME, the optional full-name override for the research-workspace +# data volume (consumed by Resolve-DaafDataVolumeName below and by the backup / +# restore tools). # # WHY: environment_settings.txt is a compose `env_file` (feeds the CONTAINER # env only). Compose *interpolation* (and build args) read the host/process @@ -82,7 +88,7 @@ if (Get-Command Read-DaafLine -ErrorAction SilentlyContinue) { return } # ports / build flag to take effect. # # PARSING SAFETY: we never dot-source the file (it holds API keys with arbitrary -# characters). We extract only the six known DAAF_* keys via a line scan and a +# characters). We extract only the seven known DAAF_* keys via a line scan and a # regex on KEY=VALUE. CR is stripped for CRLF tolerance. # # PRECEDENCE: an already-set process env var WINS over the file value (matches @@ -97,7 +103,7 @@ function Import-DaafSettingsFile { return } - $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH') + $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH', 'DAAF_DATA_VOLUME_NAME') # -Encoding UTF8 pins BOM-less UTF-8 decoding: a bare Get-Content on Windows # PowerShell 5.1 misreads BOM-less UTF-8 as legacy ANSI (cp1252), corrupting @@ -132,6 +138,44 @@ function Import-DaafSettingsFile { } } +# --- Data Volume Name Resolver --- +# PowerShell counterpart to daaf_lib.sh resolve_data_volume_name. Return the +# Docker volume name that holds the DAAF research workspace (/daaf), honoring the +# optional DAAF_DATA_VOLUME_NAME full-name override. +# +# PRECEDENCE (returned to the caller): +# 1. DAAF_DATA_VOLUME_NAME, when set non-empty -> used VERBATIM as the full +# volume name (no project prefix is added). This is the escape hatch for +# pointing two installs at ONE shared workspace volume; it must match the +# `name:` under the external `daaf-data` block in docker-compose.yml. +# 2. Otherwise "_daaf-data", where is DAAF_PROJECT_NAME or +# the "daaf" default -- the Compose-prefixed default. So an UNSET +# DAAF_DATA_VOLUME_NAME is byte-for-byte identical to the historical +# hardcoded derivation, and a second instance with DAAF_PROJECT_NAME=daaf2 +# still owns "daaf2_daaf-data". +# A set-but-EMPTY DAAF_DATA_VOLUME_NAME falls through to the derived default: an +# empty string is falsy in PowerShell, so the `if ($env:DAAF_DATA_VOLUME_NAME)` +# guard treats empty and unset alike, matching the bash `:-` semantics and the +# way Import-DaafSettingsFile adopts a whitelisted key only when it is non-empty. +# +# The claude-config volume is deliberately NOT resolved here: per-install auth +# isolation is by design, so it keeps its project-prefixed default everywhere and +# has no analogous override. +# +# Callers that need a value persisted in environment_settings.txt to win should +# Import-DaafSettingsFile first (which bridges DAAF_DATA_VOLUME_NAME into the +# process env), then call this. Strict-clean: only $env: reads and string ops. +function Resolve-DaafDataVolumeName { + [CmdletBinding()] + param() + if ($env:DAAF_DATA_VOLUME_NAME) { + return $env:DAAF_DATA_VOLUME_NAME + } + $projectName = "daaf" + if ($env:DAAF_PROJECT_NAME) { $projectName = $env:DAAF_PROJECT_NAME } + return "${projectName}_daaf-data" +} + # --- Line Input --- # Read one line of input from the user, compatible with both interactive # terminals and redirected stdin (CI pipelines). Mirrors the pattern used in diff --git a/scripts/host/daaf_lib.sh b/scripts/host/daaf_lib.sh index be1e4a97..c4ff708d 100755 --- a/scripts/host/daaf_lib.sh +++ b/scripts/host/daaf_lib.sh @@ -12,7 +12,10 @@ # Functions provided: # load_daaf_settings -- export the whitelisted DAAF_* vars (four multi-instance # keys + the DAAF_DEV build flag + the DAAF_BRANCH updater -# ref) from environment_settings.txt +# ref + the DAAF_DATA_VOLUME_NAME data-volume override) from +# environment_settings.txt +# resolve_data_volume_name -- echo the resolved data-volume name, honoring the +# optional DAAF_DATA_VOLUME_NAME full-name override # upsert_settings_key -- insert/update a single KEY=value line in a settings # file (write counterpart to load_daaf_settings) # setup_colors -- populate color variables (respects NO_COLOR + non-TTY) @@ -30,16 +33,20 @@ fi _DAAF_LIB_LOADED=1 # --- Multi-Instance / Build-Flag Settings Loader --- -# Bridge environment_settings.txt -> host shell environment for the six +# Bridge environment_settings.txt -> host shell environment for the seven # whitelisted DAAF_* variables: the four multi-instance keys # (DAAF_PROJECT_NAME, DAAF_PORT_MARIMO, DAAF_PORT_LOGVIEWER, DAAF_PORT_VSCODE) # plus DAAF_DEV, the opt-in BUILD flag consumed as # `--build-arg DAAF_DEV=${DAAF_DEV:-0}` in docker-compose.yml, plus DAAF_BRANCH, # the updater's target ref (read env-only today; whitelisting it here lets a -# value persisted in environment_settings.txt reach update_daaf.sh). The build -# flag and branch ref ride the same bridge for the same reason: env_file feeds -# the container env only, while compose interpolation (and build args) resolve -# from the host shell env. +# value persisted in environment_settings.txt reach update_daaf.sh), plus +# DAAF_DATA_VOLUME_NAME, the optional full-name override for the research-workspace +# data volume (consumed by resolve_data_volume_name below and by the backup / +# restore tools; whitelisting it lets a value persisted in the settings file reach +# those tools and compose interpolation of the external `daaf-data` volume). The +# build flag, branch ref, and data-volume override ride the same bridge for the +# same reason: env_file feeds the container env only, while compose interpolation +# (and build args) resolve from the host shell env. # # WHY THIS EXISTS: environment_settings.txt is wired into docker-compose.yml as a # service-level `env_file`, which feeds the CONTAINER environment only. Docker @@ -54,7 +61,7 @@ _DAAF_LIB_LOADED=1 # PARSING SAFETY: we deliberately do NOT `source`/`.` the file. It holds API keys # with arbitrary characters (quotes, $, backticks, spaces) that would be # interpreted by the shell -- a correctness and safety hazard. We extract only the -# six known DAAF_* keys via a line-oriented grep/sed/case scan, stripping CR for +# seven known DAAF_* keys via a line-oriented grep/sed/case scan, stripping CR for # CRLF tolerance (matches how the rest of the codebase handles container output). # # PRECEDENCE: an already-set shell environment variable WINS over the file value. @@ -80,9 +87,9 @@ load_daaf_settings() { case "${line}" in ''|'#'*) continue ;; esac - # Only lines of the form KEY=VALUE for our six known keys. + # Only lines of the form KEY=VALUE for our seven known keys. case "${line}" in - DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*) + DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*|DAAF_DATA_VOLUME_NAME=*) key="${line%%=*}" val="${line#*=}" # Strip one layer of surrounding quotes if present (tolerant of @@ -104,6 +111,34 @@ load_daaf_settings() { return 0 } +# --- Data Volume Name Resolver --- +# Echo the Docker volume name that holds the DAAF research workspace (/daaf), +# honoring the optional DAAF_DATA_VOLUME_NAME full-name override. +# +# PRECEDENCE (printed to stdout, captured by callers via `$(resolve_data_volume_name)`): +# 1. DAAF_DATA_VOLUME_NAME, when set non-empty -> used VERBATIM as the full +# volume name (no project prefix is added). This is the escape hatch for +# pointing two installs at ONE shared workspace volume; it must match the +# `name:` under the external `daaf-data` block in docker-compose.yml. +# 2. Otherwise "${DAAF_PROJECT_NAME:-daaf}_daaf-data" -- the Compose-prefixed +# default. So an UNSET DAAF_DATA_VOLUME_NAME is byte-for-byte identical to the +# historical hardcoded derivation, and a second instance with +# DAAF_PROJECT_NAME=daaf2 still owns "daaf2_daaf-data". +# A set-but-EMPTY DAAF_DATA_VOLUME_NAME falls through to the derived default (the +# `:-` test treats empty and unset alike), matching how load_daaf_settings adopts +# a whitelisted key only when it is non-empty. +# +# The claude-config volume is deliberately NOT resolved here: per-install auth +# isolation is by design, so it keeps its project-prefixed default everywhere and +# has no analogous override. +# +# Callers that need a value persisted in environment_settings.txt to win should +# `load_daaf_settings` first (which bridges DAAF_DATA_VOLUME_NAME into the shell +# env), then call this. Bash 3.2 safe: pure parameter expansion, no arrays. +resolve_data_volume_name() { + printf '%s\n' "${DAAF_DATA_VOLUME_NAME:-${DAAF_PROJECT_NAME:-daaf}_daaf-data}" +} + # --- Color Setup --- # Populate global color variables for terminal output. # Respects NO_COLOR (https://no-color.org/) and non-TTY contexts. diff --git a/scripts/host/environment_settings_example.txt b/scripts/host/environment_settings_example.txt index 32c4b359..283583ab 100644 --- a/scripts/host/environment_settings_example.txt +++ b/scripts/host/environment_settings_example.txt @@ -131,6 +131,17 @@ #DAAF_PORT_LOGVIEWER=3719 #DAAF_PORT_VSCODE=3720 # +# Data-volume override (advanced; shared workspace across installs). Normally the +# research workspace lives in a Docker volume named "_daaf-data" (the +# default install uses "daaf_daaf-data"). Set DAAF_DATA_VOLUME_NAME to a full, +# verbatim volume name to point THIS install at a specific data volume -- the +# escape hatch for sharing ONE research workspace across two installs. When set it +# must match the "name:" under the external "daaf-data" block you uncomment in +# docker-compose.yml. Leave unset for normal installs (the project-prefixed +# default derivation is used, byte-for-byte identical to the historical behavior). +# --- To activate, remove the leading '#': --- +#DAAF_DATA_VOLUME_NAME=daaf_daaf-data +# # The developer build flag DAAF_DEV also belongs to this install-time family, but # it is documented alongside the other developer options in section [6]. # diff --git a/scripts/host/install.ps1 b/scripts/host/install.ps1 index db8fe90a..1615f0fc 100644 --- a/scripts/host/install.ps1 +++ b/scripts/host/install.ps1 @@ -146,7 +146,10 @@ if ($LASTEXITCODE -ne 0) { # (DAAF_PROJECT_NAME set) are detected correctly. The value comes from the process # environment first, else DAAF_PROJECT_NAME in the existing install's # environment_settings.txt, else the "daaf" default (byte-for-byte identical to the -# former hardcoded "daaf_daaf-data"). Parse only that one key (never dot-source -- +# former hardcoded "daaf_daaf-data"). DAAF_DATA_VOLUME_NAME, when set in the +# process environment, overrides the whole derivation with a verbatim full volume +# name (the shared-workspace escape hatch; matches Resolve-DaafDataVolumeName in +# daaf_lib.ps1). Parse only that one key (never dot-source -- # the file holds API keys); process env wins; CR stripped; PS 5.1 safe. $InstallProjectName = $env:DAAF_PROJECT_NAME if ([string]::IsNullOrEmpty($InstallProjectName)) { @@ -170,6 +173,7 @@ if ([string]::IsNullOrEmpty($InstallProjectName)) { } if ([string]::IsNullOrEmpty($InstallProjectName)) { $InstallProjectName = "daaf" } $DataVolumeName = "${InstallProjectName}_daaf-data" +if ($env:DAAF_DATA_VOLUME_NAME) { $DataVolumeName = $env:DAAF_DATA_VOLUME_NAME } # --- Bridge the DAAF_DEV build flag into the environment --- # Unlike DAAF_PROJECT_NAME above (only needed locally to derive the volume name; @@ -614,7 +618,7 @@ else { $SeedKeys = @() try { Copy-Item -LiteralPath $SeedSrc -Destination $SeedDst - foreach ($k in @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH')) { + foreach ($k in @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH', 'DAAF_DATA_VOLUME_NAME')) { $v = [Environment]::GetEnvironmentVariable($k, "Process") if ([string]::IsNullOrEmpty($v)) { continue } if ($k -eq 'DAAF_BRANCH' -and -not $SeedBranchOk) { continue } diff --git a/scripts/host/install.sh b/scripts/host/install.sh index 6403fb23..2a93bf07 100755 --- a/scripts/host/install.sh +++ b/scripts/host/install.sh @@ -119,7 +119,10 @@ fi # (DAAF_PROJECT_NAME set) are detected correctly. The value comes from the shell # environment first, else DAAF_PROJECT_NAME in the existing install's # environment_settings.txt, else the "daaf" default (byte-for-byte identical to -# the former hardcoded "daaf_daaf-data"). Parse only that one key (never `source` +# the former hardcoded "daaf_daaf-data"). DAAF_DATA_VOLUME_NAME, when set in the +# shell environment, overrides the whole derivation with a verbatim full volume +# name (the shared-workspace escape hatch; matches resolve_data_volume_name in +# daaf_lib.sh). Parse only that one key (never `source` # -- the file holds API keys); shell env wins; CR stripped; Bash 3.2 safe. INSTALL_PROJECT_NAME="${DAAF_PROJECT_NAME:-}" if [ -z "${INSTALL_PROJECT_NAME}" ] && [ -f "${INSTALL_DIR}/environment_settings.txt" ]; then @@ -137,7 +140,7 @@ if [ -z "${INSTALL_PROJECT_NAME}" ] && [ -f "${INSTALL_DIR}/environment_settings esac done < "${INSTALL_DIR}/environment_settings.txt" fi -DATA_VOLUME_NAME="${INSTALL_PROJECT_NAME:-daaf}_daaf-data" +DATA_VOLUME_NAME="${DAAF_DATA_VOLUME_NAME:-${INSTALL_PROJECT_NAME:-daaf}_daaf-data}" # --- Bridge the DAAF_DEV build flag into the environment --- # Unlike DAAF_PROJECT_NAME above (only needed locally to derive the volume name; @@ -590,7 +593,7 @@ else SEED_OK=1 SEED_KEYS="" if cp "${SEED_SRC}" "${SEED_DST}"; then - for _k in DAAF_PROJECT_NAME DAAF_PORT_MARIMO DAAF_PORT_LOGVIEWER DAAF_PORT_VSCODE DAAF_DEV DAAF_BRANCH; do + for _k in DAAF_PROJECT_NAME DAAF_PORT_MARIMO DAAF_PORT_LOGVIEWER DAAF_PORT_VSCODE DAAF_DEV DAAF_BRANCH DAAF_DATA_VOLUME_NAME; do _v="${!_k:-}" [ -n "${_v}" ] || continue if [ "${_k}" = "DAAF_BRANCH" ] && [ "${SEED_BRANCH_OK}" != "1" ]; then diff --git a/scripts/host/migrate_daaf.ps1 b/scripts/host/migrate_daaf.ps1 index 7ea356b9..a4a67d50 100644 --- a/scripts/host/migrate_daaf.ps1 +++ b/scripts/host/migrate_daaf.ps1 @@ -57,6 +57,12 @@ function Wait-ForUser { $Repo = "DAAF-Contribution-Community/daaf" $Branch = if ($env:DAAF_BRANCH) { $env:DAAF_BRANCH } else { "main" } $RawBase = "https://raw.githubusercontent.com/$Repo/$Branch" +# Migration deliberately targets the DEFAULT data volume "daaf_daaf-data" and does +# NOT honor DAAF_PROJECT_NAME or the DAAF_DATA_VOLUME_NAME override. This tool is a +# one-time bootstrap for LEGACY installs that predate custom project names and the +# data-volume override -- those installs always used the single hardcoded default +# name, so deriving or overriding it here would only risk missing the volume the +# migration is meant to find. $VolumeName = "daaf_daaf-data" $ContainerName = "" $BackupCompleted = $false diff --git a/scripts/host/migrate_daaf.sh b/scripts/host/migrate_daaf.sh index c3ff46cb..31c580b5 100755 --- a/scripts/host/migrate_daaf.sh +++ b/scripts/host/migrate_daaf.sh @@ -54,6 +54,12 @@ fi REPO="DAAF-Contribution-Community/daaf" BRANCH="${DAAF_BRANCH:-main}" RAW_BASE="https://raw.githubusercontent.com/${REPO}/${BRANCH}" +# Migration deliberately targets the DEFAULT data volume "daaf_daaf-data" and does +# NOT honor DAAF_PROJECT_NAME or the DAAF_DATA_VOLUME_NAME override. This tool is a +# one-time bootstrap for LEGACY installs that predate custom project names and the +# data-volume override -- those installs always used the single hardcoded default +# name, so deriving or overriding it here would only risk missing the volume the +# migration is meant to find. VOLUME_NAME="daaf_daaf-data" CONTAINER_NAME="" BACKUP_COMPLETED=false diff --git a/scripts/host/rebuild_daaf.ps1 b/scripts/host/rebuild_daaf.ps1 index c0ce5c2a..90406e7f 100644 --- a/scripts/host/rebuild_daaf.ps1 +++ b/scripts/host/rebuild_daaf.ps1 @@ -49,7 +49,7 @@ function Wait-AndExit { function Import-DaafSettingsInline { param([string]$SettingsFile = "./environment_settings.txt") if (-not (Test-Path -LiteralPath $SettingsFile)) { return } - $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH') + $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH', 'DAAF_DATA_VOLUME_NAME') # -Encoding UTF8: PS 5.1's bare Get-Content misreads BOM-less UTF-8 as ANSI # (cp1252); the settings writer is BOM-less UTF-8, so reads are pinned to match. foreach ($rawLine in (Get-Content -LiteralPath $SettingsFile -Encoding UTF8)) { diff --git a/scripts/host/rebuild_daaf.sh b/scripts/host/rebuild_daaf.sh index efdb5bda..1c3263b6 100755 --- a/scripts/host/rebuild_daaf.sh +++ b/scripts/host/rebuild_daaf.sh @@ -51,7 +51,7 @@ _daaf_load_settings() { line="$(printf '%s' "${line}" | tr -d '\r')" case "${line}" in ''|'#'*) continue ;; esac case "${line}" in - DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*) + DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*|DAAF_DATA_VOLUME_NAME=*) key="${line%%=*}"; val="${line#*=}" case "${val}" in \"*\") val="${val#\"}"; val="${val%\"}" ;; diff --git a/scripts/host/restore_from_backup.ps1 b/scripts/host/restore_from_backup.ps1 index 303d503c..065be3b3 100644 --- a/scripts/host/restore_from_backup.ps1 +++ b/scripts/host/restore_from_backup.ps1 @@ -41,7 +41,7 @@ function Wait-AndExit { function Import-DaafSettingsInline { param([string]$SettingsFile = "./environment_settings.txt") if (-not (Test-Path -LiteralPath $SettingsFile)) { return } - $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH') + $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH', 'DAAF_DATA_VOLUME_NAME') # -Encoding UTF8: PS 5.1's bare Get-Content misreads BOM-less UTF-8 as ANSI # (cp1252); the settings writer is BOM-less UTF-8, so reads are pinned to match. foreach ($rawLine in (Get-Content -LiteralPath $SettingsFile -Encoding UTF8)) { @@ -71,9 +71,14 @@ Import-DaafSettingsInline # --- Configuration --- # The Docker named volume is project-prefixed: "_daaf-data". Default # unset => "daaf_daaf-data" (byte-for-byte identical to the previous hardcoded value). +# DAAF_DATA_VOLUME_NAME, when set, overrides the whole derivation with a verbatim +# full volume name (the shared-workspace escape hatch); unset => the derived +# default. Matches Resolve-DaafDataVolumeName in daaf_lib.ps1 (inlined here because +# this standalone script does not dot-source the library). $projectName = "daaf" if ($env:DAAF_PROJECT_NAME) { $projectName = $env:DAAF_PROJECT_NAME } $VolumeName = "${projectName}_daaf-data" +if ($env:DAAF_DATA_VOLUME_NAME) { $VolumeName = $env:DAAF_DATA_VOLUME_NAME } # Second volume: Claude Code state. Restored only when the selected backup # contains the dedicated subfolder (newer backups); older backups predating the # volume are restored data-only with a warning. diff --git a/scripts/host/restore_from_backup.sh b/scripts/host/restore_from_backup.sh index 54c16e2e..3ca56cc2 100755 --- a/scripts/host/restore_from_backup.sh +++ b/scripts/host/restore_from_backup.sh @@ -43,7 +43,7 @@ _daaf_load_settings() { line="$(printf '%s' "${line}" | tr -d '\r')" case "${line}" in ''|'#'*) continue ;; esac case "${line}" in - DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*) + DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*|DAAF_DATA_VOLUME_NAME=*) key="${line%%=*}"; val="${line#*=}" case "${val}" in \"*\") val="${val#\"}"; val="${val%\"}" ;; @@ -62,7 +62,11 @@ _daaf_load_settings # --- Configuration --- # Project-prefixed volume name "_daaf-data". Default unset => # "daaf_daaf-data" (byte-for-byte identical to the previous hardcoded value). -VOLUME_NAME="${DAAF_PROJECT_NAME:-daaf}_daaf-data" +# DAAF_DATA_VOLUME_NAME, when set, overrides the whole derivation with a verbatim +# full volume name (the shared-workspace escape hatch); unset => the derived +# default. Matches resolve_data_volume_name in daaf_lib.sh (inlined here because +# this standalone script does not source the library). +VOLUME_NAME="${DAAF_DATA_VOLUME_NAME:-${DAAF_PROJECT_NAME:-daaf}_daaf-data}" # Second volume: Claude Code state. Restored only when the selected backup # contains the dedicated subfolder (newer backups); older backups predating the # volume are restored data-only with a warning. diff --git a/scripts/host/run_daaf.ps1 b/scripts/host/run_daaf.ps1 index 3c61031f..94278cf2 100644 --- a/scripts/host/run_daaf.ps1 +++ b/scripts/host/run_daaf.ps1 @@ -34,7 +34,7 @@ function Wait-AndExit { function Import-DaafSettingsInline { param([string]$SettingsFile = "./environment_settings.txt") if (-not (Test-Path -LiteralPath $SettingsFile)) { return } - $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH') + $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH', 'DAAF_DATA_VOLUME_NAME') # -Encoding UTF8: PS 5.1's bare Get-Content misreads BOM-less UTF-8 as ANSI # (cp1252); the settings writer is BOM-less UTF-8, so reads are pinned to match. foreach ($rawLine in (Get-Content -LiteralPath $SettingsFile -Encoding UTF8)) { diff --git a/scripts/host/run_daaf.sh b/scripts/host/run_daaf.sh index 33c572a8..a9a88dbe 100755 --- a/scripts/host/run_daaf.sh +++ b/scripts/host/run_daaf.sh @@ -41,7 +41,7 @@ _daaf_load_settings() { line="$(printf '%s' "${line}" | tr -d '\r')" case "${line}" in ''|'#'*) continue ;; esac case "${line}" in - DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*) + DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*|DAAF_DATA_VOLUME_NAME=*) key="${line%%=*}"; val="${line#*=}" case "${val}" in \"*\") val="${val#\"}"; val="${val%\"}" ;; diff --git a/scripts/host/run_vscode.ps1 b/scripts/host/run_vscode.ps1 index 169d640e..2d08ffb7 100644 --- a/scripts/host/run_vscode.ps1 +++ b/scripts/host/run_vscode.ps1 @@ -39,7 +39,7 @@ function Wait-AndExit { function Import-DaafSettingsInline { param([string]$SettingsFile = "./environment_settings.txt") if (-not (Test-Path -LiteralPath $SettingsFile)) { return } - $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH') + $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH', 'DAAF_DATA_VOLUME_NAME') # -Encoding UTF8: PS 5.1's bare Get-Content misreads BOM-less UTF-8 as ANSI # (cp1252); the settings writer is BOM-less UTF-8, so reads are pinned to match. foreach ($rawLine in (Get-Content -LiteralPath $SettingsFile -Encoding UTF8)) { diff --git a/scripts/host/run_vscode.sh b/scripts/host/run_vscode.sh index dde4ebc4..860c67c5 100755 --- a/scripts/host/run_vscode.sh +++ b/scripts/host/run_vscode.sh @@ -45,7 +45,7 @@ else line="$(printf '%s' "${line}" | tr -d '\r')" case "${line}" in ''|'#'*) continue ;; esac case "${line}" in - DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*) + DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*|DAAF_DATA_VOLUME_NAME=*) key="${line%%=*}"; val="${line#*=}" case "${val}" in \"*\") val="${val#\"}"; val="${val%\"}" ;; diff --git a/scripts/host/test_migration.ps1 b/scripts/host/test_migration.ps1 index c945e7a3..da907a82 100644 --- a/scripts/host/test_migration.ps1 +++ b/scripts/host/test_migration.ps1 @@ -2682,6 +2682,36 @@ if (-not $SkipMultiInstance) { Test-Check "Default instance container still present ($ContainerMain)" (Test-DockerContainer $ContainerMain) Test-Check "Default instance data volume still present ($VolumeName)" (Test-DockerVolume $VolumeName) + # --- 8c-override. DAAF_DATA_VOLUME_NAME resolver sub-scenario --- + # Twin of the .sh 8c-override block: the checks above prove DAAF_PROJECT_NAME + # reprojects the DERIVED data-volume name; this proves the ORTHOGONAL escape + # hatch on the same machinery -- an explicit DAAF_DATA_VOLUME_NAME is honored + # VERBATIM (no project prefix) by Resolve-DaafDataVolumeName in daaf_lib.ps1, the + # single resolver every host tool (backup/restore, compose interpolation) calls + # to name the data volume, which is what lets two installs share ONE workspace + # volume. It is docker-free (pure $env reads + string ops), so we assert it + # directly in the same Test-Check style as the volume checks above. daaf_lib.ps1 + # is dot-sourced for the call and the override is set/restored via the harness's + # save-try-finally idiom (as in 8b), so the harness env is left exactly as it was + # -- the paired "parent env unchanged" check evidences that non-leak. + $DvnOverride = "daaf_shared_workspace_daaf-data" + $DvnLibPath = Join-Path $LocalRepoRoot "scripts\host\daaf_lib.ps1" + $DvnParentBefore = $env:DAAF_DATA_VOLUME_NAME + $DvnResolved = "" + try { + $env:DAAF_DATA_VOLUME_NAME = $DvnOverride + . $DvnLibPath + $DvnResolved = Resolve-DaafDataVolumeName + } catch { + $DvnResolved = "" + } finally { + if ($null -ne $DvnParentBefore) { $env:DAAF_DATA_VOLUME_NAME = $DvnParentBefore } else { Remove-Item Env:\DAAF_DATA_VOLUME_NAME -ErrorAction SilentlyContinue } + } + Test-Check "Host resolver honors DAAF_DATA_VOLUME_NAME verbatim ($DvnOverride)" ($DvnResolved -eq $DvnOverride) + # The override was restored in the finally above, so the harness's own + # DAAF_DATA_VOLUME_NAME is unchanged -- proving the set did not leak forward. + Test-Check "DAAF_DATA_VOLUME_NAME override stayed scoped to the sub-scenario (parent env unchanged)" ($env:DAAF_DATA_VOLUME_NAME -eq $DvnParentBefore) + Write-Host "" # --- 8d. Tear the second instance down completely --- diff --git a/scripts/host/test_migration.sh b/scripts/host/test_migration.sh index dcb40b3c..fb02588f 100755 --- a/scripts/host/test_migration.sh +++ b/scripts/host/test_migration.sh @@ -2356,6 +2356,39 @@ if [ "${SKIP_MULTI_INSTANCE:-}" != "1" ]; then check "Default instance data volume still present (${VOLUME_NAME})" "1" fi + # --- 8c-override. DAAF_DATA_VOLUME_NAME resolver sub-scenario --- + # The checks above prove DAAF_PROJECT_NAME reprojects the DERIVED data-volume + # name for a second instance. This sub-scenario exercises the ORTHOGONAL escape + # hatch on the same machinery: an explicit DAAF_DATA_VOLUME_NAME must be honored + # VERBATIM (no project prefix) by resolve_data_volume_name in daaf_lib.sh -- the + # single resolver every host tool (backup/restore, compose interpolation) calls + # to name the data volume, which is what lets two installs point at ONE shared + # workspace volume. It is docker-free (the resolver is pure parameter expansion), + # so we assert it directly in the same check() style as the volume checks above + # rather than standing up a third instance. The override is exported ONLY inside + # the command-substitution subshell (mirroring the 8b bring-up's scoped export), + # so the harness env is never mutated and later phases are unaffected -- the + # paired "parent env unchanged" check below evidences that non-leak. + DVN_OVERRIDE="daaf_shared_workspace_daaf-data" + DVN_PARENT_BEFORE="${DAAF_DATA_VOLUME_NAME:-}" + DVN_RESOLVED="$( + export DAAF_DATA_VOLUME_NAME="${DVN_OVERRIDE}" + source "${LOCAL_REPO_ROOT}/scripts/host/daaf_lib.sh" + resolve_data_volume_name + )" || DVN_RESOLVED="" + if [ "${DVN_RESOLVED}" = "${DVN_OVERRIDE}" ]; then + check "Host resolver honors DAAF_DATA_VOLUME_NAME verbatim (${DVN_OVERRIDE})" "0" + else + check "Host resolver honors DAAF_DATA_VOLUME_NAME verbatim (got: '${DVN_RESOLVED}')" "1" + fi + # The override was scoped to the subshell above, so the harness's own + # DAAF_DATA_VOLUME_NAME is unchanged -- proving the export did not leak forward. + if [ "${DAAF_DATA_VOLUME_NAME:-}" = "${DVN_PARENT_BEFORE}" ]; then + check "DAAF_DATA_VOLUME_NAME override stayed scoped to the sub-scenario (parent env unchanged)" "0" + else + check "DAAF_DATA_VOLUME_NAME override stayed scoped to the sub-scenario (leaked: '${DAAF_DATA_VOLUME_NAME:-}')" "1" + fi + echo "" # --- 8d. Tear the second instance down completely --- diff --git a/scripts/host/update_daaf.ps1 b/scripts/host/update_daaf.ps1 index 30387dd2..5074578b 100644 --- a/scripts/host/update_daaf.ps1 +++ b/scripts/host/update_daaf.ps1 @@ -139,7 +139,7 @@ $PersistBranch = "" function Import-DaafSettingsInline { param([string]$SettingsFile = "./environment_settings.txt") if (-not (Test-Path -LiteralPath $SettingsFile)) { return } - $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH') + $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH', 'DAAF_DATA_VOLUME_NAME') # -Encoding UTF8: PS 5.1's bare Get-Content misreads BOM-less UTF-8 as ANSI # (cp1252); the settings writer is BOM-less UTF-8, so reads are pinned to match. foreach ($rawLine in (Get-Content -LiteralPath $SettingsFile -Encoding UTF8)) { diff --git a/scripts/host/update_daaf.sh b/scripts/host/update_daaf.sh index 13531779..5aaff28e 100755 --- a/scripts/host/update_daaf.sh +++ b/scripts/host/update_daaf.sh @@ -76,7 +76,7 @@ _daaf_load_settings() { line="$(printf '%s' "${line}" | tr -d '\r')" case "${line}" in ''|'#'*) continue ;; esac case "${line}" in - DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*) + DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*|DAAF_DATA_VOLUME_NAME=*) key="${line%%=*}"; val="${line#*=}" case "${val}" in \"*\") val="${val#\"}"; val="${val%\"}" ;; diff --git a/scripts/host/view_logs.ps1 b/scripts/host/view_logs.ps1 index 7a99dd42..01b79aa1 100644 --- a/scripts/host/view_logs.ps1 +++ b/scripts/host/view_logs.ps1 @@ -40,7 +40,7 @@ function Wait-AndExit { function Import-DaafSettingsInline { param([string]$SettingsFile = "./environment_settings.txt") if (-not (Test-Path -LiteralPath $SettingsFile)) { return } - $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH') + $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH', 'DAAF_DATA_VOLUME_NAME') # -Encoding UTF8: PS 5.1's bare Get-Content misreads BOM-less UTF-8 as ANSI # (cp1252); the settings writer is BOM-less UTF-8, so reads are pinned to match. foreach ($rawLine in (Get-Content -LiteralPath $SettingsFile -Encoding UTF8)) { diff --git a/scripts/host/view_logs.sh b/scripts/host/view_logs.sh index 9625551b..06d46680 100755 --- a/scripts/host/view_logs.sh +++ b/scripts/host/view_logs.sh @@ -47,7 +47,7 @@ else line="$(printf '%s' "${line}" | tr -d '\r')" case "${line}" in ''|'#'*) continue ;; esac case "${line}" in - DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*) + DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*|DAAF_DATA_VOLUME_NAME=*) key="${line%%=*}"; val="${line#*=}" case "${val}" in \"*\") val="${val#\"}"; val="${val%\"}" ;; diff --git a/scripts/host/view_notebooks.ps1 b/scripts/host/view_notebooks.ps1 index 9493e8a4..c011cdc2 100644 --- a/scripts/host/view_notebooks.ps1 +++ b/scripts/host/view_notebooks.ps1 @@ -39,7 +39,7 @@ function Wait-AndExit { function Import-DaafSettingsInline { param([string]$SettingsFile = "./environment_settings.txt") if (-not (Test-Path -LiteralPath $SettingsFile)) { return } - $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH') + $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH', 'DAAF_DATA_VOLUME_NAME') # -Encoding UTF8: PS 5.1's bare Get-Content misreads BOM-less UTF-8 as ANSI # (cp1252); the settings writer is BOM-less UTF-8, so reads are pinned to match. foreach ($rawLine in (Get-Content -LiteralPath $SettingsFile -Encoding UTF8)) { diff --git a/scripts/host/view_notebooks.sh b/scripts/host/view_notebooks.sh index 0fbfadba..3047553f 100755 --- a/scripts/host/view_notebooks.sh +++ b/scripts/host/view_notebooks.sh @@ -45,7 +45,7 @@ else line="$(printf '%s' "${line}" | tr -d '\r')" case "${line}" in ''|'#'*) continue ;; esac case "${line}" in - DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*) + DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*|DAAF_DATA_VOLUME_NAME=*) key="${line%%=*}"; val="${line#*=}" case "${val}" in \"*\") val="${val#\"}"; val="${val%\"}" ;; diff --git a/scripts/host/view_quarto.ps1 b/scripts/host/view_quarto.ps1 index 84fab3e2..17f3ae18 100644 --- a/scripts/host/view_quarto.ps1 +++ b/scripts/host/view_quarto.ps1 @@ -72,7 +72,7 @@ function Read-QuartoSelection { function Import-DaafSettingsInline { param([string]$SettingsFile = "./environment_settings.txt") if (-not (Test-Path -LiteralPath $SettingsFile)) { return } - $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH') + $known = @('DAAF_PROJECT_NAME', 'DAAF_PORT_MARIMO', 'DAAF_PORT_LOGVIEWER', 'DAAF_PORT_VSCODE', 'DAAF_DEV', 'DAAF_BRANCH', 'DAAF_DATA_VOLUME_NAME') # -Encoding UTF8: PS 5.1's bare Get-Content misreads BOM-less UTF-8 as ANSI # (cp1252); the settings writer is BOM-less UTF-8, so reads are pinned to match. foreach ($rawLine in (Get-Content -LiteralPath $SettingsFile -Encoding UTF8)) { diff --git a/scripts/host/view_quarto.sh b/scripts/host/view_quarto.sh index b651460b..abb56069 100755 --- a/scripts/host/view_quarto.sh +++ b/scripts/host/view_quarto.sh @@ -59,7 +59,7 @@ else line="$(printf '%s' "${line}" | tr -d '\r')" case "${line}" in ''|'#'*) continue ;; esac case "${line}" in - DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*) + DAAF_PROJECT_NAME=*|DAAF_PORT_MARIMO=*|DAAF_PORT_LOGVIEWER=*|DAAF_PORT_VSCODE=*|DAAF_DEV=*|DAAF_BRANCH=*|DAAF_DATA_VOLUME_NAME=*) key="${line%%=*}"; val="${line#*=}" case "${val}" in \"*\") val="${val#\"}"; val="${val%\"}" ;; diff --git a/tests/bash/daaf_lib.bats b/tests/bash/daaf_lib.bats index 0aa7507f..9cc6fd05 100755 --- a/tests/bash/daaf_lib.bats +++ b/tests/bash/daaf_lib.bats @@ -488,6 +488,63 @@ teardown() { assert_output --partial "NAME=UNSET" } +# ========================================================================= +# Data-volume name resolver (resolve_data_volume_name) +# ========================================================================= +# Precedence: DAAF_DATA_VOLUME_NAME (verbatim) > "_daaf-data" default. +# An unset override reproduces the historical hardcoded derivation byte-for-byte. + +@test "resolve_data_volume_name derives the default when no override and no project name" { + run bash -c ' + unset _DAAF_LIB_LOADED + source "'"${REPO_ROOT}"'/scripts/host/daaf_lib.sh" + unset DAAF_DATA_VOLUME_NAME DAAF_PROJECT_NAME + resolve_data_volume_name + ' + assert_success + # Unset override + unset project name => the legacy hardcoded name, unchanged. + assert_output "daaf_daaf-data" +} + +@test "resolve_data_volume_name applies the project prefix when only DAAF_PROJECT_NAME is set" { + run bash -c ' + unset _DAAF_LIB_LOADED + source "'"${REPO_ROOT}"'/scripts/host/daaf_lib.sh" + unset DAAF_DATA_VOLUME_NAME + export DAAF_PROJECT_NAME=daaf2 + resolve_data_volume_name + ' + assert_success + # A second instance owns "_daaf-data" without touching the override. + assert_output "daaf2_daaf-data" +} + +@test "resolve_data_volume_name uses DAAF_DATA_VOLUME_NAME verbatim when set (ignores project prefix)" { + run bash -c ' + unset _DAAF_LIB_LOADED + source "'"${REPO_ROOT}"'/scripts/host/daaf_lib.sh" + export DAAF_DATA_VOLUME_NAME=shared_workspace_vol + export DAAF_PROJECT_NAME=daaf2 + resolve_data_volume_name + ' + assert_success + # Set override wins verbatim -- no project prefix is added (shared-workspace hatch). + assert_output "shared_workspace_vol" +} + +@test "resolve_data_volume_name treats a set-but-empty override as unset (derived default)" { + run bash -c ' + unset _DAAF_LIB_LOADED + source "'"${REPO_ROOT}"'/scripts/host/daaf_lib.sh" + export DAAF_DATA_VOLUME_NAME= + unset DAAF_PROJECT_NAME + resolve_data_volume_name + ' + assert_success + # The `:-` default treats empty and unset alike, so it falls through to derived. + assert_output "daaf_daaf-data" +} + # ========================================================================= # docker-compose.yml multi-instance interpolation defaults # ========================================================================= diff --git a/tests/bash/environment_settings_example.bats b/tests/bash/environment_settings_example.bats index 14559299..0fabf89e 100755 --- a/tests/bash/environment_settings_example.bats +++ b/tests/bash/environment_settings_example.bats @@ -28,7 +28,7 @@ # (hash-SPACE-key) that would activate to a valid assignment # (e) CONTENTS entries correspond 1:1 (number + title, in order) with the # "# ====" numbered section banners -# (f) the 6 host-bridged keys each appear at least once as an example key line +# (f) the 7 host-bridged keys each appear at least once as an example key line # # This test reads the real file (read-only); it never writes to it. # ============================================================================ @@ -174,12 +174,12 @@ teardown() { } # ========================================================================= -# (f) The six host-bridged keys each appear as an example key line +# (f) The seven host-bridged keys each appear as an example key line # ========================================================================= @test "(f) each host-bridged key appears as a #KEY= example line" { for key in DAAF_PROJECT_NAME DAAF_PORT_MARIMO DAAF_PORT_LOGVIEWER \ - DAAF_PORT_VSCODE DAAF_DEV DAAF_BRANCH; do + DAAF_PORT_VSCODE DAAF_DEV DAAF_BRANCH DAAF_DATA_VOLUME_NAME; do run grep -qE "^#${key}=" "${EXAMPLE_FILE}" if [ "${status}" -ne 0 ]; then echo "missing host-bridged example key: #${key}=" diff --git a/tests/bash/install.bats b/tests/bash/install.bats index 0d6755f1..d20c81ef 100755 --- a/tests/bash/install.bats +++ b/tests/bash/install.bats @@ -686,6 +686,22 @@ teardown() { assert_output --partial "DAAF_PORT_MARIMO=9990" } +@test "install.sh: seeds DAAF_DATA_VOLUME_NAME from the process environment when set" { + # The data-volume override rides the same seeding array as the multi-instance + # keys, so a value exported for the install persists into the settings file. + export DAAF_NESTED=1 + export DAAF_PROJECT_NAME="myproj" + export DAAF_DATA_VOLUME_NAME="shared_workspace_vol" + MOCK_DOCKER_EXEC_EXIT=0 + cd "${TEST_DIR}" + run bash "${REPO_ROOT}/scripts/host/install.sh" + assert_success + assert_output --partial "seeded these values" + [ -f "${TEST_DIR}/daaf-docker/environment_settings.txt" ] + run cat "${TEST_DIR}/daaf-docker/environment_settings.txt" + assert_output --partial "DAAF_DATA_VOLUME_NAME=shared_workspace_vol" +} + @test "install.sh: preserves an existing environment_settings.txt (never overwrites)" { export DAAF_NESTED=1 export DAAF_PROJECT_NAME="ignored" diff --git a/tests/bash/settings_whitelist_parity.bats b/tests/bash/settings_whitelist_parity.bats index 21983dbd..c7ac49e5 100755 --- a/tests/bash/settings_whitelist_parity.bats +++ b/tests/bash/settings_whitelist_parity.bats @@ -11,8 +11,8 @@ # + 2 optional dev-workspace pairs, enumerated below). Every copy hardcodes the # SAME whitelist of DAAF_* keys: # -# DAAF_BRANCH DAAF_DEV DAAF_PORT_LOGVIEWER DAAF_PORT_MARIMO -# DAAF_PORT_VSCODE DAAF_PROJECT_NAME +# DAAF_BRANCH DAAF_DATA_VOLUME_NAME DAAF_DEV DAAF_PORT_LOGVIEWER +# DAAF_PORT_MARIMO DAAF_PORT_VSCODE DAAF_PROJECT_NAME # # If one copy drifts (a key added to the library but not propagated to a # launcher, or vice versa), a second install's project name / published ports / @@ -76,6 +76,7 @@ OPTIONAL_COPIES=( canonical_expected() { LC_ALL=C sort -u <<'KEYS' DAAF_BRANCH +DAAF_DATA_VOLUME_NAME DAAF_DEV DAAF_PORT_LOGVIEWER DAAF_PORT_MARIMO @@ -122,29 +123,29 @@ INSTALL_SH="${REPO_ROOT}/scripts/host/install.sh" INSTALL_PS1="${REPO_ROOT}/scripts/host/install.ps1" # =========================================================================== -# (d) Canonical sets are EXACTLY the pinned literal 6 keys (both languages). +# (d) Canonical sets are EXACTLY the pinned literal 7 keys (both languages). # Placed first: if the extraction returns empty or wrong, every set-equality # test below would pass vacuously, so this pin is the anti-vacuity anchor. # =========================================================================== -@test "parity: daaf_lib.sh canonical whitelist is exactly the pinned 6 keys" { +@test "parity: daaf_lib.sh canonical whitelist is exactly the pinned 7 keys" { local got want got="$(extract_sh_whitelist "${DAAF_LIB_SH}")" want="$(canonical_expected)" [ "${got}" = "${want}" ] || { - echo "CANONICAL DRIFT: daaf_lib.sh whitelist != pinned 6 keys" + echo "CANONICAL DRIFT: daaf_lib.sh whitelist != pinned 7 keys" echo "-- extracted:"; echo "${got}" echo "-- pinned:"; echo "${want}" return 1 } } -@test "parity: daaf_lib.ps1 canonical whitelist is exactly the pinned 6 keys" { +@test "parity: daaf_lib.ps1 canonical whitelist is exactly the pinned 7 keys" { local got want got="$(extract_ps1_whitelist "${DAAF_LIB_PS1}")" want="$(canonical_expected)" [ "${got}" = "${want}" ] || { - echo "CANONICAL DRIFT: daaf_lib.ps1 \$known != pinned 6 keys" + echo "CANONICAL DRIFT: daaf_lib.ps1 \$known != pinned 7 keys" echo "-- extracted:"; echo "${got}" echo "-- pinned:"; echo "${want}" return 1 @@ -240,7 +241,7 @@ INSTALL_PS1="${REPO_ROOT}/scripts/host/install.ps1" # parsing arms are therefore bespoke, NOT the shared pattern. This test pins that # deviation to exactly {DAAF_DEV, DAAF_PROJECT_NAME} so that a well-meaning # "let's make install consistent too" edit that silently converts them to the -# full 6-key whitelist TRIPS here and forces a deliberate decision + a matching +# full 7-key whitelist TRIPS here and forces a deliberate decision + a matching # update to this allowlist. @test "deviation: install.sh parses exactly {DAAF_DEV, DAAF_PROJECT_NAME}" { @@ -278,8 +279,8 @@ INSTALL_PS1="${REPO_ROOT}/scripts/host/install.ps1" @test "parity SELF-TEST: perturbing a .sh copy's whitelist trips the comparison" { cp "${REPO_ROOT}/scripts/host/backup_daaf.sh" "${TEST_DIR}/perturbed.sh" - # Drop DAAF_BRANCH from the case-arm (a one-key regression). - sed -i 's/|DAAF_DEV=\*|DAAF_BRANCH=\*)/|DAAF_DEV=*)/' "${TEST_DIR}/perturbed.sh" + # Drop the terminal DAAF_DATA_VOLUME_NAME from the case-arm (a one-key regression). + sed -i 's/|DAAF_BRANCH=\*|DAAF_DATA_VOLUME_NAME=\*)/|DAAF_BRANCH=*)/' "${TEST_DIR}/perturbed.sh" local canon perturbed canon="$(extract_sh_whitelist "${DAAF_LIB_SH}")" @@ -291,17 +292,17 @@ INSTALL_PS1="${REPO_ROOT}/scripts/host/install.ps1" echo "${perturbed}" return 1 } - # And specifically DAAF_BRANCH must be the key that disappeared. - if echo "${perturbed}" | grep -qx 'DAAF_BRANCH'; then - echo "SELF-TEST: perturbation failed to remove DAAF_BRANCH (sed anchor stale?)" + # And specifically DAAF_DATA_VOLUME_NAME must be the key that disappeared. + if echo "${perturbed}" | grep -qx 'DAAF_DATA_VOLUME_NAME'; then + echo "SELF-TEST: perturbation failed to remove DAAF_DATA_VOLUME_NAME (sed anchor stale?)" return 1 fi } @test "parity SELF-TEST: perturbing a .ps1 copy's whitelist trips the comparison" { cp "${REPO_ROOT}/scripts/host/backup_daaf.ps1" "${TEST_DIR}/perturbed.ps1" - # Drop DAAF_BRANCH from the $known array. - sed -i "s/, 'DAAF_DEV', 'DAAF_BRANCH')/, 'DAAF_DEV')/" "${TEST_DIR}/perturbed.ps1" + # Drop the terminal DAAF_DATA_VOLUME_NAME from the $known array. + sed -i "s/, 'DAAF_BRANCH', 'DAAF_DATA_VOLUME_NAME')/, 'DAAF_BRANCH')/" "${TEST_DIR}/perturbed.ps1" local canon perturbed canon="$(extract_ps1_whitelist "${DAAF_LIB_PS1}")" @@ -312,8 +313,8 @@ INSTALL_PS1="${REPO_ROOT}/scripts/host/install.ps1" echo "${perturbed}" return 1 } - if echo "${perturbed}" | grep -qx 'DAAF_BRANCH'; then - echo "SELF-TEST: perturbation failed to remove DAAF_BRANCH (sed anchor stale?)" + if echo "${perturbed}" | grep -qx 'DAAF_DATA_VOLUME_NAME'; then + echo "SELF-TEST: perturbation failed to remove DAAF_DATA_VOLUME_NAME (sed anchor stale?)" return 1 fi } diff --git a/tests/powershell/daaf_lib.Tests.ps1 b/tests/powershell/daaf_lib.Tests.ps1 new file mode 100644 index 00000000..538e5bba --- /dev/null +++ b/tests/powershell/daaf_lib.Tests.ps1 @@ -0,0 +1,65 @@ +# ============================================================================ +# Unit tests for Resolve-DaafDataVolumeName (daaf_lib.ps1) +# ============================================================================ +# Resolve-DaafDataVolumeName is the PowerShell twin of daaf_lib.sh +# resolve_data_volume_name. It returns the Docker volume name that holds the DAAF +# research workspace (/daaf), honoring the optional DAAF_DATA_VOLUME_NAME full-name +# override. This battery mirrors the four bash cases in tests/bash/daaf_lib.bats +# (the "Data-volume name resolver" section), asserting the same precedence: +# 1. no override, no project name => "daaf_daaf-data" (legacy hardcoded) +# 2. only DAAF_PROJECT_NAME=daaf2 => "daaf2_daaf-data" (Compose prefix) +# 3. DAAF_DATA_VOLUME_NAME set => used verbatim (project prefix ignored) +# 4. DAAF_DATA_VOLUME_NAME set-but-empty => derived default (empty == unset) +# Precedence: DAAF_DATA_VOLUME_NAME (verbatim) > "_daaf-data" default. +# An unset override reproduces the historical hardcoded derivation byte-for-byte. +# ============================================================================ + +Describe "daaf_lib.ps1 Resolve-DaafDataVolumeName" { + BeforeAll { + . "$PSScriptRoot/TestHelper.ps1" + # Dot-source the library so the function is defined in this scope. + . "$RepoRoot/scripts/host/daaf_lib.ps1" + } + + BeforeEach { + # Snapshot the two env vars the resolver reads, then start each test from a + # clean slate so the ambient process env cannot skew the assertions. AfterEach + # restores the caller's exact prior state (clearing a var only when it was + # genuinely unset), so the suite leaves no environment residue. + $script:SavedVolName = $env:DAAF_DATA_VOLUME_NAME + $script:SavedProject = $env:DAAF_PROJECT_NAME + Remove-Item Env:DAAF_DATA_VOLUME_NAME -ErrorAction SilentlyContinue + Remove-Item Env:DAAF_PROJECT_NAME -ErrorAction SilentlyContinue + } + + AfterEach { + if ($null -ne $script:SavedVolName) { $env:DAAF_DATA_VOLUME_NAME = $script:SavedVolName } else { Remove-Item Env:DAAF_DATA_VOLUME_NAME -ErrorAction SilentlyContinue } + if ($null -ne $script:SavedProject) { $env:DAAF_PROJECT_NAME = $script:SavedProject } else { Remove-Item Env:DAAF_PROJECT_NAME -ErrorAction SilentlyContinue } + } + + It "derives the default when no override and no project name" { + # Unset override + unset project name => the legacy hardcoded name, unchanged. + Resolve-DaafDataVolumeName | Should -Be "daaf_daaf-data" + } + + It "applies the project prefix when only DAAF_PROJECT_NAME is set" { + # A second instance owns "_daaf-data" without touching the override. + $env:DAAF_PROJECT_NAME = "daaf2" + Resolve-DaafDataVolumeName | Should -Be "daaf2_daaf-data" + } + + It "uses DAAF_DATA_VOLUME_NAME verbatim when set (ignores project prefix)" { + # Set override wins verbatim -- no project prefix is added (shared-workspace hatch). + $env:DAAF_DATA_VOLUME_NAME = "shared_workspace_vol" + $env:DAAF_PROJECT_NAME = "daaf2" + Resolve-DaafDataVolumeName | Should -Be "shared_workspace_vol" + } + + It "treats a set-but-empty override as unset (derived default)" { + # An empty string is falsy in PowerShell, so the guard treats empty and unset + # alike, matching the bash `:-` semantics -- it falls through to the derived + # default rather than returning an empty volume name. + $env:DAAF_DATA_VOLUME_NAME = "" + Resolve-DaafDataVolumeName | Should -Be "daaf_daaf-data" + } +} diff --git a/user_reference/01_installation_and_quickstart.md b/user_reference/01_installation_and_quickstart.md index fed6f5bb..8ab8be7e 100644 --- a/user_reference/01_installation_and_quickstart.md +++ b/user_reference/01_installation_and_quickstart.md @@ -371,9 +371,64 @@ You can also use the Docker Desktop GUI to explore the DAAF docker volume by nav - **Getting a large research folder out wholesale:** rather than zipping folder by folder, take a full **backup** — from the DAAF Control Panel choose **option 7, "Create Backup"** — which copies *everything* (all your research data) out to your host machine in one step. See [**Backing Up Your Work**](#backing-up-your-work) below. - **Bringing files in:** drag and drop them from your computer straight onto the editor's file explorer sidebar — this works for single files and whole folders (see *Getting files INTO the container* above). +### Linking Host Folders into the Container (Bind Mounts) + +The drag-and-drop upload described above is the right tool for most files. But if you have a **large local dataset** — tens of gigabytes of parquet, a folder of survey extracts, a directory you refresh regularly from elsewhere — copying it through the browser editor is slow and duplicates the data into the Docker volume. For those cases you can **bind-mount** a folder on your computer straight into the container, so DAAF reads your files in place. Reach for this when the data is bulky or lives naturally on your host; stick with the [drag-and-drop upload](#viewing-and-editing-files) for the occasional single file or small dataset. + +DAAF ships this capability **off by default** as a commented-out block in `docker-compose.yml` that you opt into. The recommended pattern is a **read-only** mount of one host folder onto `/host_data` inside the container. + +**Set it up (edit in the container, then rebuild):** + +1. **Open the container's copy of `docker-compose.yml`.** Edit it *inside* the container — either ask Claude to edit `/daaf/docker-compose.yml`, or open it in the [browser-based code editor](#viewing-and-editing-files). This matters: `rebuild_daaf.sh` copies the container's compose file *out* to your host and rebuilds from it, so a host-side edit is silently overwritten on the next rebuild. Editing the container copy is the only change that survives. +2. **Uncomment the bind-mount block** in the `daaf-docker` service's `volumes:` section and set `source:` to the folder on your computer you want to expose. Leave `read_only: true` in place: + ```yaml + # In the daaf-docker service, under volumes: + - type: bind + source: /Users/yourname/datasets # <-- your host folder (Windows: C:\Users\yourname\datasets) + target: /host_data + read_only: true + ``` +3. **Rebuild so the host picks up the change:** + + **macOS / Linux (Terminal):** + ```bash + cd daaf-docker + bash rebuild_daaf.sh + ``` + + **Windows (PowerShell):** + ```powershell + cd daaf-docker + .\rebuild_daaf.ps1 + ``` + +Your files now appear at `/host_data` inside the container, and DAAF can read them directly. + +**Why the long (`type: bind`) syntax?** DAAF uses the verbose form on purpose: if you mistype `source:`, Docker **fails loudly** with a clear error. The short one-line form (`./typo:/host_data`) instead **silently creates an empty directory** at the bad path — so a typo would leave DAAF staring at an empty `/host_data` with no hint why. + +**Permissions: will your files be readable?** The container runs as a non-root user, `appuser` (UID/GID **1000**), and — unlike the two managed Docker volumes — a bind mount is **not** ownership-corrected by DAAF's `daaf-init` step. Whether `appuser` can read your files therefore depends on your platform: + +| Your platform | Read-only mount | What's happening | +|---------------|-----------------|------------------| +| **macOS (Docker Desktop)** | Generally just works | Docker Desktop translates/fakes ownership, so your files appear owned by the container user regardless of your Mac UID (behavior is version-dependent; rare file-sharing bugs exist) | +| **Windows — `/mnt/c/...` (WSL2)** or a path shared in Docker Desktop | Just works | Windows permissions are translated through the Docker Desktop file-sharing layer | +| **Native Linux** | Works **if** UID 1000 can read the files | Real UID/GID pass-through — no translation layer. Only the numbers matter, not usernames | +| **Windows — WSL filesystem path** (`/home/` inside a distro) | Works **if** UID 1000 can read the files | Native Linux rules apply inside the distro — the same case as native Linux | + +On the two "if" rows, diagnose with `ls -ln` (the `-n` shows **numeric** owner IDs): if your files aren't owned by `1000` and aren't world-readable, `appuser` can't read them. The simplest fix for a read-only data folder is to make it group- or world-readable on the host. + +**Advanced: write-enabled mounts.** Keep `read_only: true` unless you have a specific reason to let the container write back to your host folder — a read-only mount is safer and keeps DAAF from ever modifying your source data. If you *do* need writes (drop `read_only: true`), permissions get stricter: + +- **Native Linux / WSL-filesystem paths:** the container writes files as UID 1000, so align ownership — either `chown` the host folder to UID 1000, or add yourself to a shared group and make the folder group-writable. **Don't reach for `chmod 777`** — it works, but it exposes the folder to every user on the machine; group alignment is the right tool. +- **macOS and Windows `/mnt/c` paths:** writes "just work" through the Docker Desktop translation layer, with files landing under your host user on the other side. + +> **Reproducibility trade-off — a deliberate decision to make.** Bind-mounted files live **outside** DAAF's archive and hash boundary: they are not captured by `backup_daaf.sh` and not part of the project's audit trail. That means reproducing the analysis later depends on *you* guaranteeing the same files are present at the same mount point — DAAF can't freeze them for you. The recommended mitigation is to have your **fetch scripts copy** the inputs you actually use from `/host_data` into the project's own `data/raw/` directory (with the normal raw-data naming conventions). The bind mount then serves as a fast on-ramp, while the archived working copy inside the project keeps the audit trail complete. + +> **Boundary.** `/host_data` is *not* covered by `backup_daaf.sh`, so never treat it as a place to store DAAF outputs — write results into your project as usual. And never point a bind mount at the container's own system paths (`/daaf`, `/daaf/.claude`, and the like): mounting over DAAF's own files would shadow them and break the install. + ### Backing Up Your Work -Since your research files live inside the Docker volume, it'll be extremely important to regularly back up your work separately from the Docker volume. The easiest way is from the **DAAF Control Panel** — run `bash daaf.sh` (`.\daaf.ps1` on Windows) in your `daaf-docker` folder and choose **option 7, "Create Backup."** If you'd rather call the backup script directly, that works too: +Since your research files live inside the Docker volume, it'll be extremely important to regularly back up your work separately from the Docker volume. The easiest way is from the **DAAF Control Panel** — run `bash daaf.sh` (`.\daaf.ps1` on Windows) in your `daaf-docker` folder and choose **option 7, "Create Backup."** (If you've set up a [shared research workspace](#sharing-one-research-workspace-across-two-installs-advanced) across two installs, note the caveat in that section: run backups from **one** install only.) If you'd rather call the backup script directly, that works too: **macOS / Linux (Terminal):** @@ -736,7 +791,7 @@ If the installer detects a previous attempt that didn't complete successfully (e ### Running multiple DAAF instances -Most people run a single DAAF installation and never need this section. But if you want **two (or more) independent DAAF installs on the same machine** — for example, one folder for work and another for personal projects, each with its own Docker volume and research history — you can, with a small amount of configuration. +Most people run a single DAAF installation and never need this section. But if you want **two (or more) independent DAAF installs on the same machine** — for example, one folder for work and another for personal projects, each with its own Docker volume and research history — you can, with a small amount of configuration. (If instead you want two installs to **share** one research workspace — say, to work the same projects from both a Claude and a ChatGPT install — see [Sharing One Research Workspace Across Two Installs](#sharing-one-research-workspace-across-two-installs-advanced) below; that pattern is the deliberate exception to the per-instance volumes described here.) Two things must be unique per install so they don't collide: @@ -781,6 +836,72 @@ The DAAF launcher and control-panel scripts (`run_daaf`, `daaf`, the `view_*` br > **Changing these on an existing install** requires the same `docker compose down` + relaunch: the project name and published ports are baked in at container-creation time, so a running container will not adopt new values until it is recreated. Your data volume moves with the project name — renaming `DAAF_PROJECT_NAME` on an existing install points it at a *different* (empty) volume, so choose the name once, up front. +### Sharing One Research Workspace Across Two Installs (Advanced) + +The section above sets up **independent** instances — each with its own research volume and history. This one does the opposite: it points **two** DAAF installs at a **single, shared research workspace**, so both can work the same `research/` folder, projects, and audit trail. + +The motivating scenario is running **two different AI providers against one body of work** — for example, one install authenticated to your **Claude (Anthropic) subscription** and a second install on your **ChatGPT subscription** via the [provider shim](#option-f-alternate-lane-chatgpt-subscription-codex-backend). Each install keeps its **own** configuration and authentication volume (`_daaf-claude-config`), so their logins never collide — only the *research data* volume is shared. + +DAAF supports this with a setting, **`DAAF_DATA_VOLUME_NAME`**, plus a second commented opt-in block in `docker-compose.yml`. `DAAF_DATA_VOLUME_NAME` overrides the full Docker volume name for the shared research data (leave it unset and each install gets its own default `_daaf-data`, exactly as in the section above). Its value must match the `name:` in the compose file's external-volume block. The steps below convert two installs to share one volume. + +**Step 1 — Install the first instance normally.** Follow the standard [installation](#installing-daaf), let it run, and confirm your research data is where you expect. Its data volume is `_daaf-data` (where `` is its `DAAF_PROJECT_NAME`, default `daaf` — so the default volume is `daaf_daaf-data`). + +**Step 2 — Convert the first install to an external (shared) volume.** Edit the **container's** copy of `docker-compose.yml` (via Claude or the [browser editor](#viewing-and-editing-files) — remember `rebuild_daaf.sh` copies the container's copy out over the host one, so the container-side edit is the one that survives). Uncomment the external-volume block under the top-level `daaf-data:` key and set `name:` to the volume's **existing** full name: + + ```yaml + volumes: + daaf-data: + external: true + name: daaf_daaf-data # <-- your existing _daaf-data volume + ``` + + Then set the matching line in the **host** `environment_settings.txt`: + + ``` + DAAF_DATA_VOLUME_NAME=daaf_daaf-data + ``` + + Rebuild (`bash rebuild_daaf.sh` / `.\rebuild_daaf.ps1`). Nothing about this install's behavior changes — it still uses the same data — but the volume is now declared `external`, which has one important consequence: **`docker compose down -v` will no longer delete it.** Compose never removes external volumes, so your shared research data is now immune to an accidental `-v` wipe. (Deliberate deletion still works via `docker volume rm` — see the caveats.) + +**Step 3 — Install the second instance, pre-seeded to the shared volume.** Install into a **different folder**, giving it a distinct project name, distinct ports, **its own provider/auth settings** (e.g. the [ChatGPT-subscription shim lane](#option-f-alternate-lane-chatgpt-subscription-codex-backend)), and `DAAF_DATA_VOLUME_NAME` **exported at install time** so it lands in the new install's `environment_settings.txt` from the start (the value stays inert until Step 4 activates the shared volume): + + **macOS / Linux (Terminal):** + + ```bash + export DAAF_PROJECT_NAME=daaf-gpt + export DAAF_PORT_MARIMO=2818 + export DAAF_PORT_LOGVIEWER=2819 + export DAAF_PORT_VSCODE=2820 + export DAAF_DATA_VOLUME_NAME=daaf_daaf-data # same shared volume as install #1 + curl -fsSL https://raw.githubusercontent.com/DAAF-Contribution-Community/daaf/main/scripts/host/install.sh | bash + ``` + + **Windows (PowerShell):** + + ```powershell + $env:DAAF_PROJECT_NAME = "daaf-gpt" + $env:DAAF_PORT_MARIMO = "2818" + $env:DAAF_PORT_LOGVIEWER = "2819" + $env:DAAF_PORT_VSCODE = "2820" + $env:DAAF_DATA_VOLUME_NAME = "daaf_daaf-data" + irm https://raw.githubusercontent.com/DAAF-Contribution-Community/daaf/main/scripts/host/install.ps1 | iex + ``` + +**Step 4 — Convert the second install identically.** The installer still had to create the container from a compose file referencing this instance's *own* `_daaf-data`, so a transitional volume (`daaf-gpt_daaf-data` in the example) now exists and holds only the fresh repo clone. Convert this install exactly as in Step 2 — uncomment the external block in its container compose, set `name: daaf_daaf-data`, confirm `DAAF_DATA_VOLUME_NAME=daaf_daaf-data` is in its `environment_settings.txt`, and rebuild. Once it comes up on the shared volume, delete the now-orphaned transitional volume: + + ```bash + docker volume ls | grep daaf-gpt_daaf-data # confirm it exists and is unused + docker volume rm daaf-gpt_daaf-data # remove the orphaned transitional volume + ``` + +**Step 5 — Steady state.** Both installs now mount the **same** `daaf_daaf-data` volume, which means they share **one** `/daaf` — including a single canonical `docker-compose.yml` and the whole framework tree. Because there is only one copy of those files, a rebuild of *either* install reads and writes the *same* compose file, so the two installs stay consistent by construction; you never maintain two diverging compose files. Each install still keeps its own separate `*_daaf-claude-config` volume, so its provider login and session history remain its own. + +> **Caveats — read before you rely on this.** +> - **One active pipeline per research project at a time.** Docker provides *no* write coordination between two containers on one volume. Running two analysis pipelines against the same project simultaneously can corrupt files. Coordinate yourself: let one install work a given project at a time. (Working *different* projects in the shared workspace concurrently is fine.) +> - **Back up from one install only.** Both installs see the same data, so a single `backup_daaf.sh` run from *either* one captures everything — running backups from both just duplicates work and can race. +> - **Deleting the shared volume is deliberate.** Because it's external, `docker compose down -v` won't touch it. To actually delete it you must bring **both** installs down (no container may reference it) and run `docker volume rm daaf_daaf-data` explicitly. +> - **The second install's initial clone is discarded.** The transitional `_daaf-data` volume created at Step 3 held only a fresh repo clone; converting to the shared volume in Step 4 abandons it (you delete it), and install #2 adopts install #1's existing `/daaf`. That's expected. + ### Building with the developer test toolchain (DAAF_DEV) Most people never need this section — it is for **framework developers** who want to run DAAF's own shell and PowerShell test suites (`bats` and Pester) *inside* the container, so an in-container run reproduces what the project's CI does. @@ -1080,10 +1201,40 @@ A healthy ChatGPT lane logs `backend_mode=chatgpt` at startup and then healthy ` | `codex: command not found` | The image predates the universal Codex installation or was built from a stale Dockerfile. Update DAAF and rebuild (`rebuild_daaf.sh` / `rebuild_daaf.ps1`) so the current Dockerfile is used; `DAAF_DEV=1` is not required. | | Shim log tells you to re-login | The OAuth token refresh failed permanently — run `codex login --device-auth` again inside the container. | -**Running more than one container.** Each DAAF container does its **own** `codex login`, which creates an **independent refresh-token grant** — there is no credential collision between containers. They share only your ChatGPT **usage pool** (the 5-hour and weekly caps), so running two in parallel simply draws that pool down faster. This is the clean way to run parallel DAAF instances. (The subtler case is running several codex-based tools — the `codex` CLI, `codex-plugin-cc`, the shim — inside a *single* container off the *same* login: that can rarely trigger a refresh-token-rotation race. To isolate them, give each its own `codex login` under a separate `CODEX_HOME`; `CODEX_HOME` and the `SHIM_OAUTH_TOKEN_URL` / `SHIM_OAUTH_CLIENT_ID` override variables in `environment_settings_example.txt` are the hooks for that. None of this is needed for a normal single-tool setup.) +**Running more than one container.** Each DAAF container does its **own** `codex login`, which creates an **independent refresh-token grant** — there is no credential collision between containers. They share only your ChatGPT **usage pool** (the 5-hour and weekly caps), so running two in parallel simply draws that pool down faster. This is the clean way to run parallel DAAF instances. (The subtler case is running several codex-based tools — the `codex` CLI, the [Codex plugin](#using-the-codex-plugin-for-claude-code), the shim — inside a *single* container off the *same* login: that can rarely trigger a refresh-token-rotation race. To isolate them, give each its own `codex login` under a separate `CODEX_HOME`; `CODEX_HOME` and the `SHIM_OAUTH_TOKEN_URL` / `SHIM_OAUTH_CLIENT_ID` override variables in `environment_settings_example.txt` are the hooks for that. None of this is needed for a normal single-tool setup.) See the [technical FAQ entry on the ChatGPT subscription lane](07_faq_technical.md#q-can-i-use-my-chatgpt-subscription-instead-of-an-openai-api-key-option-f). +#### Using the Codex Plugin for Claude Code + +Separate from the provider-shim lanes above, OpenAI ships an official plugin — **`codex-plugin-cc`** — that lets a Claude Code session **delegate a task to Codex** for a second opinion: a code review, an adversarial review, a rescue attempt. It adds slash commands like `/codex:review` and `/codex:adversarial-review`. Think of it as a way to get an *independent* model's eyes on a piece of work, complementary to DAAF's own review agents — not a provider route for running DAAF itself. + +**What's already in the image.** The plugin shells out to the local `codex` CLI, and DAAF bakes that in (a pinned static binary — the same one the ChatGPT lane uses). Node.js is present and version-sufficient too, guaranteed by the Dockerfile as of this change. `npm` is absent (Ubuntu packages it separately from `nodejs`), but that shouldn't matter here: `npm` is only needed by `/codex:setup`'s "install Codex for you" branch, and Codex is already installed — so that branch never runs. (If any plugin step ever *does* complain about a missing `npm`, that absence is the likely cause — the codex CLI itself needs no npm.) + +**Install and authenticate (inside a Claude Code session):** + +``` +/plugin marketplace add openai/codex-plugin-cc +/plugin install codex@openai-codex +/reload-plugins +/codex:setup +``` + +Then authenticate Codex with the same headless device-code flow the ChatGPT lane uses: + +```bash +codex login --device-auth +``` + +As with the shim's ChatGPT lane, this requires **device-code login enabled in your ChatGPT security settings** (off by default — enable it first, or the login fails immediately). + +**Which setups are safe.** There are two race-free ways to use the plugin: + +1. **Two containers (the natural pairing with the ChatGPT lane).** Run the plugin in a container on the **Anthropic route** (Claude subscription/API), and keep the ChatGPT-subscription **shim lane** in a *separate* container. Because each container has its own config volume, each has its own Codex login — the plugin and the shim never touch the same token. This pairs naturally with the [shared-workspace setup above](#sharing-one-research-workspace-across-two-installs-advanced). +2. **One Anthropic-route container plus the plugin.** If the container is *not* running the ChatGPT shim lane, nothing else is consuming the Codex login, so the plugin is the sole consumer and there is no contention. + +> **Do not add the Codex plugin to a container that is already using the ChatGPT-subscription shim lane.** In that lane the shim is *itself* a consumer of your Codex login, continuously reading and refreshing the OAuth token in `auth.json`. Adding a second consumer of the same login — the plugin, or for that matter running the `codex` CLI by hand — means two processes can rotate the refresh token at nearly the same moment, and a lost rotation **permanently invalidates the login** (a `refresh_token_reused` lockout that forces a fresh `codex login`). If you genuinely must run both in one container, give each tool its own **separate `CODEX_HOME`** with its own **separate `codex login --device-auth`**, so they never share a token file. Either way, remember that all logins on one ChatGPT account draw from the **same usage pool** (the 5-hour and weekly caps), so adding a second Codex consumer spends that pool faster. + #### Known limitations of GPT sessions (both lanes) GPT support is a **supported capability we've engineered carefully** — actively tested, including through DAAF's own benchmark runs, and still gathering wider community experience across different setups. A few specifics are worth knowing: diff --git a/user_reference/02_understanding_daaf.md b/user_reference/02_understanding_daaf.md index ead338fa..9f4be163 100644 --- a/user_reference/02_understanding_daaf.md +++ b/user_reference/02_understanding_daaf.md @@ -930,6 +930,8 @@ You don't need to know what's in most of these directories -- the two that matte **Key insight for new users:** Everything you need to review, share, or reproduce is inside the project folder. You can copy the entire folder to a colleague and they'd have everything needed to understand and verify the analysis. That's the whole point of reproducibility. +**Where does this all physically live?** The whole repository sits inside a Docker-managed storage volume on your machine — not in a regular folder you can browse directly. That's why the viewing tools below exist, and it's also why backups matter. If you want to go deeper — how the volumes work, how to link a folder from your computer into the container for data access, or how to share one research workspace between two DAAF installs — the [Installation Guide's file-management section](01_installation_and_quickstart.md#how-to-manage-daaf-project-files-and-output) covers all of it. + ### Browsing and Viewing Your Work DAAF includes convenience scripts for viewing your files, notebooks, and session logs outside of the terminal. The simplest way to reach all of them is the **DAAF Control Panel** — run `bash daaf.sh` (macOS/Linux) or `.\daaf.ps1` (Windows) from your `daaf-docker` folder and pick the matching menu option (**2)** Browse Files, **4)** View Marimo Notebooks, **5)** View Quarto Notebooks, **3)** View Session Logs). Each convenience script below also runs directly from your `daaf-docker` folder on the host (i.e., outside the container): diff --git a/user_reference/03_best_practices.md b/user_reference/03_best_practices.md index c1caf28e..1cdd9269 100644 --- a/user_reference/03_best_practices.md +++ b/user_reference/03_best_practices.md @@ -510,6 +510,8 @@ The browser editor comes pre-loaded with extensions for Python and R syntax high - **Git integration:** The Source Control panel (left sidebar) shows uncommitted changes and lets you view diffs -- the most direct way to review exactly what DAAF produced during a session -- and, if you've enabled the optional "Git commit management" preference, browse commit history too. - **Search across files:** `Ctrl+Shift+F` (or `Cmd+Shift+F` on Mac) searches across all files in the project -- great for finding specific variables, scripts, or content. +> **Working with a bulk local dataset?** Uploading tens of gigabytes through the browser is slow. For large or frequently-refreshed local data, bind-mount the folder into the container read-only instead — see [Linking Host Folders into the Container (Bind Mounts)](01_installation_and_quickstart.md#linking-host-folders-into-the-container-bind-mounts) in the installation guide. + ### Alternative: Desktop VSCode with Dev Containers If you already have [VSCode](https://code.visualstudio.com) installed on your computer and prefer a native desktop experience, you can connect it directly to the DAAF container. Install the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) and use the "Attach to Running Container" command to open the DAAF container's filesystem directly in VSCode. This gives you a full file browser, integrated terminal, and your own preferred extensions and settings. diff --git a/user_reference/04_extending_daaf.md b/user_reference/04_extending_daaf.md index e263b54e..6387bfa4 100644 --- a/user_reference/04_extending_daaf.md +++ b/user_reference/04_extending_daaf.md @@ -388,6 +388,8 @@ The two files start out identical, but they can drift apart as soon as either si **The fix is simple but easy to forget:** after DAAF edits the in-container Dockerfile (or docker-compose.yml), you need to copy that updated file back to the host build directory *before* running the container rebuild. The step-by-step process below walks through this carefully, and you should follow it exactly the first time. +**A second host/container boundary to know about:** *bind mounts* — linking a folder on your computer directly into the container — are the other place where the host and container filesystems meet. If you want DAAF to read a large local dataset in place rather than copying it into the Docker volume, see [Linking Host Folders into the Container (Bind Mounts)](01_installation_and_quickstart.md#linking-host-folders-into-the-container-bind-mounts) in the installation guide. + > **One structural note before you edit:** DAAF's image defines a custom `ENTRYPOINT` (`daaf-entrypoint.sh`, written inline by a heredoc block near the end of the Dockerfile — there is no separate source file). It runs optional startup tasks -- such as auto-launching the OpenAI provider shim when `DAAF_PROVIDER_SHIM` is configured (see the [installation guide's Option F](01_installation_and_quickstart.md#option-f-openai-api-directly-daaf-provider-shim)) -- and then hands control to the normal `CMD ["bash"]`. Adding packages or changing `CMD` is safe, but adding a second `ENTRYPOINT` line would silently replace this wrapper and disable those startup behaviors. The heredoc's comment block explains exactly what it does. #### Step-by-Step Process diff --git a/user_reference/07_faq_technical.md b/user_reference/07_faq_technical.md index 43f4b7d2..6cc3d544 100644 --- a/user_reference/07_faq_technical.md +++ b/user_reference/07_faq_technical.md @@ -66,6 +66,18 @@ Another application is using one of DAAF's ports. Either close the conflicting a This usually happens when Docker Desktop's file sharing permissions haven't been configured. Open Docker Desktop → Settings → Resources → File Sharing, and ensure the relevant directories are shared. +### Bind-mounted host folder problems (empty, unreadable, or lost after rebuild) + +If you've linked a host folder into the container as a [bind mount](01_installation_and_quickstart.md#linking-host-folders-into-the-container-bind-mounts) and something isn't right, it's almost always one of these three: + +| Symptom | Cause and fix | +|---------|---------------| +| `/host_data` is **empty** inside the container | The `source:` path is mistyped, or (on Docker Desktop) the folder isn't shared. Check the path, and on macOS/Windows confirm the folder sits under a directory shared in Docker Desktop → Settings → Resources → File Sharing. | +| **Permission denied** reading the files | UID mismatch on native Linux or a WSL-filesystem path — the container user is UID 1000 and `daaf-init` does **not** fix bind-mount ownership. Run `ls -ln` on the host folder; if it isn't owned by `1000` or world-readable, make it group- or world-readable. | +| Your compose edit is **lost after a rebuild** | You edited the *host* copy of `docker-compose.yml`. `rebuild_daaf.sh` copies the container's copy out over the host one, so edit `/daaf/docker-compose.yml` *inside* the container (via Claude or the browser editor), then rebuild. | + +See [Linking Host Folders into the Container (Bind Mounts)](01_installation_and_quickstart.md#linking-host-folders-into-the-container-bind-mounts) for the full setup. + ### Claude Code asks me to log in again This should be uncommon. Claude Code's login, session history, and plugins live in a dedicated Docker volume (`daaf-claude-config`), so they persist across container restarts, `docker compose down`, and even image rebuilds -- a routine restart or update should not sign you out. If you *do* get prompted after a normal restart, just run `/login` once for an Anthropic Max/Pro subscription (or paste your API key); it will persist from then on. Note that `docker compose down -v` (with the `-v` flag) or manually deleting the volume erases this state, so avoid `-v` unless you mean to wipe everything. If you'd rather never log in interactively, set your credentials in the `environment_settings.txt` file in the `daaf-docker/` folder -- it is read on every container start. @@ -349,6 +361,12 @@ The shim reads the OAuth token from `auth.json` and **refreshes it automatically See the full walkthrough in [**01. Installation & Quick Start — Option F, alternate lane: ChatGPT subscription**](01_installation_and_quickstart.md#option-f-alternate-lane-chatgpt-subscription-codex-backend). +### Q: Can I run Claude and ChatGPT DAAF installs at the same time, on the same research workspace? + +Yes. You can run **two DAAF installs in parallel** — one on your Claude (Anthropic) subscription and one on your ChatGPT subscription via the provider shim — and point them at a **single shared research workspace** so both work the same projects and audit trail. Each install keeps its own separate configuration/authentication volume, so the two logins never collide; only the research-data volume is shared. The full step-by-step is in [**01. Installation — Sharing One Research Workspace Across Two Installs**](01_installation_and_quickstart.md#sharing-one-research-workspace-across-two-installs-advanced). + +**One concurrency caveat:** Docker gives you no write coordination between the two containers, so don't run analysis pipelines against the *same research project* from both installs at once — let one install work a given project at a time. (Working different projects in the shared workspace concurrently is fine.) + ### Q: A scripted `claude -p` call on a GPT model returned an empty result Occasionally a GPT turn ends with a reasoning-only block and no visible text, which can surface as an empty `result` field in scripted (non-interactive) `claude -p` usage. This is a GPT quirk, not a DAAF fault, and is only relevant to automated/batch tooling — interactive sessions are unaffected. If you hit it in a script, re-issue the call or add a follow-up turn that requests the answer explicitly. From 1b1373df6692e2f2f20b221864be296c44d5d1eb Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 16:10:58 +0000 Subject: [PATCH 09/66] config(settings): make effort level an overridable default Replace the CLAUDE_CODE_EFFORT_LEVEL=high env-var pin (a hard per-session override that /effort and the /model picker could not change) with the top-level "effortLevel": "high" setting, which keeps high as the default while allowing live /effort overrides. Co-Authored-By: Claude Opus 4.8 (1M context) --- .claude/settings.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/settings.json b/.claude/settings.json index 3c25fb5f..7612cb4d 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -106,13 +106,13 @@ }, "env": { "ANTHROPIC_MODEL": "claude-opus-4-8[1m]", - "CLAUDE_CODE_EFFORT_LEVEL": "high", "CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1", "CLAUDE_CODE_DISABLE_BACKGROUND_TASKS": "1", "DISABLE_AUTOUPDATER": "1", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1", "ENABLE_PROMPT_CACHING_1H": "1" }, + "effortLevel": "high", "outputStyle": "Explanatory", "disableWorkflows": true, "verbose": true, From c5558305bf4db2e7e9378f781544715501966864 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 18:29:37 +0000 Subject: [PATCH 10/66] feat(hooks): add GPT collaborator guidance Inject concise, warm, equal-stakeholder communication guidance for GPT orchestrator sessions and harden the model-cache transport against malformed identities and control-character aliasing. Add focused hook and statusline regression coverage. Co-Authored-By: Claude --- .claude/hooks/remind-orchestrator.sh | 94 +++++++- .claude/scripts/context-bar.sh | 164 +++++++++---- tests/bash/context_bar.bats | 172 +++++++++++++- tests/bash/remind_orchestrator.bats | 342 +++++++++++++++++++++++++++ 4 files changed, 722 insertions(+), 50 deletions(-) create mode 100755 tests/bash/remind_orchestrator.bats diff --git a/.claude/hooks/remind-orchestrator.sh b/.claude/hooks/remind-orchestrator.sh index 563b70ac..0a23a3d2 100755 --- a/.claude/hooks/remind-orchestrator.sh +++ b/.claude/hooks/remind-orchestrator.sh @@ -1,12 +1,18 @@ #!/usr/bin/env bash # remind-orchestrator.sh — Remind main-session LLM to load daaf-orchestrator # -# Hook: UserPromptSubmit (main session only — subagents never trigger this) +# Hook: UserPromptSubmit (main session only — explicit caller guard below) # # On each user message, checks whether the daaf-orchestrator skill has been # loaded this session. If not, injects a reminder via stdout that appears as # text in the LLM's context. # +# While that normal reminder branch is active, genuine terminal gpt-* session +# models also receive a compact collaborator-guidance reminder. Model identity +# comes only from the session-scoped /tmp/claude-model- cache written +# by context-bar.sh. Missing, empty, or malformed identity is a silent no-op for +# GPT-specific guidance; the ordinary orchestrator reminder still fires. +# # Additionally, on the user's FIRST EVER session (detected via activity.log # line count), injects a first-run transparency statement that Claude must # present before proceeding with normal workflow. The transparency content @@ -41,13 +47,93 @@ # 0 = always (never block user messages) INPUT=$(cat) -SESSION_ID=$(echo "$INPUT" | jq -r '.session_id // "default"' 2>/dev/null) || SESSION_ID="default" -FLAG="${HOME}/.claude/daaf-state/orchestrator-loaded-${SESSION_ID}" +INPUT_PARSED=0 +SESSION_ID="" +CALLER_ELIGIBLE=0 + +# One jq pass validates caller eligibility and session identity before either +# value crosses into Bash. The path-safe session grammar is complete and +# anchored inside jq, so command substitution/read cannot normalize a newline, +# NUL, unit separator, or other control into a different cache identity. Caller +# eligibility is also reduced to a trusted 0/1 scalar inside jq: raw agent fields +# never enter Bash, and non-string or malformed nonempty identities stay silent. +# Missing jq, malformed/non-object JSON, or extraction failure preserves the +# ordinary reminder but can never authorize a model-cache read or GPT guidance. +if command -v jq >/dev/null 2>&1; then + if IFS=$'\x1f' read -r SESSION_ID CALLER_ELIGIBLE < <(printf '%s' "$INPUT" | jq -er ' + def valid_session_id: + if type == "string" then + (test("[[:cntrl:]]") | not) and + test("^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") + else false + end; + + if type != "object" then error("hook payload must be an object") + else + [ (if (.session_id? | valid_session_id) + then .session_id + else "" + end), + (if ((.agent_id? == null) or + ((.agent_id? | type) == "string" and .agent_id == "")) and + ((.agent_type? == null) or + ((.agent_type? | type) == "string" and + (.agent_type == "" or .agent_type == "orchestrator"))) + then "1" + else "0" + end) ] + | join("") + end + ' 2>/dev/null); then + INPUT_PARSED=1 + fi +fi + +# Explicit main-thread guard. Successfully parsed subagent or malformed caller +# identity stays silent. Unresolved payload identity preserves the fail-open +# ordinary reminder while GPT-specific guidance remains disabled below. +if [[ "$INPUT_PARSED" -eq 1 && "$CALLER_ELIGIBLE" -ne 1 ]]; then + exit 0 +fi + +# Session IDs become path components for both the transcript-lifetime flag and +# the session-runtime model cache. Only jq-validated IDs are exposed to Bash. +# Invalid or absent IDs retain historical "default" LOADED-FLAG semantics, but +# never select a shared default model cache. +SESSION_ID_VALID=0 +FLAG_SESSION_ID="default" +if [[ "$INPUT_PARSED" -eq 1 && -n "$SESSION_ID" ]]; then + SESSION_ID_VALID=1 + FLAG_SESSION_ID="$SESSION_ID" +fi + +FLAG="${HOME}/.claude/daaf-state/orchestrator-loaded-${FLAG_SESSION_ID}" if [[ ! -f "$FLAG" ]]; then # Always remind to load the orchestrator echo "You are interacting with a human user. You MUST IMMEDIATELY invoke the daaf-orchestrator skill (Skill tool with skill: \"daaf-orchestrator\") BEFORE doing any other work." + # The statusline payload is the authoritative source for the active session + # model. No transcript or environment fallback is used: stale or unresolved + # identity must suppress GPT-specific guidance rather than guess. + MODEL_ID="" + if [[ "$INPUT_PARSED" -eq 1 && "$SESSION_ID_VALID" -eq 1 ]]; then + MODEL_CACHE_DIR="${DAAF_REMIND_MODEL_CACHE_DIR:-/tmp}" + MODEL_CACHE="${MODEL_CACHE_DIR}/claude-model-${SESSION_ID}" + if [[ -s "$MODEL_CACHE" ]]; then + # Command substitution removes trailing newlines. Append a sentinel + # byte before substitution, then remove only that byte afterward so + # cache whitespace — including a final newline — remains detectable. + MODEL_ID_WITH_SENTINEL=$(cat "$MODEL_CACHE" 2>/dev/null; printf 'x') + MODEL_ID="${MODEL_ID_WITH_SENTINEL%x}" + fi + fi + + MODEL_SLUG="${MODEL_ID##*/}" + if [[ -n "$MODEL_ID" && "$MODEL_ID" != *[[:space:]]* && "$MODEL_SLUG" == gpt-?* ]]; then + echo "For this GPT session, treat the user as an equal stakeholder and thoughtful collaborator. Default to concise, user-relevant responses in plain language at their demonstrated technical depth. Lead with the answer or decision; put an immediate action first only when action is what they need. Make required actions, options, recommendations, and consequences easy to spot. When several independent decisions are genuinely needed, present them separately rather than collapsing them into one question. Put secondary technical and implementation detail in clearly marked optional sections or artifacts. Be warm, direct, and easy to scan, never patronizing. Keep full evidence and reproducibility detail available, while surfacing the uncertainty, caveats, citations, safety warnings, and consent checkpoints that affect the current decision." + fi + # Check if this is the user's first-ever session ACTIVITY_LOG="${CLAUDE_PROJECT_DIR:-.}/.claude/logs/activity.log" LINE_COUNT=0 @@ -60,7 +146,7 @@ if [[ ! -f "$FLAG" ]]; then if [[ "$LINE_COUNT" -le 1 ]]; then # First session ever — inject transparency statement HOOK_DIR="$(cd "$(dirname "$0")" && pwd)" - TRANSPARENCY_FILE="${HOOK_DIR}/first-run-transparency.txt" + TRANSPARENCY_FILE="${DAAF_REMIND_TRANSPARENCY_FILE:-${HOOK_DIR}/first-run-transparency.txt}" if [[ -f "$TRANSPARENCY_FILE" ]]; then echo "" cat "$TRANSPARENCY_FILE" diff --git a/.claude/scripts/context-bar.sh b/.claude/scripts/context-bar.sh index bca525c9..f3e8b3dc 100755 --- a/.claude/scripts/context-bar.sh +++ b/.claude/scripts/context-bar.sh @@ -41,44 +41,116 @@ is_canonical_positive_decimal() { input=$(cat) -# Single consolidated jq pass over the payload: extract every field used below -# as one tab-separated record, then read into shell variables. This replaces -# what were ~5 separate `jq` invocations on "$input" (one process fork each). -# Field order and defaults are preserved byte-for-byte from the prior per-field -# calls so downstream logic (OpenRouter override, transcript parsing, the -# /tmp/claude-ctx-window write) behaves identically: -# model = .model.display_name // .model.id // "?" -# cwd = .cwd // "" -# transcript_path = .transcript_path // "" -# max_context = .context_window.context_window_size // 200000 -# session_id = .session_id // "default" -# model_id = .model.id // "" (used by the OpenRouter block) -# New optional segments (all default to empty when absent, e.g. API-key sessions): -# effort_level = .effort.level -# rl_5h = .rate_limits.five_hour.used_percentage -# rl_5h_reset = .rate_limits.five_hour.resets_at -# rl_7d = .rate_limits.seven_day.used_percentage -# rl_7d_reset = .rate_limits.seven_day.resets_at -# Fields are joined with the ASCII unit separator \x1f, NOT @tsv: tab is IFS -# *whitespace* in bash, so consecutive tabs collapse and any EMPTY field (e.g. -# transcript_path at session start, or the absent effort/rate-limit fields) -# would silently shift every later field left. A non-whitespace IFS preserves -# empty fields. See subagent-bar.sh FIELD-JOINING NOTE for the discovery story. -IFS=$'\x1f' read -r model cwd transcript_path max_context session_id model_id \ - effort_level rl_5h rl_5h_reset rl_7d rl_7d_reset < <(echo "$input" | jq -r ' - [ (.model.display_name // .model.id // "?"), - (.cwd // ""), - (.transcript_path // ""), - (.context_window.context_window_size // 200000 | tostring), - (.session_id // "default"), - (.model.id // ""), - (.effort.level // ""), - (.rate_limits.five_hour.used_percentage // ""), - (.rate_limits.five_hour.resets_at // ""), - (.rate_limits.seven_day.used_percentage // ""), - (.rate_limits.seven_day.resets_at // "") ] - | map(tostring) | join("\u001f") -') +# One jq pass validates and normalizes every field before joining it with an +# ASCII unit separator. Identity/path strings containing C0/DEL controls become +# empty; display-only strings have those controls removed; numeric fields accept +# only their expected JSON scalar type. The session ID is validated against the +# complete path-safe grammar inside jq, before Bash can normalize a newline or +# drop a NUL. Therefore every emitted field is delimiter/control-free and the +# fixed-order read cannot be shifted by untrusted JSON content. +model="?" +cwd="" +transcript_path="" +max_context="200000" +session_id="" +model_id="" +effort_level="" +rl_5h="" +rl_5h_reset="" +rl_7d="" +rl_7d_reset="" +payload_parsed=0 +if command -v jq >/dev/null 2>&1; then + if IFS=$'\x1f' read -r model cwd transcript_path max_context session_id model_id \ + effort_level rl_5h rl_5h_reset rl_7d rl_7d_reset < <(printf '%s' "$input" | jq -er ' + def has_control: + test("[[:cntrl:]]"); + def display_string($fallback): + if type == "string" then gsub("[[:cntrl:]]"; "") + else $fallback + end; + def control_free_string: + if type == "string" then + if has_control then "" else . end + else "" + end; + def reset_scalar: + if type == "number" then tostring + elif type == "string" then + if has_control then "" else . end + else "" + end; + def valid_session_id: + if type == "string" then + (has_control | not) and + test("^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") + else false + end; + + if type != "object" then error("statusline payload must be an object") + else + [ ((.model.display_name? // .model.id? // "?") | display_string("?")), + ((.cwd? // "") | control_free_string), + ((.transcript_path? // "") | control_free_string), + (if (.context_window.context_window_size? | type) == "number" then + if .context_window.context_window_size > 0 and + (.context_window.context_window_size | floor) == .context_window.context_window_size + then (.context_window.context_window_size | tostring) + else "200000" + end + else "200000" + end), + (if (.session_id? | valid_session_id) + then .session_id + else "" + end), + ((.model.id? // "") | control_free_string), + ((.effort.level? // "") | display_string("")), + (if (.rate_limits.five_hour.used_percentage? | type) == "number" + then (.rate_limits.five_hour.used_percentage | tostring) + else "" + end), + ((.rate_limits.five_hour.resets_at? // "") | reset_scalar), + (if (.rate_limits.seven_day.used_percentage? | type) == "number" + then (.rate_limits.seven_day.used_percentage | tostring) + else "" + end), + ((.rate_limits.seven_day.resets_at? // "") | reset_scalar) ] + | join("\u001f") + end + ' 2>/dev/null); then + payload_parsed=1 + fi +fi + +# Production caches live in /tmp. The override is a deterministic-test seam so +# Bats can exercise writes inside project scratch without touching live session +# state. Cache eligibility requires a successfully parsed payload plus a +# nonempty session ID in the established UUID-safe character set and bounded +# length. All other cases keep statusline rendering fail-open while skipping +# every session-scoped cache read and write below. +context_cache_dir="${DAAF_CONTEXT_BAR_CACHE_DIR:-/tmp}" +session_id_safe=0 +if [[ "$payload_parsed" -eq 1 && \ + "$session_id" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ && \ + ${#session_id} -le 128 ]]; then + session_id_safe=1 +fi + +# Seed the authoritative main-session model cache from the statusline's +# .model.id. Replace through a same-directory sibling so reminder/reporting +# hooks never observe a partial write. An empty authoritative ID intentionally +# replaces any stale value with an empty file, causing GPT-specific consumers to +# treat identity as unresolved rather than guess. +if [[ "$session_id_safe" -eq 1 ]]; then + model_cache="${context_cache_dir}/claude-model-${session_id}" + model_tmp="${model_cache}.tmp.$$" + if printf '%s' "$model_id" > "$model_tmp" 2>/dev/null; then + mv "$model_tmp" "$model_cache" 2>/dev/null || rm -f "$model_tmp" 2>/dev/null + else + rm -f "$model_tmp" 2>/dev/null + fi +fi # Guard against an unparseable payload leaving max_context empty (which would # cause a divide-by-zero in the pct arithmetic below). Valid payloads always @@ -110,11 +182,9 @@ fi # authoritative catalogue result and must not be mistaken for the generic # fallback by the static map below. or_context_resolved=0 -# Production uses /tmp so hooks and statuslines share one session cache. The -# override is a deterministic-test seam only: Bats points it at project scratch -# so fake OpenRouter catalogues never write fixture data outside the repository. -context_cache_dir="${DAAF_CONTEXT_BAR_CACHE_DIR:-/tmp}" -if [[ "${ANTHROPIC_BASE_URL:-}" == *openrouter.ai* ]]; then +# context_cache_dir and the session-ID path guard are initialized immediately +# after payload extraction. Malformed session IDs skip this cache-backed branch. +if [[ "$session_id_safe" -eq 1 && "${ANTHROPIC_BASE_URL:-}" == *openrouter.ai* ]]; then # model_id already extracted in the consolidated jq pass above # (byte-identical to the old `.model.id // empty`). or_cache="${context_cache_dir}/claude-or-models-${session_id}" @@ -235,8 +305,12 @@ fi max_k=$((max_context / 1000)) -# Share context window size with hooks (which don't receive it in their input payload) -echo "$max_context" > "${context_cache_dir}/claude-ctx-window-${session_id}" 2>/dev/null +# Share context window size with hooks (which don't receive it in their input +# payload). Unsafe session IDs skip the path construction and retain fail-open +# statusline output without creating an attacker-controlled cache path. +if [[ "$session_id_safe" -eq 1 ]]; then + printf '%s\n' "$max_context" > "${context_cache_dir}/claude-ctx-window-${session_id}" 2>/dev/null +fi # Calculate context bar from transcript if [[ -n "$transcript_path" && -f "$transcript_path" ]]; then diff --git a/tests/bash/context_bar.bats b/tests/bash/context_bar.bats index f3bf1edb..cf1a96f1 100755 --- a/tests/bash/context_bar.bats +++ b/tests/bash/context_bar.bats @@ -19,6 +19,7 @@ setup() { MOCK_BIN="${SCRATCH_DIR}/bin" DAAF_CONTEXT_BAR_CACHE_DIR="${SCRATCH_DIR}/cache" CTX_CACHE="${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-ctx-window-${FAKE_SESSION}" + MODEL_CACHE="${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-model-${FAKE_SESSION}" OR_CACHE="${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-or-models-${FAKE_SESSION}" mkdir -p "$MOCK_BIN" "$DAAF_CONTEXT_BAR_CACHE_DIR" cat > "${MOCK_BIN}/curl" <<'MOCK_CURL' @@ -32,7 +33,7 @@ MOCK_CURL unset DAAF_PROVIDER_SHIM unset SHIM_BACKEND_MODE export CONTEXT_BAR_SH FAKE_SESSION SCRATCH_DIR MOCK_BIN - export DAAF_CONTEXT_BAR_CACHE_DIR CTX_CACHE OR_CACHE + export DAAF_CONTEXT_BAR_CACHE_DIR CTX_CACHE MODEL_CACHE OR_CACHE } teardown() { @@ -51,6 +52,175 @@ _payload() { assert_success } +@test "authoritative statusline model id is cached atomically for the session" { + run bash "$CONTEXT_BAR_SH" < <(_payload "openrouter/openai/gpt-5.6-terra[1m]") + assert_success + + run cat "$MODEL_CACHE" + assert_success + assert_output "openrouter/openai/gpt-5.6-terra[1m]" + + run find "$DAAF_CONTEXT_BAR_CACHE_DIR" -maxdepth 1 -name 'claude-model-*.tmp.*' -print + assert_success + assert_output "" +} + +@test "empty authoritative model id replaces a stale model cache with unresolved identity" { + printf '%s' 'gpt-5.6-sol' > "$MODEL_CACHE" + run bash "$CONTEXT_BAR_SH" < "${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-model-safe" + printf '%s' '111111' > "${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-ctx-window-safe" + printf '%s' 'seed-victim-model' > "${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-model-victim" + printf '%s' '222222' > "${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-ctx-window-victim" + + local payload + for payload in \ + '{"model":{"id":"claude-fable-5","display_name":"Fable 5"},"session_id":"safe\nid","context_window":{"context_window_size":1000000}}' \ + '{"model":{"id":"claude-fable-5","display_name":"Fable 5"},"session_id":"safe\n","context_window":{"context_window_size":1000000}}'; do + run bash "$CONTEXT_BAR_SH" <<< "$payload" + assert_success + assert_output --partial "Fable 5" + refute_output --partial "of 1050k tokens" + done + + printf -v payload '{"model":{"id":"claude-fable-5","display_name":"Fable 5"},"session_id":"victim\\u%04x","context_window":{"context_window_size":1000000}}' 0 + run bash "$CONTEXT_BAR_SH" <<< "$payload" + assert_success + assert_output --partial "Fable 5" + refute_output --partial "of 1050k tokens" + + printf -v payload '{"model":{"id":"claude-fable-5","display_name":"Fable 5"},"session_id":"victim\\u%04xgpt-5.6-sol","context_window":{"context_window_size":1000000}}' 31 + run bash "$CONTEXT_BAR_SH" <<< "$payload" + assert_success + assert_output --partial "Fable 5" + refute_output --partial "of 1050k tokens" + + run cat "${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-model-safe" + assert_success + assert_output "seed-safe-model" + run cat "${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-ctx-window-safe" + assert_success + assert_output "111111" + run cat "${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-model-victim" + assert_success + assert_output "seed-victim-model" + run cat "${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-ctx-window-victim" + assert_success + assert_output "222222" +} + +@test "controls in earlier display and path fields cannot shift later identity or rate fields" { + local payload + printf -v payload '{"model":{"id":"gpt-5.6-sol","display_name":"GPT\\u%04xDisplay\\u%04xName"},"cwd":"bad\\u%04xpath","transcript_path":"bad\\u%04xtranscript","session_id":"%s","context_window":{"context_window_size":1050000},"effort":{"level":"high\\u%04xshift"},"rate_limits":{"five_hour":{"used_percentage":"42\\u%04xgpt","resets_at":"bad\\u%04xreset"},"seven_day":{"used_percentage":13,"resets_at":0}}}' 10 31 10 31 "$FAKE_SESSION" 10 31 10 + run bash "$CONTEXT_BAR_SH" <<< "$payload" + assert_success + assert_output --partial "GPTDisplayName" + assert_output --partial "of 1050k tokens" + assert_output --partial "Plan usage:" + assert_output --partial "7d:13%" + refute_output --partial "5h:" + + run cat "$MODEL_CACHE" + assert_success + assert_output "gpt-5.6-sol" + run cat "$CTX_CACHE" + assert_success + assert_output "1050000" +} + +@test "control-bearing model identity is unresolved and cannot poison a stale cache" { + local payload + printf '%s' 'gpt-5.6-sol' > "$MODEL_CACHE" + printf -v payload '{"model":{"id":"claude-fable-5\\u%04xgpt-5.6-sol","display_name":"Fable 5"},"cwd":"%s","transcript_path":"","session_id":"%s","context_window":{"context_window_size":200000}}' 31 "$TEST_DIR" "$FAKE_SESSION" + run bash "$CONTEXT_BAR_SH" <<< "$payload" + assert_success + assert_output --partial "Fable 5" + assert_output --partial "of 200k tokens" + refute_output --partial "of 1050k tokens" + + run test ! -s "$MODEL_CACHE" + assert_success +} + +@test "missing null empty and non-string session ids never use a shared default cache" { + local payload + for payload in \ + '{"model":{"id":"gpt-5.6-sol","display_name":"gpt-5.6-sol"},"context_window":{"context_window_size":200000}}' \ + '{"model":{"id":"gpt-5.6-sol","display_name":"gpt-5.6-sol"},"session_id":null,"context_window":{"context_window_size":200000}}' \ + '{"model":{"id":"gpt-5.6-sol","display_name":"gpt-5.6-sol"},"session_id":"","context_window":{"context_window_size":200000}}' \ + '{"model":{"id":"gpt-5.6-sol","display_name":"gpt-5.6-sol"},"session_id":42,"context_window":{"context_window_size":200000}}' \ + '{"model":{"id":"gpt-5.6-sol","display_name":"gpt-5.6-sol"},"session_id":true,"context_window":{"context_window_size":200000}}' \ + '{"model":{"id":"gpt-5.6-sol","display_name":"gpt-5.6-sol"},"session_id":{"value":"object-id"},"context_window":{"context_window_size":200000}}' \ + '{"model":{"id":"gpt-5.6-sol","display_name":"gpt-5.6-sol"},"session_id":["array-id"],"context_window":{"context_window_size":200000}}'; do + run bash "$CONTEXT_BAR_SH" <<< "$payload" + assert_success + assert_output --partial "gpt-5.6-sol" + done + + run find "$DAAF_CONTEXT_BAR_CACHE_DIR" -mindepth 1 -maxdepth 1 -print + assert_success + assert_output "" +} + +@test "malformed statusline input exits successfully without writing session caches" { + run bash "$CONTEXT_BAR_SH" <<< 'not-json {{{' + assert_success + + run find "$DAAF_CONTEXT_BAR_CACHE_DIR" -mindepth 1 -maxdepth 1 -print + assert_success + assert_output "" +} + +@test "missing jq leaves statusline fail-open and creates no session or default cache" { + local bindir="${SCRATCH_DIR}/nojq-bin" + mkdir -p "$bindir" + ln -s "$(command -v bash)" "${bindir}/bash" + ln -s "$(command -v cat)" "${bindir}/cat" + ln -s "$(command -v basename)" "${bindir}/basename" + + run env -i \ + PATH="$bindir" \ + DAAF_CONTEXT_BAR_CACHE_DIR="$DAAF_CONTEXT_BAR_CACHE_DIR" \ + bash "$CONTEXT_BAR_SH" < <(_payload "gpt-5.6-sol") + assert_success + assert_output --partial "of 200k tokens" + + run find "$DAAF_CONTEXT_BAR_CACHE_DIR" -mindepth 1 -maxdepth 1 -print + assert_success + assert_output "" +} + @test "gpt-5.6-sol[1m] displays 1050k and caches the 1050000 physical window" { run bash "$CONTEXT_BAR_SH" < <(_payload "gpt-5.6-sol[1m]") assert_success diff --git a/tests/bash/remind_orchestrator.bats b/tests/bash/remind_orchestrator.bats new file mode 100755 index 00000000..4c24704c --- /dev/null +++ b/tests/bash/remind_orchestrator.bats @@ -0,0 +1,342 @@ +#!/usr/bin/env bats +# ============================================================================ +# Tests for remind-orchestrator.sh -- first-prompt orchestrator reminder +# ============================================================================ +# The protected deployed hook is not edited by automation. SCRIPT UNDER TEST is +# parameterized so this suite can validate the workspace draft before a user +# copies it into .claude/hooks/: +# REMIND_ORCHESTRATOR_SH=/path/to/draft/remind-orchestrator.sh \ +# bats tests/bash/remind_orchestrator.bats +# ============================================================================ + +load 'test_helper' + +REMIND_ORCHESTRATOR_SH="${REMIND_ORCHESTRATOR_SH:-${REPO_ROOT}/.claude/hooks/remind-orchestrator.sh}" +CONTEXT_BAR_SH="${CONTEXT_BAR_SH:-${REPO_ROOT}/.claude/scripts/context-bar.sh}" +FAKE_SESSION="bats-remind-orchestrator-session" +ORDINARY_REMINDER='You are interacting with a human user. You MUST IMMEDIATELY invoke the daaf-orchestrator skill' +GUIDANCE_MARKER='For this GPT session, treat the user as an equal stakeholder and thoughtful collaborator' +GUIDANCE_CONCISE='Default to concise, user-relevant responses in plain language' +GUIDANCE_DEPTH='at their demonstrated technical depth' +GUIDANCE_ANSWER='Lead with the answer or decision' +GUIDANCE_ACTION='put an immediate action first only when action is what they need' +GUIDANCE_OPTIONS='Make required actions, options, recommendations, and consequences easy to spot' +GUIDANCE_DECISIONS='When several independent decisions are genuinely needed, present them separately' +GUIDANCE_DETAIL='Put secondary technical and implementation detail in clearly marked optional sections or artifacts' +GUIDANCE_RIGOR='Keep full evidence and reproducibility detail available' +GUIDANCE_UNCERTAINTY='surfacing the uncertainty, caveats, citations, safety warnings, and consent checkpoints that affect the current decision' + +setup() { + common_setup + TEST_HOME="${TEST_DIR}/home" + PROJECT_DIR="${TEST_DIR}/project" + MODEL_CACHE_DIR="${TEST_DIR}/model-cache" + STATE_DIR="${TEST_HOME}/.claude/daaf-state" + ACTIVITY_LOG="${PROJECT_DIR}/.claude/logs/activity.log" + TRANSPARENCY_FILE="${REPO_ROOT}/.claude/hooks/first-run-transparency.txt" + mkdir -p "$STATE_DIR" "$(dirname "$ACTIVITY_LOG")" "$MODEL_CACHE_DIR" + printf 'session-one\nsession-two\n' > "$ACTIVITY_LOG" + export REMIND_ORCHESTRATOR_SH CONTEXT_BAR_SH FAKE_SESSION TEST_HOME PROJECT_DIR + export MODEL_CACHE_DIR STATE_DIR ACTIVITY_LOG TRANSPARENCY_FILE + export ORDINARY_REMINDER GUIDANCE_MARKER GUIDANCE_CONCISE GUIDANCE_DEPTH + export GUIDANCE_ANSWER GUIDANCE_ACTION GUIDANCE_OPTIONS GUIDANCE_DECISIONS + export GUIDANCE_DETAIL GUIDANCE_RIGOR GUIDANCE_UNCERTAINTY +} + +teardown() { + common_teardown +} + +_payload_main() { + printf '{"hook_event_name":"UserPromptSubmit","session_id":"%s"}' "$FAKE_SESSION" +} + +_payload_agent_id() { + printf '{"hook_event_name":"UserPromptSubmit","session_id":"%s","agent_id":"agent-123","agent_type":"general-purpose"}' "$FAKE_SESSION" +} + +_payload_agent_type() { + printf '{"hook_event_name":"UserPromptSubmit","session_id":"%s","agent_type":"%s"}' "$FAKE_SESSION" "$1" +} + +_seed_model() { + printf '%s' "$1" > "${MODEL_CACHE_DIR}/claude-model-${FAKE_SESSION}" +} + +_run_hook() { + run env \ + HOME="$TEST_HOME" \ + CLAUDE_PROJECT_DIR="$PROJECT_DIR" \ + DAAF_REMIND_MODEL_CACHE_DIR="$MODEL_CACHE_DIR" \ + DAAF_REMIND_TRANSPARENCY_FILE="$TRANSPARENCY_FILE" \ + bash "$REMIND_ORCHESTRATOR_SH" +} + +@test "remind-orchestrator.sh draft parses without errors" { + run bash -n "$REMIND_ORCHESTRATOR_SH" + assert_success +} + +@test "bare GPT Sol receives the stable collaborator policy clauses without a one-question constraint" { + _seed_model 'gpt-5.6-sol' + _run_hook < <(_payload_main) + assert_success + assert_output --partial "$ORDINARY_REMINDER" + assert_output --partial "$GUIDANCE_MARKER" + assert_output --partial "$GUIDANCE_CONCISE" + assert_output --partial "$GUIDANCE_DEPTH" + assert_output --partial "$GUIDANCE_ANSWER" + assert_output --partial "$GUIDANCE_ACTION" + assert_output --partial "$GUIDANCE_OPTIONS" + assert_output --partial "$GUIDANCE_DECISIONS" + assert_output --partial "$GUIDANCE_DETAIL" + assert_output --partial "$GUIDANCE_RIGOR" + assert_output --partial "$GUIDANCE_UNCERTAINTY" + assert_output --partial 'Be warm, direct, and easy to scan, never patronizing.' + refute_output --partial 'ask one' + refute_output --partial 'single question' + refute_output --partial 'one clear question' + refute_output --partial 'at most one' + refute_output --partial 'only one question' + refute_output --partial 'one question at a time' +} + +@test "statusline model-cache writer feeds the first-prompt guidance path end to end" { + run env DAAF_CONTEXT_BAR_CACHE_DIR="$MODEL_CACHE_DIR" \ + bash "$CONTEXT_BAR_SH" < "${MODEL_CACHE_DIR}/claude-model-${FAKE_SESSION}" + _run_hook < <(_payload_main) + assert_success + assert_output --partial "$ORDINARY_REMINDER" + refute_output --partial "$GUIDANCE_MARKER" +} + +@test "malformed cache content does not enable GPT-specific guidance" { + printf 'gpt-5.6-sol\ntrailing-data' > "${MODEL_CACHE_DIR}/claude-model-${FAKE_SESSION}" + _run_hook < <(_payload_main) + assert_success + assert_output --partial "$ORDINARY_REMINDER" + refute_output --partial "$GUIDANCE_MARKER" +} + +@test "trailing newline in the model cache makes identity ineligible" { + printf 'gpt-5.6-sol\n' > "${MODEL_CACHE_DIR}/claude-model-${FAKE_SESSION}" + _run_hook < <(_payload_main) + assert_success + assert_output --partial "$ORDINARY_REMINDER" + refute_output --partial "$GUIDANCE_MARKER" +} + +@test "malformed input cannot read a seeded shared default model cache" { + printf '%s' 'gpt-5.6-sol' > "${MODEL_CACHE_DIR}/claude-model-default" + _run_hook <<< 'not-json {{{' + assert_success + assert_output --partial "$ORDINARY_REMINDER" + refute_output --partial "$GUIDANCE_MARKER" +} + +@test "missing null and empty session ids cannot read a seeded shared default model cache" { + local payload + printf '%s' 'gpt-5.6-sol' > "${MODEL_CACHE_DIR}/claude-model-default" + for payload in \ + '{"hook_event_name":"UserPromptSubmit"}' \ + '{"hook_event_name":"UserPromptSubmit","session_id":null}' \ + '{"hook_event_name":"UserPromptSubmit","session_id":""}'; do + _run_hook <<< "$payload" + assert_success + assert_output --partial "$ORDINARY_REMINDER" + refute_output --partial "$GUIDANCE_MARKER" + done +} + +@test "newline NUL and unit-separator session identities cannot read seeded alias caches" { + local payload + printf '%s' 'gpt-5.6-sol' > "${MODEL_CACHE_DIR}/claude-model-safe" + printf '%s' 'gpt-5.6-sol' > "${MODEL_CACHE_DIR}/claude-model-victim" + touch "${STATE_DIR}/orchestrator-loaded-safe" "${STATE_DIR}/orchestrator-loaded-victim" + + for payload in \ + '{"hook_event_name":"UserPromptSubmit","session_id":"safe\nid"}' \ + '{"hook_event_name":"UserPromptSubmit","session_id":"safe\n"}'; do + _run_hook <<< "$payload" + assert_success + assert_output --partial "$ORDINARY_REMINDER" + refute_output --partial "$GUIDANCE_MARKER" + done + + printf -v payload '{"hook_event_name":"UserPromptSubmit","session_id":"victim\\u%04x"}' 0 + _run_hook <<< "$payload" + assert_success + assert_output --partial "$ORDINARY_REMINDER" + refute_output --partial "$GUIDANCE_MARKER" + + printf -v payload '{"hook_event_name":"UserPromptSubmit","session_id":"victim\\u%04xgpt-5.6-sol"}' 31 + _run_hook <<< "$payload" + assert_success + assert_output --partial "$ORDINARY_REMINDER" + refute_output --partial "$GUIDANCE_MARKER" +} + +@test "malformed nonempty caller identity is silent and cannot enable GPT guidance" { + local payload + _seed_model 'gpt-5.6-sol' + + for payload in \ + '{"hook_event_name":"UserPromptSubmit","session_id":"bats-remind-orchestrator-session","agent_id":"agent\nnormalized"}' \ + '{"hook_event_name":"UserPromptSubmit","session_id":"bats-remind-orchestrator-session","agent_id":{"id":"agent-123"}}' \ + '{"hook_event_name":"UserPromptSubmit","session_id":"bats-remind-orchestrator-session","agent_type":"orchestrator\nsubagent"}' \ + '{"hook_event_name":"UserPromptSubmit","session_id":"bats-remind-orchestrator-session","agent_type":42}'; do + _run_hook <<< "$payload" + assert_success + assert_output "" + done +} + +@test "explicit empty caller identity remains eligible for GPT guidance" { + _seed_model 'gpt-5.6-sol' + _run_hook < "$ACTIVITY_LOG" + _seed_model 'gpt-5.6-sol' + _run_hook < <(_payload_main) + assert_success + assert_output --partial "$ORDINARY_REMINDER" + assert_output --partial "$GUIDANCE_MARKER" + assert_output --partial 'FIRST-TIME USER DETECTED: ONBOARDING PROCESS TRIGGERED.' + assert_output --partial 'KEY POINTS TO COVER:' +} + +@test "missing jq cannot read a seeded shared default cache and remains fail-open" { + local bindir="${TEST_DIR}/nojq-bin" + mkdir -p "$bindir" + ln -s "$(command -v bash)" "${bindir}/bash" + ln -s "$(command -v cat)" "${bindir}/cat" + ln -s "$(command -v wc)" "${bindir}/wc" + ln -s "$(command -v tr)" "${bindir}/tr" + printf '%s' 'gpt-5.6-sol' > "${MODEL_CACHE_DIR}/claude-model-default" + + run env -i \ + PATH="$bindir" \ + HOME="$TEST_HOME" \ + CLAUDE_PROJECT_DIR="$PROJECT_DIR" \ + DAAF_REMIND_MODEL_CACHE_DIR="$MODEL_CACHE_DIR" \ + bash "$REMIND_ORCHESTRATOR_SH" < <(_payload_main) + assert_success + assert_output --partial "$ORDINARY_REMINDER" + refute_output --partial "$GUIDANCE_MARKER" +} + +@test "unsafe session id cannot select a model cache but ordinary reminder survives" { + printf '%s' 'gpt-5.6-sol' > "${MODEL_CACHE_DIR}/claude-model-default" + run env \ + HOME="$TEST_HOME" \ + CLAUDE_PROJECT_DIR="$PROJECT_DIR" \ + DAAF_REMIND_MODEL_CACHE_DIR="$MODEL_CACHE_DIR" \ + bash "$REMIND_ORCHESTRATOR_SH" \ + <<<'{"hook_event_name":"UserPromptSubmit","session_id":"../unsafe"}' + assert_success + assert_output --partial "$ORDINARY_REMINDER" + refute_output --partial "$GUIDANCE_MARKER" +} From 496c59360bcf5e527129690f89fde68e7115f256 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 18:30:54 +0000 Subject: [PATCH 11/66] fix(shim): v1.2.14 robustness + diagnostics (R1-R6, D1/D2/D5) + lifecycle (D3/D4) Release verified 158/158 Python + 41/41 bats, three-angle review clean. Includes post-review fix-it round: no-reasoning-turn test, trailing-thinking history-replay test, bounded error-body read (MAX_ERROR_BODY_BYTES 1 MiB). Co-Authored-By: Claude Fable 5 --- .../provider_shim/anthropic_openai_shim.py | 1243 ++++++++++++++--- scripts/provider_shim/start_shim.sh | 117 +- tests/bash/start_shim.bats | 108 ++ tests/provider_shim/_loopback_harness.py | 247 +++- .../test_historical_regressions.py | 2 +- .../test_reasoning_formatting.py | 2 +- .../provider_shim/test_request_translation.py | 87 ++ tests/provider_shim/test_stream_hardening.py | 152 +- .../test_v1214_heartbeat_and_quota.py | 211 +++ .../provider_shim/test_v1214_observability.py | 428 ++++++ .../test_v1214_retry_and_tail.py | 172 +++ .../test_v1214_tolerant_reducer.py | 510 +++++++ 12 files changed, 3070 insertions(+), 209 deletions(-) create mode 100644 tests/provider_shim/test_v1214_heartbeat_and_quota.py create mode 100644 tests/provider_shim/test_v1214_observability.py create mode 100644 tests/provider_shim/test_v1214_retry_and_tail.py create mode 100644 tests/provider_shim/test_v1214_tolerant_reducer.py diff --git a/scripts/provider_shim/anthropic_openai_shim.py b/scripts/provider_shim/anthropic_openai_shim.py index 86e76c67..2c2be411 100644 --- a/scripts/provider_shim/anthropic_openai_shim.py +++ b/scripts/provider_shim/anthropic_openai_shim.py @@ -63,6 +63,100 @@ # * GET /health endpoint for the manager's idempotency/status checks. # # Changelog: +# v1.2.14 (2026-07-20): Robustness (R1-R6) + diagnostics (D1-D5) hardening pass, +# driven by the v1.2.8/v1.2.13 self-inflicted-outage retrospective. Doctrine: +# "strict emit, tolerant accept" — every change moves an unexpected upstream +# input from {silent drop | hard fail} to {tolerate + count + log}, while the +# downstream Anthropic SSE emission stays exactly as strict as before. +# ROBUSTNESS: +# R1 (observability): a bounded process map + per-request counters make unknown +# SSE event types, unmodeled output_item types, and unrecognized error +# envelopes VISIBLE (terminal record: unknown_events/unknown_items, 0 when +# clean) instead of vanishing. Logged via the v1.2.12 wire-divergence channel +# (first + every 100th), never per-event, never bodies. A live obfuscation +# field on arguments.delta is explicitly tolerated (no count, no failure). +# R2 (classification): ONE code-driven classifier on all failure paths (pre- +# stream HTTP, in-band error, response.failed, non-stream adapter). Envelope +# tolerance parses {error:{...}}, {status,error:{...}}, flat root code/message, +# and the Codex {detail:"..."} shape; first match wins, unrecognized is R1- +# counted then status-classified. insufficient_quota/usage_not_included -> +# invalid_request_error (mirrors Anthropic's own credit-balance 400); unknown- +# code 4xx -> invalid_request_error (CHANGED from retryable api_error). +# ADJUDICATED: a bare/unknown-code 503 -> api_error (retryable); overloaded_ +# error is reserved for recognized overload codes and status 529 (both classes +# retryable, so client behavior is equivalent). +# R4 (retry policy): retry gating is now classification-driven (RETRY_STATUSES +# stays as the envelope-less fallback). insufficient_quota-coded 429 fails +# fast (was retried as a transient 429 — a confirmed defect class). Per-attempt +# delay precedence parsed -> Retry-After -> backoff; the rate-limit-delay hint +# ("try again in ") is parsed for the NUMBER only (gated on +# rate_limit_exceeded, never logged). Rate-limit-class cap raised 30s -> 60s; +# a delay beyond the cap fails fast with rate_limit_error so the client owns +# the long wait. Terminal record gains retry_delay_source (parsed|header| +# backoff). Retry counts and the no-retry-after-first-byte invariant unchanged. +# The ≥400 error body read for classification is bounded at 1 MiB +# (MAX_ERROR_BODY_BYTES): a pathologically large body is truncated before the +# retry-classification parse and falls back to HTTP-status gating; the full body +# stays cached on resp for the caller's error re-read (behavior unchanged). +# R5 (SSE tail): BEHAVIOR CHANGE — at EOF a non-blank-terminated tail whose +# pending event parses cleanly is now FLUSHED as the final event (recovers a +# complete terminal whose trailing blank line a proxy trimmed, B7); a malformed +# tail after the terminal frame is R1-counted and ignored; before the terminal +# it stays a fatal framing failure. One pinned test intentionally flipped +# (..._fails -> ..._flushes). +# R3 (tolerant reducer): open-block bookkeeping is a deferred-open scheduler +# (pending tools opened FIFO in added order; at most one Anthropic tool_use +# block open at a time; downstream blocks stay strictly non-overlapping). +# null/[] response.completed.output after tool events is guarded (streamed +# state wins); a text block still opens on output_text.delta independent of +# content_part.added; a no-reasoning turn is tolerated. Malformed-JSON on NON- +# load-bearing status events is skipped + R1-counted; load-bearing events +# (deltas, argument events, output_item.*, terminal, error) keep strict +# failure. Out-of-order text/reasoning arriving while a block is open is +# buffered and emitted as a TRAILING text/thinking block (ratified shapes). +# DEVIATION: _finalize_stream_failure's block-close sort is filtered to OPENED +# tools only (a still-deferred tool has no downstream block and a None index) — +# semantically inert. DEFERRED: oversized-payload tolerance (the 16 MiB cap +# fires inside the bounded reader before the event type is knowable; a type- +# aware reader would disturb the R5 EOF-tail region). Four pinned strict- +# ordering tests intentionally flipped to the tolerant contract. +# R6 (heartbeat): while the upstream is silent after message_start, the streaming +# path emits Anthropic `ping` SSE events every SHIM_PING_INTERVAL_S (env, float- +# tolerant, default 15, <=0 disables) through the SAME single-writer lock as +# emit(), so a ping can never split a partial frame. The watchdog stops before +# the terminal frames and on failure/disconnect (reaped by the request resource +# finalizer as a safety net); a failed ping write is treated as a disconnect +# (fast dead-client detection). Streaming only; /health unchanged. Terminal +# record gains max_idle_gap_ms (max upstream inter-event silence, ms). +# DIAGNOSTICS: +# D1 (quota capture): DIAG_HEADER_ALLOWLIST gains prefix matching (x-codex-*) +# plus exact x-oai-request-id; a chatgpt-lane 2xx now emits ONE grep-stable +# event=quota_snapshot line of numeric/enum fields (plan/limit, primary+ +# secondary used-percent/window-minutes/reset-after, credits flags/balance; +# absent -> "-", values machine-field-encoded, never free text). openai lane +# and every non-2xx unchanged (no line). +# D2 (terminal enrichment): additive-only terminal fields — unknown_events/ +# unknown_items, populated backend_type/backend_code on in-band failures, +# retry_delay_source, max_idle_gap_ms. All existing fields unchanged. +# D5 (400 diagnosis): a backend-classified invalid_request_error emits one +# request-correlated event=request_shape line of sorted top-level request key +# NAMES (+ text/reasoning sub-key names) — names only, never values. +# D3/D4 (lifecycle honesty + --auto footgun): implemented in start_shim.sh by a +# parallel dispatch (supervisor.state running|gave_up_storm|stopped surfaced by +# --status; an unrecognized non-empty DAAF_PROVIDER_SHIM warns once and still +# exits 0). See start_shim.sh; not in this file. +# R1 ALLOWLIST-SYNC OBLIGATION: the R1 unknown-event/item counters fire only for +# types outside _KNOWN_EVENT_TYPES/_KNOWN_ITEM_TYPES; those allowlists MUST stay +# in sync with the R3 reducer's handled/skippable event sets or "0 when clean" +# regresses (A1b tripwire tests guard this). +# ENVIRONMENTAL NOTE: shim STATE under /daaf (the repo/volume) is install-shared +# across containers on the same mount, while OAuth auth under $HOME/.claude +# (CODEX_HOME) is per-install — a token refresh in one install does not +# propagate, so concurrent installs must not be assumed to share auth. +# SCOPE: no auth-layer changes (the OAuth/token-delegation direction is the +# BindMounts session's Tier 3). Additive-only logging; scrubber discipline +# preserved. INTENTIONAL TEST UPDATES: 6 total (2 R5-region + 4 R3-region flips). +# SHIM_VERSION -> 1.2.14. # v1.2.13 (2026-07-19): System-role message admission (live-outage fix). Current # Claude Code appends a role:"system" message inside `messages`; the v1.2.12 # request-validation role check (user/assistant only) and the translator's @@ -634,6 +728,7 @@ import os import sys import json +import re import uuid import time import random @@ -648,7 +743,7 @@ import uvicorn # --- Config --- -SHIM_VERSION = "1.2.13" +SHIM_VERSION = "1.2.14" SHIM_SERVICE_ID = "daaf-anthropic-openai-shim" SHIM_PORT = int(os.environ.get("SHIM_PORT", "4141")) @@ -812,6 +907,17 @@ def _resolve_startup_verbosity(): BACKOFF_BASE = 0.5 # seconds; doubles each attempt BACKOFF_CAP = 8.0 # seconds; ceiling before jitter RETRY_AFTER_CAP = 30.0 # seconds; never honor an absurd Retry-After +# v1.2.14 (R4): rate-limit-class retries may honor a longer advertised/parsed delay +# than other classes (a genuine per-minute rate window can exceed 30s), but a delay +# beyond THIS cap is failed fast to the client rather than slept internally, so the +# client owns any multi-minute wait instead of a silent shim stall. Non-rate-limit +# classes keep RETRY_AFTER_CAP. +RATE_LIMIT_RETRY_AFTER_CAP = 60.0 # seconds +# v1.2.14 (R4): Codex-style rate-limit delay hint embedded in the backend message +# text. Parsed for the NUMBER only (gated on code rate_limit_exceeded); the prose is +# never logged or reflected. "s"/"seconds" -> seconds, "ms" -> milliseconds. +_RATE_LIMIT_DELAY_RE = re.compile( + r"try again in (\d+(?:\.\d+)?)\s*(s|ms|seconds?)", re.IGNORECASE) # Backend-error diagnostics distinguish status plus exact structured type/code # (for example insufficient_quota versus rate_limit_exceeded) and retain only @@ -822,6 +928,23 @@ def _resolve_startup_verbosity(): # payloads, so 16 MiB is deliberately generous while still preventing an unbounded # upstream line from exhausting shim memory. The raw SSE transcript is never stored. MAX_RESPONSES_SSE_EVENT_BYTES = 16 * 1024 * 1024 +# v1.2.14 (F3): cap the ≥400 error body handed to the retry classifier at 1 MiB. A +# legitimate backend error envelope is tiny; a pathological/oversized body must not +# drive an unbounded json.loads/scan in _plan_retry. Beyond-cap bodies are truncated +# and parse-fail on the truncated bytes, falling back to HTTP-status classification. +MAX_ERROR_BODY_BYTES = 1024 * 1024 + +# v1.2.14 (R6): downstream heartbeat interval in seconds. After message_start is on +# the wire, the streaming path emits Anthropic `ping` SSE events every +# SHIM_PING_INTERVAL_S while the upstream is silent, so an intermediary/client keeps +# the connection warm and a dead client is detected within one interval instead of +# only at the next upstream event. Float-tolerant: an unparseable value falls back to +# the 15s default; a value <= 0 disables the heartbeat entirely (no watchdog task is +# started). Streaming responses only — non-streaming paths never start a watchdog. +try: + SHIM_PING_INTERVAL_S = float(os.environ.get("SHIM_PING_INTERVAL_S", "15")) +except (ValueError, TypeError): + SHIM_PING_INTERVAL_S = 15.0 # v1.2.11 request-local lifecycle accounting. The mutable record is intentionally # stored as one ContextVar value: child tasks inherit the same record, while @@ -841,6 +964,93 @@ def _resolve_startup_verbosity(): _WIRE_DIVERGENCE_LAST_EMITTED = {} _WIRE_DIVERGENCE_LOCK = threading.Lock() +# v1.2.14 (R1): forward-compatible observability for wire shapes the shim does not +# model. Bounded process-level aggregation keyed by (kind, name) with kind in +# {event_type, item_type, envelope}. The cap bounds distinct keys; once reached, +# further novel keys collapse into (kind, "__overflow__") so a hostile or highly +# variable backend cannot grow the map without limit. Lock-guarded independently +# of the wire-divergence lock it forwards into. +_UNKNOWN_WIRE_COUNTS = {} +_UNKNOWN_WIRE_CAP = 256 +_UNKNOWN_WIRE_LOCK = threading.Lock() + +# Event types the shim either explicitly handles or benignly ignores. A clean +# stream contains ONLY these, so the reducer/accumulator catch-alls can count any +# event type outside this set as genuinely unknown wire (R1) while keeping +# unknown_events at 0 for well-behaved backends. Kept in sync with the streaming +# reducer in _handle_messages and _accumulate_terminal_response: the first group +# is actively translated; the second is known status/lifecycle scaffolding the +# reducer skips (they still reach the catch-all today and must not be miscounted). +_KNOWN_EVENT_TYPES = frozenset({ + # Actively handled by the reducer. + "response.reasoning_summary_text.delta", + "response.output_text.delta", + "response.output_item.added", + "response.output_item.done", + "response.function_call_arguments.delta", + "response.function_call_arguments.done", + "response.completed", + "response.incomplete", + "response.failed", + "error", + # Known status/lifecycle events the reducer legitimately ignores. + "response.created", + "response.in_progress", + "response.queued", + "response.content_part.added", + "response.content_part.done", + "response.output_text.done", + "response.reasoning_summary_text.done", + "response.reasoning_summary_part.added", + "response.reasoning_summary_part.done", + "response.reasoning_text.delta", + "response.reasoning_text.done", +}) +# output_item.added/done item types the reducer models. Any other item type is +# counted as unknown_items (R1) — observability only; it is still not translated. +_KNOWN_ITEM_TYPES = frozenset({"function_call", "message", "reasoning"}) + +# v1.2.14 (R3.5): non-load-bearing event types whose FRAME may be tolerated when it +# fails strict JSON parse. These are pure status/lifecycle scaffolding the reducer +# already ignores when well-formed, so a malformed instance carries nothing the +# translation needs — it is counted (R1) and skipped rather than failing the stream. +# A load-bearing event type (deltas, tool-argument events, terminal frames, error) +# is deliberately absent, so a malformed instance of one still fails strictly. This +# is a subset of the ignored group in _KNOWN_EVENT_TYPES and must stay consistent +# with it. Tolerance is malformed-JSON only; an oversized payload is capped inside +# the bounded SSE reader before the type is knowable and is deferred (see changelog). +_MALFORMED_TOLERANT_EVENT_TYPES = frozenset({ + "response.created", + "response.in_progress", + "response.queued", + "response.content_part.added", + "response.content_part.done", + "response.reasoning_summary_part.added", + "response.reasoning_summary_part.done", +}) +# Extract a clean, bounded, unescaped `"type":"..."` token from the RAW bytes of an +# SSE frame that failed strict JSON parse. Only an ASCII type of 1-64 chars with no +# quote/backslash qualifies; anything else yields None (strict failure preserved). +_MALFORMED_TYPE_PROBE = re.compile(rb'"type"\s*:\s*"([^"\\]{1,64})"') + + +def _probe_malformed_event_type(data_bytes): + # v1.2.14 (R3.5): best-effort event-type recovery from a malformed SSE frame. + # INTENT: decide whether a frame that failed json.loads is a tolerable + # non-load-bearing status event. Probes only the first 4 KiB so a large junk + # frame cannot drive an unbounded scan. Returns the type string or None. + # REASONING: a regex over the raw bytes never executes the malformed JSON and + # cannot be tricked into recovering a load-bearing type via escapes (the token + # class excludes quote and backslash). The caller checks membership in + # _MALFORMED_TOLERANT_EVENT_TYPES; a non-match falls through to strict failure. + match = _MALFORMED_TYPE_PROBE.search(data_bytes[:4096]) + if not match: + return None + try: + return match.group(1).decode("ascii") + except (UnicodeDecodeError, AttributeError): + return None + class _RequestLifecycleState: def __init__(self): @@ -859,10 +1069,35 @@ def __init__(self): self.effort_value = "-" self.effort_source = "-" self.reasoning_cache_misses = 0 + # v1.2.14 (R1/D2): per-request counts of unknown SSE event types and + # unmodeled output_item types, surfaced on the terminal record (0 when clean). + self.unknown_events = 0 + self.unknown_items = 0 + # v1.2.14 (R6/D2): maximum upstream inter-event idle gap in whole + # milliseconds (monotonic clock), surfaced on the terminal record for stall + # triage. last_upstream_event_at holds the monotonic timestamp of the previous + # upstream event yield; the first event has no predecessor and therefore + # contributes no gap (it only seeds the timer). + self.max_idle_gap_ms = 0 + self.last_upstream_event_at = None + # v1.2.14 (R6): the downstream heartbeat watchdog task for a streaming + # response (None until message_start is emitted; torn down before the terminal + # frames and reaped by the request-level resource finalizer as a safety net). + self.heartbeat_task = None + # v1.2.14 (D5): sorted top-level request key names (+ text/reasoning sub-key + # names) captured once after translation, emitted once if the backend + # classifies the request as invalid_request_error. Names only, never values. + self.request_shape = None + self.invalid_request_shape_logged = False self.attempts = 0 self.retries = 0 self.last_retry_reason = "-" self.last_retry_source = "-" + # v1.2.14 (R4/D2): source of the delay used for the most recent retry sleep + # (parsed = backend message "try again in" text; header = Retry-After; backoff + # = local exponential backoff). Stays "-" when no retry slept; on a fail-fast + # beyond the rate-limit cap it records the advertised source that triggered it. + self.retry_delay_source = "-" self.client_request_id = self.req_id self.transport_exception = "-" self.upstream_request_id = "-" @@ -974,6 +1209,16 @@ def _record_upstream_headers(response): upstream_req_id_header=state.upstream_request_id_header, upstream_req_id=state.upstream_request_id, ) + # v1.2.14 (D1): on the chatgpt lane a 2xx carries the full x-codex-* subscription + # quota surface. Emit ONE compact, grep-stable quota_snapshot line of numeric/enum + # fields for subscription-window triage. The openai lane (no x-codex-*) and every + # non-2xx response are unchanged — no line is emitted. + if SHIM_BACKEND_MODE == "chatgpt" and 200 <= response.status_code < 300: + snapshot = {} + for field, header_name in _QUOTA_SNAPSHOT_HEADER_FIELDS: + value = response.headers.get(header_name) + snapshot[field] = value if value is not None else "-" + _lifecycle_event("quota_snapshot", **snapshot) def _record_upstream_first_event(): @@ -985,6 +1230,28 @@ def _record_upstream_first_event(): _lifecycle_event("upstream_first_event") +def _record_upstream_event_gap(): + # v1.2.14 (R6): timestamp each dispatched upstream SSE event and track the maximum + # inter-event idle gap (monotonic clock) for stall triage on the terminal record. + # INTENT: give the terminal record a single max_idle_gap_ms number that summarizes + # the longest silent stretch between upstream events, without storing a + # transcript or per-event timing. + # REASONING: the first event has no predecessor, so it only seeds the timer; every + # later event updates the running maximum with the elapsed span since the prior + # event when it is larger. Purely additive telemetry — it never alters the + # control flow of the bounded SSE reader that calls it at each yield point. + # ASSUMES: called once per yielded upstream event, on the same request context. + state = _request_state() + if state is None: + return + now = time.monotonic() + if state.last_upstream_event_at is not None: + gap_ms = int((now - state.last_upstream_event_at) * 1000) + if gap_ms > state.max_idle_gap_ms: + state.max_idle_gap_ms = gap_ms + state.last_upstream_event_at = now + + def _record_downstream_first_content(): state = _request_state() if state is None or state.downstream_first_content_at is not None: @@ -1070,6 +1337,15 @@ def _record_backend_error(error_type="api_error", backend_type="-", backend_code backend_code=state.backend_code, anthropic_type=error_type, failure_phase=state.failure_phase, ) + # v1.2.14 (D5): a backend-classified invalid_request_error gets one + # request-correlated shape line to aid 400 diagnosis. Guarded by its own flag + # (independent of backend_error_logged) so it emits exactly once even when the + # backend_error line was already recorded by an earlier failure hop. Names only. + if (error_type == "invalid_request_error" + and not state.invalid_request_shape_logged + and state.request_shape is not None): + state.invalid_request_shape_logged = True + _lifecycle_event("request_shape", **state.request_shape) def _mark_success(): @@ -1124,6 +1400,8 @@ def _log_terminal_once(): attempts=state.attempts, retries=state.retries, retry_reason=state.last_retry_reason, retry_source=state.last_retry_source, + # v1.2.14 (R4/D2): additive. "-" when no retry slept. + retry_delay_source=state.retry_delay_source, transport_exception=state.transport_exception, client_req_id=state.client_request_id, upstream_req_id=state.upstream_request_id, @@ -1132,6 +1410,16 @@ def _log_terminal_once(): body_close_send=state.body_close_send, disconnect=state.disconnect_observed, disconnect_phase=state.disconnect_phase, + # v1.2.14 (D2): additive observability. unknown_events/unknown_items are 0 + # on a clean stream; backend_type/backend_code default to "-" and carry the + # extracted structural metadata on an in-band or HTTP backend failure. + unknown_events=state.unknown_events, + unknown_items=state.unknown_items, + backend_type=state.backend_type, + backend_code=state.backend_code, + # v1.2.14 (R6/D2): additive. Longest observed upstream inter-event silence in + # whole milliseconds (0 on a single-event or non-streaming response). + max_idle_gap_ms=state.max_idle_gap_ms, dur_ms=_elapsed_ms(state), ) @@ -1346,11 +1634,41 @@ async def _await_or_disconnect(operation, disconnect_event, discard_result=None, "x-ratelimit-limit-tokens", "x-ratelimit-remaining-tokens", "x-ratelimit-reset-tokens", + # v1.2.14 (D1): the chatgpt lane's request-id header (openai lane uses the + # x-*request-id family already captured for upstream correlation). + "x-oai-request-id", +) +# v1.2.14 (D1): prefix-allowlisted diagnostic header families. The live chatgpt lane +# emits a broad, evolving x-codex-* quota surface (used-percent, window-minutes, +# reset-after, plan-type, credits, per-model-family buckets) that a fixed name list +# cannot track, so it is matched by prefix. Authorization and any credential-bearing +# header remain structurally unreachable — a prefix here is never "authorization". +DIAG_HEADER_PREFIX_ALLOWLIST = ( + "x-codex-", +) +# v1.2.14 (D1): the compact chatgpt-lane subscription quota snapshot. ONLY numeric and +# enum x-codex-* fields are surfaced — never a free-text header such as a promo message +# or limit name. Each pair maps one header to one stable grep-line key; an absent header +# renders as "-". Values still pass the shared machine-field encoding, so a hostile +# value cannot break the single-token grammar. Header names are lowercased because +# httpx.Headers.get() is case-insensitive. (Live capture: notes/07 §8.) +_QUOTA_SNAPSHOT_HEADER_FIELDS = ( + ("plan_type", "x-codex-plan-type"), + ("active_limit", "x-codex-active-limit"), + ("primary_used_pct", "x-codex-primary-used-percent"), + ("primary_window_min", "x-codex-primary-window-minutes"), + ("primary_reset_s", "x-codex-primary-reset-after-seconds"), + ("secondary_used_pct", "x-codex-secondary-used-percent"), + ("secondary_window_min", "x-codex-secondary-window-minutes"), + ("secondary_reset_s", "x-codex-secondary-reset-after-seconds"), + ("credits_has", "x-codex-credits-has-credits"), + ("credits_balance", "x-codex-credits-balance"), + ("credits_unlimited", "x-codex-credits-unlimited"), ) # Defense in depth: even though credentials and Authorization are architecturally # excluded from logs, upstream-controlled metadata and error text receive one shared # sensitive-text pass before any truncation or lifecycle-field encoding. -import re +# (re is imported at module top for the v1.2.14 rate-limit-delay pattern.) _SK_KEY_RE = re.compile(r"(?i)\b(?:sk|rk|org|proj|sess)[-_][A-Za-z0-9_-]{8,}") _AUTH_SCHEME_RE = re.compile( r"(?i)\b((?:authorization\s*:\s*)?(?:bearer|basic))" @@ -1444,6 +1762,51 @@ def _record_wire_divergence(kind): return count +def _record_unknown_wire(kind, name): + # v1.2.14 (R1): record one observation of an unmodeled wire shape. + # INTENT: make unknown SSE event types, unmodeled output_item types, and + # unrecognized error envelopes VISIBLE instead of vanishing, without ever + # logging event bodies or values. Three effects, in order: (a) bump the bounded + # process aggregate, (b) route first/every-100th logging through the existing + # wire-divergence channel, (c) bump the per-request terminal-record counter. + # REASONING: the process map is bounded and overflow-collapsed so a novel-name + # flood cannot grow it; the name is scrubbed and length-bounded before it can + # reach a log line. The behavior is observability-only — unknown shapes are + # still not translated (there is nothing to translate them to). + # ASSUMES: kind is a source constant ("event_type"/"item_type"/"envelope"); + # only `name` is backend-influenced and therefore scrubbed. + safe_name = _sanitize_sensitive_text(name, max_len=64) if isinstance( + name, str) and name else "-" + with _UNKNOWN_WIRE_LOCK: + key = (kind, safe_name) + if key not in _UNKNOWN_WIRE_COUNTS and ( + len(_UNKNOWN_WIRE_COUNTS) >= _UNKNOWN_WIRE_CAP): + key = (kind, "__overflow__") + _UNKNOWN_WIRE_COUNTS[key] = _UNKNOWN_WIRE_COUNTS.get(key, 0) + 1 + _record_wire_divergence("unknown_%s:%s" % (kind, key[1])) + state = _request_state() + if state is not None: + if kind == "item_type": + state.unknown_items += 1 + else: + state.unknown_events += 1 + + +def _request_shape_fields(payload): + # v1.2.14 (D5): capture only the SORTED top-level request key NAMES (plus + # text/reasoning sub-key names) for a possible invalid_request_error diagnosis + # line. Names only, never values; the outbound Responses payload keys are + # shim-constructed identifiers, and _lifecycle_event scrubs each value anyway. + if not isinstance(payload, dict): + return None + fields = {"keys": ",".join(sorted(str(k) for k in payload.keys()))} + for sub in ("text", "reasoning"): + val = payload.get(sub) + if isinstance(val, dict): + fields["%s_keys" % sub] = ",".join(sorted(str(k) for k in val.keys())) + return fields + + # v1.2.0: reasoning-item cache (module-level, bounded LRU). # INTENT: the Responses API pairs a `reasoning` output item with the # function_call(s) that follow it. On store:false stateless replay the caller @@ -2523,7 +2886,14 @@ def _validate_terminal_response(event_type, response): status = response.get("status") if not isinstance(status, str) or status != expected_status: raise _ProtocolError("backend terminal event/status mismatch") - if "output" in response and not isinstance(response["output"], list): + if ( + "output" in response + and response["output"] is not None + and not isinstance(response["output"], list) + ): + # v1.2.14 (R3.3): `output: null` is a captured Codex quirk on a completed + # response after a tool turn. Tolerate it here (streamed/collected state is + # the fallback source); only a non-null, non-list output is a real violation. raise _ProtocolError("backend terminal response output is not a list") if "usage" in response: usage = response["usage"] @@ -2644,6 +3014,11 @@ def _validate_stream_event_fields(event): _require_protocol_string( event.get("delta"), "function argument delta", allow_empty=True ) + # v1.2.14 (R1): only item_id/delta are load-bearing here. Known-but-unmodeled + # extra fields (e.g. the live `obfuscation` string the Codex backend attaches + # to arguments.delta) are tolerated by design — validated-when-present is not + # required, and they are deliberately NOT counted as unknown wire (too noisy + # to flag a benign per-delta field). elif event_type == "response.function_call_arguments.done": _require_protocol_string(event.get("item_id"), "function item id") # v1.2.8 wire tolerance: the public API documents name/arguments on this @@ -2867,6 +3242,15 @@ def _diag_headers(headers): val = headers.get(name) if val is not None: parts.append(f"{name}={_sanitize_sensitive_text(val)}") + # v1.2.14 (D1): also surface any prefix-allowlisted header (e.g. x-codex-*). Iterate + # actual headers only for the prefix rule so the exact-match ordering above is + # unchanged; skip a name already emitted by the exact loop so it is never doubled. + for name, val in headers.items(): + lname = name.lower() + if val is None or lname in DIAG_HEADER_ALLOWLIST: + continue + if lname.startswith(DIAG_HEADER_PREFIX_ALLOWLIST): + parts.append(f"{lname}={_sanitize_sensitive_text(val)}") return " ".join(parts) if parts else "(none)" @@ -2909,6 +3293,118 @@ def _retry_delay(attempt, retry_after): return random.uniform(0, base) +def _parse_rate_limit_delay_seconds(message_text): + # v1.2.14 (R4): extract the numeric "try again in " hint from a backend + # rate-limit message. Returns float seconds or None. The prose is inspected + # locally ONLY and is never logged, reflected, or persisted. + if not isinstance(message_text, str) or not message_text: + return None + match = _RATE_LIMIT_DELAY_RE.search(message_text) + if not match: + return None + try: + value = float(match.group(1)) + except (ValueError, TypeError): + return None + if value < 0: + return None + return value / 1000.0 if match.group(2).lower() == "ms" else value + + +def _numeric_retry_after_seconds(retry_after): + # Parse a Retry-After header value as non-negative seconds, else None (an + # HTTP-date form is intentionally treated as absent -> fall through to backoff). + if retry_after is None: + return None + try: + value = float(retry_after) + except (ValueError, TypeError): + return None + return value if value >= 0 else None + + +def _retry_signals_from_body(raw_body): + # v1.2.14 (R4): classify a backend error body for retry gating. Returns + # (backend_code, backend_type, code_norm, message_text, signal_recognized). + # message_text is retained ONLY for the internal rate-limit delay regex. + try: + payload = json.loads(raw_body) if raw_body else {} + except (ValueError, TypeError, UnicodeDecodeError): + payload = {} + backend_type, backend_code, _recognized, _param = _backend_error_fields(payload) + code_norm = backend_code.strip().lower() if backend_code != "-" else "" + type_norm = backend_type.strip().lower() if backend_type != "-" else "" + root = payload if isinstance(payload, dict) else {} + nested = root.get("error") if isinstance(root.get("error"), dict) else root + message = nested.get("message") if isinstance(nested, dict) else None + message_text = message if isinstance(message, str) else "" + signal_recognized = bool( + (code_norm and code_norm in _BACKEND_SIGNAL_CLASSIFICATION) + or (type_norm and type_norm in _BACKEND_SIGNAL_CLASSIFICATION) + ) + return backend_code, backend_type, code_norm, message_text, signal_recognized + + +def _plan_retry(status, retry_after_header, raw_body, attempt): + # v1.2.14 (R4): classification-driven retry decision + delay selection for one + # non-2xx attempt. Returns (should_retry, delay_seconds, delay_source). + # INTENT: retry iff the classifier marks the failure retryable when a recognized + # backend code/type is present; otherwise reproduce today's bare-status behavior + # (RETRY_STATUSES, plus 408 as standard-retryable). A recognized deterministic + # code OVERRIDES the status (e.g. insufficient_quota-coded 429 fails fast). + # REASONING: delay precedence per attempt is (1) parsed rate-limit message hint + # (gated on code rate_limit_exceeded), (2) Retry-After header, (3) local backoff. + # Rate-limit-class delays cap at 60s; a parsed/advertised delay beyond the cap + # fails fast (should_retry False) so the client owns the long wait rather than a + # silent shim stall. Non-rate-limit classes keep the 30s clamp. + # ASSUMES: caller has already handled the chatgpt lazy-401 path; message prose is + # never logged (only its parsed number is used). + backend_code, backend_type, code_norm, message_text, signal_recognized = \ + _retry_signals_from_body(raw_body) + anthropic_type, _client_message, retryable = _classify_backend_error( + status, backend_code, backend_type) + if signal_recognized: + should_retry = retryable + else: + # 408 is standard-retryable (retries up to MAX_RETRIES) — the once-only + # special case in the design was judged not worth its bookkeeping. + should_retry = (status in RETRY_STATUSES) or (status == 408) + if not should_retry or attempt >= MAX_RETRIES: + return False, 0.0, "-" + is_rate_limit_class = (anthropic_type == "rate_limit_error") + delay = None + source = "backoff" + if code_norm == "rate_limit_exceeded": + parsed = _parse_rate_limit_delay_seconds(message_text) + if parsed is not None: + delay, source = parsed, "parsed" + if delay is None: + header_delay = _numeric_retry_after_seconds(retry_after_header) + if header_delay is not None: + delay, source = header_delay, "header" + if delay is None: + base = min(BACKOFF_BASE * (2 ** attempt), BACKOFF_CAP) + return True, random.uniform(0, base), "backoff" + cap = RATE_LIMIT_RETRY_AFTER_CAP if is_rate_limit_class else RETRY_AFTER_CAP + if delay > cap: + if is_rate_limit_class: + # Fail fast: do not sleep a multi-minute wait internally. Record the + # advertised source so the terminal record explains the fast-fail. + return False, 0.0, source + delay = cap + return True, delay, source + + +def _record_retry_delay_source(source): + # v1.2.14 (R4): persist the selected delay source on the request record (both a + # taken retry and a fail-fast-beyond-cap set it; a plain non-retryable keeps "-"). + if source == "-": + return + state = _request_state() + if state is not None: + state.retry_delay_source = source + + async def _post_with_retry(url, headers, payload, disconnect_event): # HARDENING: non-streaming POST with bounded retry on transient errors. # Each potentially blocking POST/sleep races this response's disconnect event; @@ -2949,20 +3445,30 @@ async def _post_with_retry(url, headers, payload, disconnect_event): phase="upstream_request") raise _record_upstream_headers(r) - if r.status_code in RETRY_STATUSES and attempt < MAX_RETRIES: - delay = _retry_delay(attempt, r.headers.get("retry-after")) - _record_retry("status_%d" % r.status_code) - # Retry diagnostics retain status, attempt, delay, and allowlisted headers - # only. Backend body prose is neither needed for retry policy nor safe to - # persist because it can reflect request-derived content. - log.warning( - "backend %d (attempt %d/%d), retrying in %.2fs | headers: %s", - r.status_code, attempt + 1, MAX_RETRIES + 1, delay, - _diag_headers(r.headers), - ) - await r.aclose() - await _await_or_disconnect(asyncio.sleep(delay), disconnect_event) - continue + if r.status_code >= 400 and attempt < MAX_RETRIES: + # v1.2.14 (R4): classify the error body ONCE (for gating + delay source), + # then decide. r.content is already buffered (non-streaming POST); the + # prose is parsed in memory for structural signals and never logged. + should_retry, delay, delay_source = _plan_retry( + r.status_code, r.headers.get("retry-after"), r.content, attempt) + if should_retry: + _record_retry_delay_source(delay_source) + _record_retry("status_%d" % r.status_code) + # Retry diagnostics retain status, attempt, delay, delay-source, and + # allowlisted headers only. Backend body prose is neither needed for + # retry policy nor safe to persist (it can reflect request content). + log.warning( + "backend %d (attempt %d/%d), retrying in %.2fs (source=%s) | headers: %s", + r.status_code, attempt + 1, MAX_RETRIES + 1, delay, + delay_source, _diag_headers(r.headers), + ) + await r.aclose() + await _await_or_disconnect(asyncio.sleep(delay), disconnect_event) + continue + # Not retrying: a non-retryable code (e.g. insufficient_quota-coded 429), + # a bare status outside RETRY_STATUSES, or a rate-limit delay beyond cap + # (fail fast). Record the advertised source when the cap forced the choice. + _record_retry_delay_source(delay_source) return r, attempt # Unreachable in practice (loop returns or raises), but keeps intent explicit. if last_exc: @@ -3008,6 +3514,22 @@ async def _settle_request_resources(state): except asyncio.CancelledError as cancellation: owner_cancellation = cancellation + # v1.2.14 (R6): the downstream heartbeat watchdog is normally stopped before the + # terminal frames; this is the request-teardown safety net for the paths that + # return without a terminal (mid-stream disconnect, early exit). Settled exactly + # like the disconnect watcher. + heartbeat = state.heartbeat_task + state.heartbeat_task = None + if heartbeat is not None: + kind, outcome, heartbeat_cancellation = await _settle_owned_task( + heartbeat, cancel=True) + if kind == "exception": + _record_cleanup_result(False, type(outcome).__name__) + else: + _record_cleanup_result(True) + if heartbeat_cancellation is not None: + owner_cancellation = heartbeat_cancellation + watcher = state.disconnect_watcher state.disconnect_watcher = None if watcher is not None: @@ -3108,13 +3630,13 @@ async def _open_backend_stream(url, headers, payload, disconnect_event): # preserve the retry budget and reconnect before semantic content. continue - if resp.status_code in RETRY_STATUSES and attempt < MAX_RETRIES: - delay = _retry_delay(attempt, resp.headers.get("retry-after")) - _record_retry("status_%d" % resp.status_code) + if resp.status_code >= 400 and attempt < MAX_RETRIES: + # v1.2.14 (R4): read the error body ONCE for classification-driven gating + # and delay-source selection. The body is parsed in memory for structural + # signals only (never decoded into a log line). On disconnect the read is + # cancelled and the owned context is settled before propagating. try: - # Drain for connection-pool behavior, but never decode or retain the - # free-form backend body: it can reflect request-derived content. - await _await_or_disconnect( + raw_err = await _await_or_disconnect( resp.aread(), disconnect_event, ) @@ -3122,20 +3644,38 @@ async def _open_backend_stream(url, headers, payload, disconnect_event): await _settle_stream_context(stream_cm) raise except Exception: - pass - await _settle_stream_context(stream_cm) - attempt += 1 - retry_count = attempt - # Keep the structural diagnostic adjacent to the raced delay. The - # real-subprocess regression uses the line plus a bounded scheduler turn - # as evidence that the target phase is retry sleep, not header acquisition. - log.warning( - "backend stream %d (attempt %d/%d), retrying in %.2fs | headers: %s", - resp.status_code, attempt, MAX_RETRIES + 1, delay, - _diag_headers(resp.headers), - ) - await _await_or_disconnect(asyncio.sleep(delay), disconnect_event) - continue + raw_err = b"" + # v1.2.14 (F3): bound the body handed to the classifier at MAX_ERROR_BODY_BYTES + # (1 MiB). resp.aread() caches the full body on resp for the caller's error + # re-read (comment below), so this truncation caps only the structural + # retry-classification parse — a pathologically large error body cannot drive + # an unbounded json.loads/scan. Beyond-cap bodies parse-fail on the truncated + # bytes and fall back to HTTP-status classification (RETRY_STATUSES). + if len(raw_err) > MAX_ERROR_BODY_BYTES: + raw_err = raw_err[:MAX_ERROR_BODY_BYTES] + should_retry, delay, delay_source = _plan_retry( + resp.status_code, resp.headers.get("retry-after"), raw_err, attempt) + if should_retry: + _record_retry_delay_source(delay_source) + _record_retry("status_%d" % resp.status_code) + await _settle_stream_context(stream_cm) + attempt += 1 + retry_count = attempt + # Keep the structural diagnostic adjacent to the raced delay. The + # real-subprocess regression uses the line plus a bounded scheduler + # turn as evidence that the phase is retry sleep, not header acquisition. + log.warning( + "backend stream %d (attempt %d/%d), retrying in %.2fs (source=%s) | headers: %s", + resp.status_code, attempt, MAX_RETRIES + 1, delay, + delay_source, _diag_headers(resp.headers), + ) + await _await_or_disconnect(asyncio.sleep(delay), disconnect_event) + continue + # Not retrying: fall through to return this final non-2xx response. Its + # body is already cached on resp, so the caller's error path re-reads it + # without a second network round-trip. Record the advertised delay source + # when a rate-limit cap forced the fail-fast. + _record_retry_delay_source(delay_source) # Transfer ownership to the request lifecycle before this tuple is visible to # caller code. The outer ASGI finalizer can therefore settle the context even if @@ -3352,55 +3892,131 @@ def _backend_error_fields(payload): # arbitrary prose is safe, so the durable boundary is structural extraction only. # ASSUMES: exact type/code strings remain approved bounded metadata and are scrubbed # by lifecycle/log serializers before persistence. + # v1.2.14 (R2): recognize the four observed envelope shapes and report whether + # the payload was a structured error envelope at all. The four shapes: + # 1. {"error": {type, code, message, param}} (openai lane, captured) + # 2. {"status": N, "error": {…}} (chatgpt lane, captured) + # 3. flat root {code, message, param} (documented in-stream error) + # 4. {"detail": "…"} (Codex model-rejection, survey) + # Shape 2's top-level `status` is metadata only — the HTTP status is passed + # separately to the classifier; here we just descend into root["error"]. root = payload if isinstance(payload, dict) else {} nested = root.get("error") if isinstance(root.get("error"), dict) else root if nested is root and isinstance(root.get("incomplete_details"), dict): nested = root["incomplete_details"] backend_type = nested.get("type") if isinstance(nested.get("type"), str) else "-" backend_code = nested.get("code") if isinstance(nested.get("code"), str) else "-" + param = nested.get("param") if isinstance(nested.get("param"), str) else "-" message = nested.get("message") if isinstance(message, dict): if backend_type == "-" and isinstance(message.get("type"), str): backend_type = message["type"] if backend_code == "-" and isinstance(message.get("code"), str): backend_code = message["code"] - return backend_type, backend_code + # "recognized" is about SHAPE, not whether the type/code VALUE is modeled: an + # unknown-valued type is still a recognized error envelope. A payload is + # recognized when it carries any structured error field (type/code/param), a + # string message, a Codex-style string `detail` (shape 4, code-less), or + # incomplete_details. Truly shapeless payloads ({} from an unparseable body, or + # a dict with no error-ish fields) are unrecognized and counted via R1. + recognized = ( + backend_type != "-" + or backend_code != "-" + or param != "-" + or isinstance(nested.get("message"), str) + or isinstance(root.get("detail"), str) + or isinstance(root.get("incomplete_details"), dict) + ) + return backend_type, backend_code, recognized, param + + +def _status_fallback_classification(status): + # v1.2.14 (R2) pure HTTP-status fallback, used only when no recognized backend + # code/type is present. Type mapping per the R2 design table. Two deliberate + # points: (1) other-4xx -> invalid_request_error (non-retryable) is the + # intentional change from the prior retryable api_error, so a deterministic + # rejection is not client-retried; (2) overloaded_error is reserved for the + # recognized codes server_is_overloaded/slow_down and status 529 — a bare 503 + # with an UNKNOWN code falls to the "unknown code + 5xx -> api_error" row (design + # table row 9), which also preserves the v1.2.10 pinned-status behavior. The + # client MESSAGE keeps the v1.2.11 fixed-message rule unchanged. + if status in _BACKEND_ERROR_MESSAGE_BY_STATUS: + client_message = _BACKEND_ERROR_MESSAGE_BY_STATUS[status] + elif status >= 500: + client_message = "backend server error" + else: + client_message = "backend request failed" + if status == 401: + return "authentication_error", client_message, False + if status == 429: + return "rate_limit_error", client_message, True + if status == 529: + return "overloaded_error", client_message, True + if status == 408: + return "api_error", client_message, True + if 400 <= status < 500: + return "invalid_request_error", client_message, False + return "api_error", client_message, True + + +# v1.2.14 (R2): recognized backend code/type -> (anthropic_type, message, retryable). +# Ordered checks inside the classifier; a code match beats a type match, and both +# beat the HTTP status. Fixed messages only (v1.2.11 no-prose rule). +_BACKEND_SIGNAL_CLASSIFICATION = { + "context_length_exceeded": ( + "invalid_request_error", "backend context length exceeded", False), + "insufficient_quota": ( + "invalid_request_error", "backend quota or plan limit reached", False), + "usage_not_included": ( + "invalid_request_error", "backend quota or plan limit reached", False), + "invalid_prompt": ("invalid_request_error", "backend rejected the request", False), + "bio_policy": ("invalid_request_error", "backend rejected the request", False), + "cyber_policy": ("invalid_request_error", "backend rejected the request", False), + "token_invalidated": ("authentication_error", "backend authentication failed", False), + "rate_limit_exceeded": ("rate_limit_error", "backend rate limit exceeded", True), + "server_is_overloaded": ("overloaded_error", "backend overloaded", True), + "slow_down": ("overloaded_error", "backend overloaded", True), + "server_error": ("api_error", "backend server error", False), +} def _classify_backend_error(status, backend_code, backend_type): - # INTENT: map status > code > type to an Anthropic type and a fixed client message. - # REASONING: fixed classification-derived text preserves useful failure families - # without reflecting any free-form backend response prose to clients or logs. - # ASSUMES: callers without a real non-2xx status pass None; unknown structured - # values keep the historical api_error mapping and use one fixed fallback. - if status is not None and status >= 400: - anthropic_type = _anthropic_error_type_for_status(status) - if status in _BACKEND_ERROR_MESSAGE_BY_STATUS: - client_message = _BACKEND_ERROR_MESSAGE_BY_STATUS[status] - elif status >= 500: - client_message = "backend server error" - else: - client_message = "backend request failed" - return anthropic_type, client_message - + # v1.2.14 (R2): map a backend failure to (anthropic_type, client_message, + # retryable) with CODE/TYPE precedence over HTTP status. + # INTENT: one classifier for every failure path (pre-stream HTTP, in-band error, + # response.failed, chatgpt inbound-nonstream). Returns a retryable hint. + # REASONING: known code/type first (most specific), else HTTP status, else the + # historical bodyless api_error/retryable fallback. Code beats type (mirrors the + # captured backend precedence, e.g. code=server_error over type=context_length). + # RETRYABLE: COMPUTED here but NOT yet consumed — A1 leaves the retry loops + # (RETRY_STATUSES-gated in _post_with_retry/_open_backend_stream) byte-identical; + # A2 rewires gating to consume this flag. Fixed messages preserve the v1.2.11 + # no-prose rule. code_norm = backend_code.strip().lower() if backend_code != "-" else "" type_norm = backend_type.strip().lower() if backend_type != "-" else "" - if code_norm == "context_length_exceeded": - return "invalid_request_error", "backend context length exceeded" - if code_norm == "server_error": - return "api_error", "backend server error" - if type_norm == "context_length_exceeded": - return "invalid_request_error", "backend context length exceeded" - if type_norm == "server_error": - return "api_error", "backend server error" - return "api_error", "backend request failed" + for signal in (code_norm, type_norm): + if signal and signal in _BACKEND_SIGNAL_CLASSIFICATION: + return _BACKEND_SIGNAL_CLASSIFICATION[signal] + if status is not None and status >= 400: + return _status_fallback_classification(status) + return "api_error", "backend request failed", True def _normalize_backend_error(payload, status=None, phase=None): # Extract and persist only approved structural metadata. The client message is # selected from local constants and never from payload.message or raw body text. - backend_type, backend_code = _backend_error_fields(payload) - anthropic_type, client_message = _classify_backend_error( + backend_type, backend_code, recognized, _param = _backend_error_fields(payload) + if not recognized: + # v1.2.14 (R1): an unrecognized error envelope is counted for observability + # and then classified by HTTP status. The tag is a bounded shape descriptor, + # never payload content: "empty" for a non-dict/empty body (typically an + # unparseable one collapsed to {}), "unstructured" for a dict with no + # error-ish fields. + tag = "empty" if not isinstance(payload, dict) or not payload else "unstructured" + _record_unknown_wire("envelope", tag) + # v1.2.14 (R2): retryable is returned by the classifier but intentionally dropped + # here — A1 does not consume it; A2 threads it through to the retry loops. + anthropic_type, client_message, _retryable = _classify_backend_error( status, backend_code, backend_type) _record_backend_error( anthropic_type, backend_type=backend_type, backend_code=backend_code, @@ -3493,7 +4109,7 @@ def _consume_sse_line(raw_line, data_lines, event_size, event_open): return None, event_size, event_open -async def _iter_bounded_sse_data(resp, disconnect_event): +async def _iter_bounded_sse_data(resp, disconnect_event, terminal_seen=None): # v1.2.7 incremental SSE reader. Do not use aiter_lines(): it may allocate an # unbounded line before the caller can reject it. This parser retains only the # current line and current data event, each capped at 16 MiB, and never stores @@ -3540,9 +4156,36 @@ async def _iter_bounded_sse_data(resp, disconnect_event): raw_line, data_lines, event_size, event_open) if payload is not None: _record_upstream_first_event() + _record_upstream_event_gap() # v1.2.14 (R6) idle-gap timestamp yield payload - if line_buffer or event_open or data_lines: - raise ValueError("upstream SSE ended before a blank-line event boundary") + if not (line_buffer or event_open or data_lines): + return + # v1.2.14 (R5): a non-blank-terminated tail at EOF. If a complete, dispatchable + # event is pending (data lines were seen and any residual partial line is NOT a + # data-field continuation that would join into it), flush it as the final event. + # This recovers a fully generated terminal response whose trailing blank line a + # proxy trimmed (B7); downstream keeps its strict JSON/terminal validation, so a + # semantically bad flushed payload still fails there. Any residual non-data bytes + # after the complete event are a malformed tail: counted for observability, then + # dropped. + partial_data = _pending_sse_data_bytes(line_buffer) + if data_lines and partial_data is None: + if line_buffer: + _record_unknown_wire("event_type", "malformed_sse_tail") + payload = b"\n".join(data_lines) + data_lines.clear() + _record_upstream_first_event() + _record_upstream_event_gap() # v1.2.14 (R6) idle-gap timestamp + yield payload + return + # Otherwise the residual is a genuinely incomplete event (a partial data line, or + # a dangling field with no dispatchable data). Tolerate it ONLY once a terminal + # semantic frame has already been accepted (the response is complete, so trailing + # junk is harmless); before the terminal frame it stays a fatal framing failure. + if terminal_seen is not None and terminal_seen[0]: + _record_unknown_wire("event_type", "malformed_sse_tail") + return + raise ValueError("upstream SSE ended before a blank-line event boundary") async def _accumulate_terminal_response(resp, disconnect_event): @@ -3555,8 +4198,12 @@ async def _accumulate_terminal_response(resp, disconnect_event): # are validated in arrival order so malformed content cannot hide behind a later # superficially valid terminal object. The first valid terminal ends consumption. # Returns (terminal_response, failure_message). Exactly one is non-None. + # v1.2.14 (R5): a mutable one-slot flag lets the bounded SSE reader tolerate a + # malformed tail at EOF only after a terminal semantic frame was accepted. + terminal_seen = [False] try: - async for data_bytes in _iter_bounded_sse_data(resp, disconnect_event): + async for data_bytes in _iter_bounded_sse_data( + resp, disconnect_event, terminal_seen): if data_bytes.strip() == b"[DONE]": return None, "backend stream ended without a terminal response" try: @@ -3565,6 +4212,7 @@ async def _accumulate_terminal_response(resp, disconnect_event): return None, "backend stream contained malformed SSE JSON" etype = _validate_stream_event_fields(ev) if etype in ("response.completed", "response.incomplete"): + terminal_seen[0] = True terminal = _validate_terminal_response(etype, ev.get("response")) return terminal, None if etype == "response.failed": @@ -3588,6 +4236,12 @@ async def _accumulate_terminal_response(resp, disconnect_event): phase="upstream_stream", ) return None, message + # v1.2.14 (R1): terminal/error events above return; a non-terminal event + # reaching here is ignored (this adapter only extracts the terminal + # object). Count only genuinely unknown event types, not the known + # status/content events a normal stream interleaves before its terminal. + if etype not in _KNOWN_EVENT_TYPES: + _record_unknown_wire("event_type", etype) except httpx.HTTPError as error: _record_transport_failure( error, _transport_failure_phase(error, post_stream_start=True), False, @@ -3741,6 +4395,9 @@ async def _watch_disconnect(): state.effort_value = effort_value state.effort_source = effort_source state.reasoning_cache_misses = missing_reasoning + # v1.2.14 (D5): capture the request key-name shape once, for a possible + # backend invalid_request_error diagnosis line (emitted in _record_backend_error). + state.request_shape = _request_shape_fields(responses_payload) _lifecycle_event( "request_parsed", model=model, stream="y" if stream else "n", msgs=n_msgs, tools=n_tools, @@ -3995,8 +4652,19 @@ async def _send_connected(message): # the client. Accumulation/conversion failures after a 200 (no # backend_status) retain the original 502 api_error shape. if backend_status is not None: + # v1.2.14 (R2): surface the CLASSIFIER type (code/type precedence + # over status), set by _normalize_backend_error on the successful + # parse. Fall back to the status-only mapping only when the body + # was unparseable and no classifier type was recorded. Preserves + # the real backend HTTP status passthrough. + state = _request_state() + passthrough_type = ( + state.anthropic_error_type + if state is not None and state.anthropic_error_type != "-" + else _anthropic_error_type_for_status(backend_status) + ) await _send_json(send, backend_status, {"type": "error", "error": { - "type": _anthropic_error_type_for_status(backend_status), + "type": passthrough_type, "message": safe_message}}) else: state = _request_state() @@ -4139,6 +4807,11 @@ async def _send_connected(message): # this — notes file 05 §3c). So key tool routing state on item_id. # item_id -> lifecycle/canonical replay state for one function call. tool_state = {} + # v1.2.14 (R3): FIFO order of function_call item_ids as they were ADDED. The + # tolerant scheduler opens at most one tool_use block at a time and drains the + # rest in this order, so concurrently/interleaved-added tools still translate + # to non-overlapping Anthropic blocks in added order. + tool_added_order = [] # Every output_item.added id is single-use, regardless of item kind. Completed # item values retain canonical JSON so identical replays are idempotent while # conflicting replays fail before duplicate stop/cache effects. @@ -4158,32 +4831,91 @@ async def _send_connected(message): failure_message = "backend stream failed" # fixed local/classification text only failure_error_type = "api_error" saw_terminal_response = False + # v1.2.14 (R5): one-slot mutable flag shared with the bounded SSE reader so it + # can tolerate a malformed tail at EOF only after a terminal frame was seen. + terminal_seen = [False] failure_finalized = False input_tokens = None output_tokens = None accumulated_text = "" # for usage estimation fallback usage_estimated = False retries = 0 + # v1.2.14 (R3): out-of-order text/reasoning that arrives while a tool (or, for + # text, another text) block is open is buffered here and flushed as a TRAILING + # text/thinking block once all preceding blocks close (ratified downstream + # shapes). This keeps Anthropic content blocks strictly non-overlapping while + # tolerating the interleaved-emission wire instead of failing the stream. + deferred_text = "" + deferred_thinking = "" + + # v1.2.14 (R6): single-writer lock shared by every emit() call AND the ping + # watchdog below. Each downstream frame is written under this lock, so a + # heartbeat ping can never be interleaved into the middle of a partial content + # frame — the writer discipline stays exactly single-writer. + emit_lock = asyncio.Lock() async def emit(ev, data): state = _request_state() is_terminal = ev in ("message_stop", "error") - if is_terminal and state is not None: - state.terminal_frame_send = "attempted" - _record_downstream_first_content() - try: - await send({"type": "http.response.body", "body": _sse(ev, data), "more_body": True}) - except _ClientDisconnected: + async with emit_lock: if is_terminal and state is not None: - state.terminal_frame_send = "skipped_disconnect" - _record_disconnect("terminal_frame_send" if is_terminal else "content_send") - raise - except Exception: + state.terminal_frame_send = "attempted" + _record_downstream_first_content() + try: + await send({"type": "http.response.body", "body": _sse(ev, data), "more_body": True}) + except _ClientDisconnected: + if is_terminal and state is not None: + state.terminal_frame_send = "skipped_disconnect" + _record_disconnect("terminal_frame_send" if is_terminal else "content_send") + raise + except Exception: + if is_terminal and state is not None: + state.terminal_frame_send = "write_failed" + raise if is_terminal and state is not None: - state.terminal_frame_send = "write_failed" - raise - if is_terminal and state is not None: - state.terminal_frame_send = "send_completed" + state.terminal_frame_send = "send_completed" + + async def _heartbeat_loop(): + # v1.2.14 (R6): keep the downstream SSE connection warm during upstream + # silence and detect a dead client within one interval. + # INTENT: emit an Anthropic `ping` event every SHIM_PING_INTERVAL_S once + # message_start is on the wire, through emit() (hence emit_lock) so a ping + # shares the single-writer discipline and never splits a partial frame. + # REASONING: a failed ping write is positive evidence the client is gone. + # emit() has already recorded the disconnect; we set disconnect_event so + # the main reader loop tears down promptly (the fast dead-client-detection + # benefit) and then stop pinging. CancelledError (normal teardown at the + # terminal/failure boundary) is a BaseException and is intentionally NOT + # caught by `except Exception`, so it propagates and cancels cleanly. + # ASSUMES: started only for a streaming response with SHIM_PING_INTERVAL_S + # > 0; runs in the request's inherited context so _request_state() and the + # downstream `send` resolve to this request. + while True: + await asyncio.sleep(SHIM_PING_INTERVAL_S) + try: + await emit("ping", {"type": "ping"}) + except _ClientDisconnected: + disconnect_event.set() + return + except Exception: + # A non-disconnect write failure is still evidence the client is + # gone; surface it as a disconnect and stop the heartbeat. + disconnect_event.set() + return + + async def _stop_heartbeat(): + # v1.2.14 (R6): tear the watchdog down exactly once, before the terminal + # frames (so no ping can follow message_delta/message_stop) and as the + # request-teardown safety net. Idempotent: a no-op once already stopped. + st = _request_state() + if st is None or st.heartbeat_task is None: + return + task = st.heartbeat_task + st.heartbeat_task = None + _kind, _outcome, owner_cancellation = await _settle_owned_task( + task, cancel=True) + if owner_cancellation is not None: + raise owner_cancellation def _reasoning_part_key(ev): # v1.2.6 chatgpt-only semantic identity extraction. @@ -4305,6 +5037,96 @@ async def _ensure_text_open(): "index": text_index, "content_block": {"type": "text", "text": ""}}) + async def _flush_tool_args_sanitized(st, complete_args): + # v1.2.14 (R3): sanitize-mode single input_json_delta emission for one tool + # block. The caller guarantees the block is OPEN and its args have not been + # emitted yet; the sanitize/parse/fallback logic is identical to the + # pre-R3 inline sites — only the emission point (open, not pending) moved. + try: + parsed = json.loads(complete_args or "{}") + parsed, dropped = _sanitize_tool_args(st["name"], parsed) + if dropped: + log.info("sanitize tool=%s dropped=%s", st["name"], ",".join(dropped)) + out_json = json.dumps(parsed) + except (ValueError, TypeError): + # Fail-open: unparseable args pass through verbatim — sanitization must + # never break a working tool call. + log.warning("sanitize: unparseable args for tool=%s; passing through", st["name"]) + out_json = complete_args or "{}" + await emit("content_block_delta", {"type": "content_block_delta", + "index": st["anth_index"], + "delta": {"type": "input_json_delta", "partial_json": out_json}}) + st["args_emitted"] = True + + async def _open_tool(st): + # v1.2.14 (R3): open one deferred/serialized tool_use block. anth_index is + # assigned HERE (not at output_item.added) so Anthropic block indices stay + # monotonic in EMISSION order. A serialized tool has no other tool open when + # it is added, so it opens immediately and its index/bytes equal what the + # pre-R3 single-slot reducer produced (byte-identical serialized path). + nonlocal next_index + st["anth_index"] = next_index + next_index += 1 + st["opened"] = True + await emit("content_block_start", {"type": "content_block_start", + "index": st["anth_index"], + "content_block": {"type": "tool_use", "id": st["call_id"], + "name": st["name"], "input": {}}}) + if not SHIM_SANITIZE_TOOLS: + # Flush any argument bytes buffered while this tool waited its turn as + # ONE input_json_delta. A serialized tool has nothing buffered at open + # (its deltas arrive after), so this is a no-op on the byte-identical + # path; a deferred tool flushes its whole buffer here. + pending = st["args_buf"][st["emitted_len"]:] + if pending: + await emit("content_block_delta", {"type": "content_block_delta", + "index": st["anth_index"], + "delta": {"type": "input_json_delta", "partial_json": pending}}) + st["emitted_len"] = len(st["args_buf"]) + elif st["arguments_done"] is not None and not st["args_emitted"]: + # Sanitize mode: the complete args already arrived while this tool was + # deferred — emit the single sanitized delta now that the block is open. + await _flush_tool_args_sanitized(st, st["arguments_done"][1]) + + async def _close_tool(st): + # v1.2.14 (R3): emit the terminal content_block_stop for one open tool. + await emit("content_block_stop", { + "type": "content_block_stop", "index": st["anth_index"]}) + st["closed"] = True + + async def _try_open_next_tool(): + # v1.2.14 (R3): FIFO-drain deferred tools when the active tool closes. Open + # the next not-yet-opened tool in added order; if it is ALREADY fully + # finalized (its output_item.done arrived while it waited), close it + # contiguously and continue, so a run of finalized deferrals emits + # back-to-back start->delta->stop blocks. Stop at the first tool still + # awaiting its own events — that one becomes the active open block and its + # remaining deltas/done flow through the normal handlers. + while True: + nxt = None + for iid in tool_added_order: + cand = tool_state[iid] + if not cand["opened"] and not cand["closed"]: + nxt = cand + break + if nxt is None: + return + await _open_tool(nxt) + if nxt["finalized"]: + await _close_tool(nxt) + continue + return + + async def _drain_pending_tools(): + # v1.2.14 (R3): terminal-success flush — open and close every still-deferred + # tool in added order so each becomes a complete non-overlapping tool_use + # block. Called after the single active tool (if any) has been closed. + for iid in tool_added_order: + st = tool_state[iid] + if not st["opened"] and not st["closed"]: + await _open_tool(st) + await _close_tool(st) + async def _finalize_stream_failure(message, error_type="api_error"): # v1.2.7 common post-start failure finalizer. # INTENT: leave every emitted block structurally closed, then make the @@ -4321,6 +5143,9 @@ async def _finalize_stream_failure(message, error_type="api_error"): # (mid-stream protocol/framing/transport failures with no backend status) # keeps the default api_error, preserving the v1.2.8 finalizer semantics. nonlocal text_block_open, thinking_block_open, failure_finalized + # v1.2.14 (R6): stop the heartbeat before emitting any terminal-failure + # frames so a ping can never interleave with the error/body-close sequence. + await _stop_heartbeat() _record_backend_error(error_type, phase="upstream_stream") if failure_finalized or disconnect_event.is_set(): return False @@ -4338,9 +5163,16 @@ async def _finalize_stream_failure(message, error_type="api_error"): "delta": {"type": "signature_delta", "signature": ""}}) await emit("content_block_stop", { "type": "content_block_stop", "index": thinking_index}) - for item_id in sorted(tool_state, key=lambda key: tool_state[key]["anth_index"]): + # v1.2.14 (R3): only OPENED tools have a downstream block (and a real + # anth_index) to close; a still-deferred tool never emitted a + # content_block_start, so it is skipped here. Filtering to opened tools + # also keeps the sort key from comparing a pending tool's None index. + for item_id in sorted( + (key for key in tool_state if tool_state[key]["opened"]), + key=lambda key: tool_state[key]["anth_index"], + ): st = tool_state[item_id] - if st["opened"] and not st.get("closed"): + if not st.get("closed"): st["closed"] = True await emit("content_block_stop", { "type": "content_block_stop", "index": st["anth_index"]}) @@ -4469,8 +5301,18 @@ async def _finalize_stream_failure(message, error_type="api_error"): "usage": {"input_tokens": 0, "output_tokens": 0}}}) started = True + # v1.2.14 (R6): message_start is on the wire — start the downstream + # heartbeat. A `ping` every SHIM_PING_INTERVAL_S keeps the connection warm + # during upstream silence; SHIM_PING_INTERVAL_S <= 0 disables it. This is + # the ONLY start site, so the watchdog never runs for a non-streaming + # response or for the pre-content error branch above (which returns first). + if SHIM_PING_INTERVAL_S > 0: + _hb_state = _request_state() + if _hb_state is not None: + _hb_state.heartbeat_task = asyncio.create_task(_heartbeat_loop()) + async for data_bytes in _iter_bounded_sse_data( - resp, disconnect_event + resp, disconnect_event, terminal_seen ): # HARDENING (client-disconnect mid-stream): stop pulling from the # backend the moment the client goes away. @@ -4485,6 +5327,15 @@ async def _finalize_stream_failure(message, error_type="api_error"): try: ev = json.loads(data_bytes) except (ValueError, UnicodeDecodeError): + # v1.2.14 (R3.5): a malformed frame whose recoverable type is a + # non-load-bearing status event is counted (R1) and skipped rather + # than failing the whole stream — the frame carried nothing the + # translation needs. Any other (or unrecoverable) type stays a + # strict failure, so a malformed load-bearing frame still aborts. + probed_type = _probe_malformed_event_type(data_bytes) + if probed_type in _MALFORMED_TOLERANT_EVENT_TYPES: + _record_unknown_wire("event_type", probed_type) + continue stream_failed = True failure_message = "backend stream contained malformed SSE JSON" break @@ -4503,11 +5354,13 @@ async def _finalize_stream_failure(message, error_type="api_error"): for st in tool_state.values() ) if text_block_open or unfinished_tool: - stream_failed = True - failure_message = ( - "backend emitted reasoning while a content block was open" - ) - break + # v1.2.14 (R3): tolerate out-of-order reasoning. A text or + # tool block is open, so buffer this reasoning and emit it + # as a TRAILING thinking block once those blocks close + # (ratified shape) — keeps Anthropic blocks non-overlapping + # instead of failing the stream as pre-R3 did. + deferred_thinking += delta + continue await _ensure_thinking_open() if SHIM_BACKEND_MODE == "chatgpt": # v1.2.6: synthesize a boundary only before the first @@ -4561,9 +5414,13 @@ async def _finalize_stream_failure(message, error_type="api_error"): for st in tool_state.values() ) if unfinished_tool: - raise _ProtocolError( - "backend emitted text while a tool block was open" - ) + # v1.2.14 (R3): tolerate text emitted while a tool block is + # open. Buffer it and emit as a TRAILING text block once the + # tool closes (ratified shape) — keeps Anthropic blocks + # non-overlapping instead of failing the stream as pre-R3 did. + deferred_text += delta + accumulated_text += delta + continue await _ensure_text_open() accumulated_text += delta await emit("content_block_delta", {"type": "content_block_delta", @@ -4582,50 +5439,57 @@ async def _finalize_stream_failure(message, error_type="api_error"): ) added_item_ids.add(item_id) if item.get("type") == "function_call": - # Before a tool opens, every prior tool must be fully closed. - # A second concurrent tool would overlap Anthropic blocks and - # must fail rather than silently overwrite/reroute state. - unfinished_tool = any( + if item_id in tool_state: + raise _ProtocolError("backend reused a function item id") + # v1.2.14 (R3): tolerant scheduler. A function_call added while + # another tool block is still OPEN is DEFERRED (registered + # pending, args buffered), not rejected as a protocol failure — + # it opens later via _try_open_next_tool when the active tool + # closes, producing non-overlapping Anthropic tool_use blocks in + # added order (strict-emit downstream). On a SERIALIZED wire no + # tool is ever open here, so `other_tool_open` is always False + # and this reduces to the pre-R3 immediate-open path. + other_tool_open = any( st["opened"] and not st.get("closed") for st in tool_state.values() ) - if unfinished_tool: - raise _ProtocolError( - "backend opened a tool while another tool was unfinished" - ) - if item_id in tool_state: - raise _ProtocolError("backend reused a function item id") - - # Close ALL open non-tool blocks before opening tool_use. This - # preserves strict Anthropic block serialization while still - # supporting normal text -> tool transitions. - if thinking_block_open: - await emit("content_block_delta", {"type": "content_block_delta", - "index": thinking_index, - "delta": {"type": "signature_delta", "signature": ""}}) - await emit("content_block_stop", {"type": "content_block_stop", "index": thinking_index}) - thinking_block_open = False - if text_block_open: - await emit("content_block_stop", {"type": "content_block_stop", "index": text_index}) - text_block_open = False - anth_index = next_index - next_index += 1 st = { - "anth_index": anth_index, - "opened": True, + "anth_index": None, # assigned at open (emission order) + "opened": False, + "closed": False, "call_id": item["call_id"], "name": item["name"], "args_buf": "", + "emitted_len": 0, # sanitize-off incremental cursor "arguments_done": None, + "args_emitted": False, # sanitize-on single-delta guard + "finalized": False, # output_item.done seen "completed_item": None, - "closed": False, } tool_state[item_id] = st + tool_added_order.append(item_id) saw_tool_use = True - await emit("content_block_start", {"type": "content_block_start", - "index": anth_index, - "content_block": {"type": "tool_use", "id": st["call_id"], - "name": st["name"], "input": {}}}) + if not other_tool_open: + # Serialized path: close ALL open non-tool blocks before + # opening tool_use (normal text/thinking -> tool transition), + # then open immediately. While another tool IS open, text and + # thinking are already closed, so the deferred branch has + # nothing to close and emits nothing until it drains. + if thinking_block_open: + await emit("content_block_delta", {"type": "content_block_delta", + "index": thinking_index, + "delta": {"type": "signature_delta", "signature": ""}}) + await emit("content_block_stop", {"type": "content_block_stop", "index": thinking_index}) + thinking_block_open = False + if text_block_open: + await emit("content_block_stop", {"type": "content_block_stop", "index": text_index}) + text_block_open = False + await _open_tool(st) + elif item.get("type") not in _KNOWN_ITEM_TYPES: + # v1.2.14 (R1): message/reasoning items are known and handled + # elsewhere; any other output_item type is unmodeled and + # counted for observability (still not translated). + _record_unknown_wire("item_type", item.get("type")) continue # --- function_call argument deltas (route by item_id) --- @@ -4648,10 +5512,14 @@ async def _finalize_stream_failure(message, error_type="api_error"): # in both sanitize modes. Sanitize mode still defers emission # to .done; sanitize-off still forwards incrementally. st["args_buf"] += frag - if not SHIM_SANITIZE_TOOLS: + # v1.2.14 (R3): forward incrementally only once the block is OPEN. + # A still-deferred tool just buffers (emitted_len stays behind, and + # _open_tool flushes args_buf[emitted_len:] as one delta on open). + if not SHIM_SANITIZE_TOOLS and st["opened"]: await emit("content_block_delta", {"type": "content_block_delta", "index": st["anth_index"], "delta": {"type": "input_json_delta", "partial_json": frag}}) + st["emitted_len"] = len(st["args_buf"]) continue # --- function_call arguments finalized --- @@ -4699,26 +5567,15 @@ async def _finalize_stream_failure(message, error_type="api_error"): "backend completed conflicting arguments after tool closure" ) complete_args = canonical_done[1] - if SHIM_SANITIZE_TOOLS: - # SANITIZE MODE: parse the complete string, sanitize, and - # emit as ONE input_json_delta. Identical replay is ignored - # above, so downstream sees this exact delta at most once. - try: - parsed = json.loads(complete_args or "{}") - parsed, dropped = _sanitize_tool_args(st["name"], parsed) - if dropped: - log.info("sanitize tool=%s dropped=%s", st["name"], ",".join(dropped)) - out_json = json.dumps(parsed) - except (ValueError, TypeError): - # Fail-open: unparseable args pass through verbatim — - # sanitization must never break a working tool call. - log.warning("sanitize: unparseable args for tool=%s; passing through", st["name"]) - out_json = complete_args or "{}" - await emit("content_block_delta", {"type": "content_block_delta", - "index": st["anth_index"], - "delta": {"type": "input_json_delta", "partial_json": out_json}}) - # Sanitize-off already forwarded the deltas; nothing to emit. st["arguments_done"] = canonical_done + # v1.2.14 (R3): SANITIZE MODE emits the sanitized args as ONE + # input_json_delta, but only once the block is OPEN. A still-deferred + # tool records arguments_done now; _open_tool emits the sanitized + # delta when it drains. Identical replay is ignored above, so + # downstream sees this exact delta at most once. Sanitize-off already + # forwarded (open) or buffered (deferred) the deltas — nothing to emit. + if SHIM_SANITIZE_TOOLS and st["opened"] and not st["args_emitted"]: + await _flush_tool_args_sanitized(st, complete_args) continue # --- an output item completed --- @@ -4753,6 +5610,12 @@ async def _finalize_stream_failure(message, error_type="api_error"): "backend replayed a conflicting completed output item" ) + # v1.2.14 (R1): count unmodeled completed item types for + # observability. function_call/message/reasoning are known; any + # other type is still cached below but not otherwise translated. + if item.get("type") not in _KNOWN_ITEM_TYPES: + _record_unknown_wire("item_type", item.get("type")) + # Validate all state relationships before adding the item to the # completed/cache set or emitting a fallback delta/stop. if item.get("type") == "function_call": @@ -4779,28 +5642,23 @@ async def _finalize_stream_failure(message, error_type="api_error"): "backend completed a tool after an inconsistent closure" ) - # Fallback: if no arguments.done event fired, emit the full - # item arguments once under sanitize mode. - if SHIM_SANITIZE_TOOLS and prior_done is None: - raw = item["arguments"] - try: - parsed = json.loads(raw or "{}") - parsed, dropped = _sanitize_tool_args(st["name"], parsed) - if dropped: - log.info("sanitize tool=%s dropped=%s", st["name"], ",".join(dropped)) - out_json = json.dumps(parsed) - except (ValueError, TypeError): - log.warning("sanitize: unparseable args for tool=%s; passing through", st["name"]) - out_json = raw or "{}" - await emit("content_block_delta", {"type": "content_block_delta", - "index": st["anth_index"], - "delta": {"type": "input_json_delta", "partial_json": out_json}}) if prior_done is None: st["arguments_done"] = item_done - await emit("content_block_stop", { - "type": "content_block_stop", "index": st["anth_index"]}) - st["closed"] = True + # v1.2.14 (R3): the tool is now fully finalized on the wire. + # finalized lets a still-deferred tool drain as a contiguous + # start->delta->stop block when its turn comes. + st["finalized"] = True st["completed_item"] = item + if st["opened"]: + # Sanitize-mode fallback: if arguments.done never fired, emit + # the single sanitized delta from the completed item's + # arguments now, before the block closes. + if SHIM_SANITIZE_TOOLS and not st["args_emitted"]: + await _flush_tool_args_sanitized(st, st["arguments_done"][1]) + await _close_tool(st) + # An active tool just closed — drain the next deferred tool. + await _try_open_next_tool() + # else: still deferred — it opens and closes on drain later. completed_item_values[item_id] = item completed_items.append(item) @@ -4809,10 +5667,15 @@ async def _finalize_stream_failure(message, error_type="api_error"): # --- terminal SUCCESS/TRUNCATION events --- if etype in ("response.completed", "response.incomplete"): r_obj = _validate_terminal_response(etype, ev.get("response")) - terminal_output = r_obj.get("output", []) + # v1.2.14 (R3.3): tolerate `output: null`/absent — fall back to the + # streamed/collected state (cache_items below uses completed_items + # when terminal_output_items is empty). `or []` covers both the + # null (validated-tolerated) and the missing-key cases. + terminal_output = r_obj.get("output") or [] for item in terminal_output: _validate_output_item(item) saw_terminal_response = True + terminal_seen[0] = True final_status = r_obj["status"] usage = r_obj.get("usage", {}) input_tokens = usage.get("input_tokens", input_tokens) @@ -4870,7 +5733,13 @@ async def _finalize_stream_failure(message, error_type="api_error"): ) stream_failed = True break - # Any other event type is ignored gracefully. + # v1.2.14 (R1): every handled event type above continues or breaks, + # so anything reaching here is unhandled. Known status/lifecycle + # events (created, in_progress, content_part.*, *_text.done, reasoning + # summary part boundaries) are ignored silently; a genuinely unknown + # event type is counted for observability without translation. + if etype not in _KNOWN_EVENT_TYPES: + _record_unknown_wire("event_type", etype) # v1.2.7 terminal invariant: [DONE] and clean EOF are framing signals, # never proof that generation completed. Only a parsed complete terminal @@ -4908,12 +5777,45 @@ async def _finalize_stream_failure(message, error_type="api_error"): "delta": {"type": "signature_delta", "signature": ""}}) await emit("content_block_stop", {"type": "content_block_stop", "index": thinking_index}) thinking_block_open = False - # Close any tool blocks that never received an output_item.done. - for item_id in sorted(tool_state, key=lambda k: tool_state[k]["anth_index"]): + # Close the active tool that never received an output_item.done (at most one + # is open under the R3 scheduler), then drain any still-deferred tools so + # each becomes a complete non-overlapping tool_use block in added order. + for item_id in tool_added_order: st = tool_state[item_id] if st["opened"] and not st.get("closed"): await emit("content_block_stop", {"type": "content_block_stop", "index": st["anth_index"]}) st["closed"] = True + await _drain_pending_tools() + + # v1.2.14 (R3): flush reasoning/text that arrived out of order (while a tool + # or text block was open) as TRAILING thinking/text blocks (ratified shapes). + # By construction all preceding blocks are now closed, so these open with + # fresh monotonic indices and stay non-overlapping. + if deferred_thinking: + deferred_thinking_index = next_index + next_index += 1 + await emit("content_block_start", {"type": "content_block_start", + "index": deferred_thinking_index, + "content_block": {"type": "thinking", "thinking": ""}}) + await emit("content_block_delta", {"type": "content_block_delta", + "index": deferred_thinking_index, + "delta": {"type": "thinking_delta", "thinking": deferred_thinking}}) + await emit("content_block_delta", {"type": "content_block_delta", + "index": deferred_thinking_index, + "delta": {"type": "signature_delta", "signature": ""}}) + await emit("content_block_stop", {"type": "content_block_stop", "index": deferred_thinking_index}) + deferred_thinking = "" + if deferred_text: + deferred_text_index = next_index + next_index += 1 + await emit("content_block_start", {"type": "content_block_start", + "index": deferred_text_index, + "content_block": {"type": "text", "text": ""}}) + await emit("content_block_delta", {"type": "content_block_delta", + "index": deferred_text_index, + "delta": {"type": "text_delta", "text": deferred_text}}) + await emit("content_block_stop", {"type": "content_block_stop", "index": deferred_text_index}) + deferred_text = "" # HARDENING (empty-response guard): if neither text, thinking, nor a # tool call was produced, emit an empty text block so Claude Code sees @@ -4946,6 +5848,9 @@ async def _finalize_stream_failure(message, error_type="api_error"): state.usage_source = "estimated" if usage_estimated else "backend" state.tools_called = len(tool_state) + # v1.2.14 (R6): stop the heartbeat before the terminal frames so no ping + # can be written after message_delta/message_stop. + await _stop_heartbeat() await emit("message_delta", {"type": "message_delta", "delta": {"stop_reason": stop_reason, "stop_sequence": None}, "usage": {"input_tokens": input_tokens, "output_tokens": output_tokens}}) diff --git a/scripts/provider_shim/start_shim.sh b/scripts/provider_shim/start_shim.sh index 8712e60f..3b0def9e 100755 --- a/scripts/provider_shim/start_shim.sh +++ b/scripts/provider_shim/start_shim.sh @@ -15,7 +15,9 @@ # --restart Under one lifecycle lock: stop any identity-verified generation, # wait for exit, launch one replacement, and require strict readiness. # --stop Stop the shim and its supervisor. -# --status Print running/stopped + strictly validated health JSON. +# --status Print running/stopped + strictly validated health JSON. A stop +# caused by the supervisor giving up after a restart storm is +# reported distinctly (with timestamp) via the supervisor.state file. # --help Print public action usage. # # Config (env, all optional — defaults mirror the shim's own): @@ -48,6 +50,11 @@ # # Keepalive: the supervisor restarts the shim after an unexpected exit. A guard # stops after 10 crashes in 60 seconds so a broken configuration cannot spin. +# On every lifecycle transition the supervisor records its phase +# (running|gave_up_storm|stopped) in logs/supervisor.state next to the PID file, +# so a storm give-up stays visible to --status instead of looking like a clean +# stop. The state file carries the PID file's safety conventions (symlink +# refusal, mode 0600, atomic write, cleanup on stop). # # Logging: shim stderr and supervisor records share logs/shim.log. Supervisor # records are emitted once to stderr; the supervisor's single outer redirect is @@ -74,6 +81,7 @@ readonly LOG_DIR="${SCRIPT_DIR}/logs" readonly LOG_FILE="${LOG_DIR}/shim.log" readonly PID_FILE="${LOG_DIR}/shim.pid" readonly SUP_PID_FILE="${LOG_DIR}/supervisor.pid" +readonly SUP_STATE_FILE="${LOG_DIR}/supervisor.state" readonly PGID_FILE="${LOG_DIR}/pgid" readonly STOP_FILE="${LOG_DIR}/stop.requested" readonly LOCK_DIR="${LOG_DIR}/lifecycle.lock" @@ -141,7 +149,7 @@ ensure_log_dir() { state_targets_are_safe() { local target generation - for target in "$LOG_FILE" "$PID_FILE" "$SUP_PID_FILE" "$PGID_FILE" "$STOP_FILE"; do + for target in "$LOG_FILE" "$PID_FILE" "$SUP_PID_FILE" "$SUP_STATE_FILE" "$PGID_FILE" "$STOP_FILE"; do if ! path_is_safe_file_target "$target"; then printf 'ERROR: refusing unsafe shim state target: %s\n' "$target" >&2 printf ' Fix: remove the symlink/non-regular object and retry.\n' >&2 @@ -202,6 +210,57 @@ release_log_write_lock() { rmdir "$LOG_WRITE_LOCK_DIR" 2>/dev/null || true } +# --- Supervisor state file (lifecycle status honesty) ----------------------- +write_supervisor_state() { + # Record the supervisor's lifecycle phase (running|gave_up_storm|stopped) so + # --status can distinguish a storm give-up from a clean stop or a never-started + # daemon. Written by the supervisor directly — like the PID files it also + # writes, and unlike the log records, it is not serialized under the + # log-write lock; the lifecycle lock already serializes the manager actions + # that would race with it. It carries the same safety conventions as the PID + # file: symlink/non-regular refusal before any write, mode 0600, and an atomic + # rename so --status never observes a half-written record. + # + # Shared-workspace note: in shared-workspace deployments two containers may + # share /daaf, including this state directory, so supervisor.state — exactly + # like the PID file today — is install-shared, not per-container; docs advise + # running one active pipeline at a time. + local phase="$1" record tmp + case "$phase" in + running|gave_up_storm|stopped) ;; + *) return 1 ;; + esac + path_is_safe_file_target "$SUP_STATE_FILE" || return 1 + tmp="${SUP_STATE_FILE}.tmp.${BASHPID:-$$}" + path_is_safe_file_target "$tmp" || return 1 + record="$(printf '%s %s' "$phase" "$(date -u '+%Y-%m-%d %H:%M:%S')")" + printf '%s\n' "$record" > "$tmp" || { rm -f "$tmp" 2>/dev/null || true; return 1; } + chmod 0600 "$tmp" 2>/dev/null || true + # Atomic replace: SUP_STATE_FILE was preflighted as a non-symlink, so mv + # cannot be redirected through a planted link. + mv "$tmp" "$SUP_STATE_FILE" || { rm -f "$tmp" 2>/dev/null || true; return 1; } + return 0 +} + +read_supervisor_state() { + # Emit "phasetimestamp" for a valid, non-symlink state file, or fail + # quietly. Only the three known phase tokens are honored; anything else (a + # truncated or garbage file) is treated as absent so --status falls back to + # plain "stopped" rather than trusting an unrecognized record. + local line phase rest + path_is_safe_file_target "$SUP_STATE_FILE" || return 1 + [ -f "$SUP_STATE_FILE" ] || return 1 + line="$(awk 'NR == 1 { print; exit }' "$SUP_STATE_FILE" 2>/dev/null)" || return 1 + phase="${line%% *}" + rest="${line#* }" + case "$phase" in + running|gave_up_storm|stopped) ;; + *) return 1 ;; + esac + [ "$rest" != "$line" ] || rest="" + printf '%s\t%s' "$phase" "$rest" +} + append_log_record() { local record="$1" rc=0 acquire_log_write_lock || return 1 @@ -546,7 +605,7 @@ run_supervisor() { printf '%s\n' "$$" > "$SUP_PID_FILE" || exit 1 chmod 0600 "$SUP_PID_FILE" || exit 1 - local pgid child_pid logger_pid log_pipe window_start crashes rc now + local pgid child_pid logger_pid log_pipe window_start crashes rc now gave_up_storm pgid="$(ps -o pgid= -p "$$" 2>/dev/null | tr -d ' ')" if [ "$pgid" = "$$" ]; then printf '%s\n' "$pgid" > "$PGID_FILE" || exit 1 @@ -561,6 +620,7 @@ run_supervisor() { rm -f "$STOP_FILE" 2>/dev/null || true child_pid="" logger_pid="" + gave_up_storm=0 log_pipe="${LOG_DIR}/shim.stream.$$" if [ -e "$log_pipe" ] || [ -L "$log_pipe" ]; then printf 'ERROR: refusing existing shim log-stream target: %s\n' "$log_pipe" >&2 @@ -578,6 +638,11 @@ run_supervisor() { wait "$logger_pid" 2>/dev/null || true fi rm -f "$log_pipe" "$PID_FILE" "$SUP_PID_FILE" "$PGID_FILE" 2>/dev/null || true + # A storm give-up already recorded gave_up_storm and must persist so + # --status can report it distinctly; every other exit is a clean stop. + if [ "$gave_up_storm" -eq 0 ]; then + write_supervisor_state stopped || true + fi log_line "supervisor exiting" || true } stop_supervisor_signal() { @@ -589,6 +654,7 @@ run_supervisor() { window_start="$(date +%s)" crashes=0 + write_supervisor_state running || true log_line "starting shim on port ${SHIM_PORT} backend_mode=${EXPECTED_BACKEND_MODE} version=${EXPECTED_SHIM_VERSION} (keepalive on)" while true; do @@ -623,6 +689,10 @@ run_supervisor() { crashes=$((crashes + 1)) log_line "shim exited rc=${rc} (crash ${crashes}/${storm_limit} in ${storm_window}s window)" if [ "$crashes" -ge "$storm_limit" ]; then + # Record the give-up transition and mark it so cleanup_supervisor + # preserves (rather than overwrites) this terminal state for --status. + gave_up_storm=1 + write_supervisor_state gave_up_storm || true log_line "RESTART_STORM crashes=${crashes} window_s=${storm_window} action=give_up log=${LOG_FILE}" exit 1 fi @@ -698,7 +768,7 @@ stop_processes_locked() { return 2 fi - rm -f "$PID_FILE" "$SUP_PID_FILE" "$PGID_FILE" "$STOP_FILE" 2>/dev/null || return 2 + rm -f "$PID_FILE" "$SUP_PID_FILE" "$SUP_STATE_FILE" "$PGID_FILE" "$STOP_FILE" 2>/dev/null || return 2 if [ "$stopped" -eq 1 ]; then return 0 fi @@ -763,7 +833,7 @@ do_start_locked() { return 1 fi - rm -f "$PID_FILE" "$SUP_PID_FILE" "$PGID_FILE" "$STOP_FILE" 2>/dev/null || true + rm -f "$PID_FILE" "$SUP_PID_FILE" "$SUP_STATE_FILE" "$PGID_FILE" "$STOP_FILE" 2>/dev/null || true rotate_log_if_needed || return 1 if command -v setsid >/dev/null 2>&1 && [ "${DAAF_SHIM_TEST_NO_SETSID:-0}" != "1" ]; then @@ -1045,12 +1115,47 @@ do_status() { emit_readiness_failure return 1 fi + # A supervisor that exhausted its restart-storm budget exits and removes its + # PID files, leaving a state otherwise indistinguishable from "never started". + # The persisted supervisor.state record makes that terminal condition visible. + local state_line state_phase state_ts + state_line="$(read_supervisor_state)" || state_line="" + if [ -n "$state_line" ]; then + state_phase="${state_line%%$'\t'*}" + state_ts="${state_line#*$'\t'}" + if [ "$state_phase" = "gave_up_storm" ]; then + printf 'STATUS: stopped (supervisor gave up after restart storm at %s)\n' \ + "${state_ts:-unknown}" + return 0 + fi + fi printf 'STATUS: stopped\n' return 0 } do_auto() { - if [ "${DAAF_PROVIDER_SHIM:-}" != "openai" ]; then + local requested="${DAAF_PROVIDER_SHIM:-}" observed + # Empty/unset stays a silent no-op: the shim is opt-in and must not announce + # itself on every boot where it was never requested. + if [ -z "$requested" ]; then + exit 0 + fi + # An unrecognized non-empty value is almost always a config footgun (e.g. a + # user who set the switch to a lane label like "chatgpt"). Make it visible + # instead of silently not starting, but still exit 0 so a misconfiguration + # never blocks container startup. ensure_log_dir runs first so the MANAGER + # record is never lost to the early return the old code took before it. + if [ "$requested" != "openai" ]; then + # Sanitize the observed value to a bounded single-line token before it + # reaches the grep-stable log format or the terminal: it is user-supplied + # and could otherwise inject a newline/space and forge a second record. + observed="$(printf '%s' "$requested" | tr -c 'A-Za-z0-9._-' '_' | cut -c1-64)" + if ensure_log_dir; then + manager_log_line "SHIM_AUTO_SKIPPED status=skipped reason=unrecognized_provider_shim observed=${observed} accepted=openai continuing_boot=y" || \ + printf 'SHIM_AUTO_SKIPPED status=skipped reason=unrecognized_provider_shim observed=%s accepted=openai continuing_boot=y\n' "$observed" >&2 + fi + printf 'WARNING: DAAF_PROVIDER_SHIM=%s is not a recognized shim auto-start value; the only accepted value is "openai". Leaving the shim stopped and continuing boot.\n' \ + "$observed" >&2 exit 0 fi ensure_log_dir || { diff --git a/tests/bash/start_shim.bats b/tests/bash/start_shim.bats index 5eb84ab7..5a71d6b7 100644 --- a/tests/bash/start_shim.bats +++ b/tests/bash/start_shim.bats @@ -742,3 +742,111 @@ start_external_fixture() { wait_for_dead "$new_pp" [ "$(grep -c 'process-group isolation unavailable' "$LOG_FILE")" -eq 2 ] } + +# --- D3: lifecycle status honesty (supervisor.state) ------------------------ + +@test "status reports a restart-storm give-up distinctly from a clean stop" { + # Drive the supervisor directly (as at boot via --auto), so no foreground + # manager is watching to run its post-failure cleanup. This is the scenario + # the give-up marker exists for: a detached supervisor that exhausts its + # storm budget long after launch, with nobody to observe the failure live. + export FAKE_SHIM_BEHAVIOR=crash + export FAKE_SHIM_CRASH_CODE=42 + export DAAF_SHIM_TEST_STORM_LIMIT=3 + export DAAF_SHIM_TEST_RESTART_DELAY=0.05 + mkdir -p "$LOG_DIR" + setsid "$MANAGER" __supervise >> "$LOG_FILE" 2>&1 & + DIRECT_SUPERVISOR_PID=$! + export DIRECT_SUPERVISOR_PID + wait_for_log_text "RESTART_STORM" + wait_for_log_text "supervisor exiting" + wait_for_dead "$DIRECT_SUPERVISOR_PID" + DIRECT_SUPERVISOR_PID="" + + # The give-up record persists after the supervisor exits and removes its pids. + run cat "${LOG_DIR}/supervisor.state" + assert_output --partial "gave_up_storm" + [ ! -s "${LOG_DIR}/supervisor.pid" ] + + run "$MANAGER" --status + assert_success + assert_output --partial "STATUS: stopped (supervisor gave up after restart storm at " +} + +@test "supervisor.state records running across start and restart and is cleaned on stop" { + run "$MANAGER" --start + assert_manager_success + wait_for_file "${LOG_DIR}/supervisor.state" + run cat "${LOG_DIR}/supervisor.state" + assert_output --partial "running" + [ "$(stat -c '%a' "${LOG_DIR}/supervisor.state")" = "600" ] + + run "$MANAGER" --restart + assert_manager_success + wait_for_ready + [ -f "${LOG_DIR}/supervisor.state" ] + run cat "${LOG_DIR}/supervisor.state" + assert_output --partial "running" + + run "$MANAGER" --stop + assert_success + [ ! -e "${LOG_DIR}/supervisor.state" ] + + # A clean stop is indistinguishable from never-started: plain "stopped". + run "$MANAGER" --status + assert_success + assert_output "STATUS: stopped" +} + +@test "unsafe symlink supervisor.state target is rejected by lifecycle preflight" { + mkdir -p "$LOG_DIR" + local referent="${SHIM_TEST_ROOT}/state-referent" + printf 'state-referent-unchanged\n' > "$referent" + ln -s "$referent" "${LOG_DIR}/supervisor.state" + run "$MANAGER" __rotate_logs + assert_failure + assert_output --partial "refusing unsafe shim state target" + grep -Fx 'state-referent-unchanged' "$referent" + [ -L "${LOG_DIR}/supervisor.state" ] +} + +# --- D4: --auto config footgun ---------------------------------------------- + +@test "auto warns and stays boot-safe on an unrecognized DAAF_PROVIDER_SHIM value" { + export DAAF_PROVIDER_SHIM=chatgpt + run "$MANAGER" --auto + assert_success + assert_output --partial "WARNING: DAAF_PROVIDER_SHIM=chatgpt is not a recognized" + assert_output --partial 'accepted value is "openai"' + assert_output --partial "SHIM_AUTO_SKIPPED status=skipped reason=unrecognized_provider_shim observed=chatgpt accepted=openai" + [ "$(grep -c 'MANAGER SHIM_AUTO_SKIPPED status=skipped' "$LOG_FILE")" -eq 1 ] + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 0 ] + [ ! -e "${LOG_DIR}/supervisor.pid" ] +} + +@test "auto sanitizes an injecting DAAF_PROVIDER_SHIM value into a single log token" { + export DAAF_PROVIDER_SHIM='chatgpt bogus' + run "$MANAGER" --auto + assert_success + # The space is neutralized so the value cannot forge a second log field. + assert_output --partial "observed=chatgpt_bogus accepted=openai" + [ "$(grep -c 'MANAGER SHIM_AUTO_SKIPPED status=skipped' "$LOG_FILE")" -eq 1 ] +} + +@test "auto is a silent no-op when DAAF_PROVIDER_SHIM is unset" { + unset DAAF_PROVIDER_SHIM + run "$MANAGER" --auto + assert_success + assert_output "" + [ ! -e "$LOG_FILE" ] + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 0 ] +} + +@test "auto is a silent no-op when DAAF_PROVIDER_SHIM is empty" { + export DAAF_PROVIDER_SHIM="" + run "$MANAGER" --auto + assert_success + assert_output "" + [ ! -e "$LOG_FILE" ] + [ "$(count_exact_arg_processes "${SHIM_TEST_ROOT}/anthropic_openai_shim.py")" -eq 0 ] +} diff --git a/tests/provider_shim/_loopback_harness.py b/tests/provider_shim/_loopback_harness.py index 4d165453..0391ac10 100644 --- a/tests/provider_shim/_loopback_harness.py +++ b/tests/provider_shim/_loopback_harness.py @@ -113,6 +113,10 @@ class Scenario: attempt_headers: Optional[list[dict[str, str]]] = None disconnect_phase: Optional[str] = None disconnect_delay: float = 1.8 + # v1.2.14 (R6): when > 0, the streaming success path pauses this many seconds + # just before the terminal event (WITHOUT disconnecting the client) to create a + # silent upstream gap the downstream heartbeat must bridge with ping frames. + heartbeat_gap_s: float = 0.0 @dataclass(frozen=True) @@ -299,7 +303,18 @@ def _append_tool_item( output_index: int, call_id: str = "call_fixture_1", item_id: str = "fc_fixture_1", + *, + arg_delta_fields: Optional[dict[str, Any]] = None, ) -> dict[str, Any]: + # ``arg_delta_fields`` injects extra top-level fields onto every + # function_call_arguments.delta event (keyword-only, default None so every + # existing caller's fixture is byte-identical). It exists for the v1.2.14 R1 + # `obfuscation` tolerance fixture: the live Codex wire carries an + # `obfuscation` field on arguments.delta that the shim must accept silently + # (no unknown_events increment, no stream failure) while still translating + # the tool call. Modeling it as a superset parameter keeps the ~dozens of + # existing _append_tool_item callers undisturbed. + extra = dict(arg_delta_fields or {}) arguments = '{"file_path":"/daaf/README.md"}' builder.add( "response.output_item.added", @@ -317,12 +332,14 @@ def _append_tool_item( item_id=item_id, output_index=output_index, delta='{"file_path":"/daaf/', + **extra, ) builder.add( "response.function_call_arguments.delta", item_id=item_id, output_index=output_index, delta='README.md"}', + **extra, ) # LIVE-WIRE SHAPE (2026-07-16 shim.log evidence): the Codex backend omits # `name` on arguments.done — the shim resolves it from output_item.added. @@ -607,6 +624,30 @@ def full_response_scenario(*, reject_nonstream: bool = False) -> Scenario: ) +def heartbeat_text_scenario(name: str = "heartbeat", *, gap_s: float = 0.6) -> Scenario: + """A single text turn whose upstream terminal is delayed by a silent gap. + + v1.2.14 (R6): the shim emits message_start and the text block promptly, then the + upstream falls silent for ``gap_s`` before the terminal arrives (no disconnect). + The downstream heartbeat must bridge that gap with Anthropic ``ping`` frames and + the turn must still complete as a clean success. + """ + + builder = _EventBuilder() + builder.add( + "response.created", response={"id": "resp_heartbeat", "status": "in_progress"} + ) + output = [_append_text_item(builder, 0, text="Held-open answer.")] + _finish_response(builder, "resp_heartbeat", output) + scenario = Scenario( + name=name, + stream_events=builder.events, + nonstream_response=_nonstream_response("resp_heartbeat_nonstream", output), + ) + scenario.heartbeat_gap_s = gap_s + return scenario + + def events_scenario( name: str, events: list[dict[str, Any]], @@ -784,6 +825,157 @@ def sequential_two_tools_scenario() -> Scenario: ) +def interleaved_two_tools_scenario() -> Scenario: + """Two function calls whose wire events INTERLEAVE (v1.2.14 R3). + + The second tool is ``output_item.added`` while the first is still open, and the + two tools' ``function_call_arguments.delta`` events alternate. The tolerant R3 + scheduler defers the second tool, buffers its args, and drains it after the first + tool closes — so the downstream Anthropic stream is two NON-OVERLAPPING + ``tool_use`` blocks in ADDED order, each with its own input reconstructed + correctly (strict emit, tolerant accept). This is the additive companion to + ``sequential_two_tools_scenario`` (which never interleaves on the wire). + """ + + builder = _EventBuilder() + builder.add( + "response.created", + response={"id": "resp_interleaved", "status": "in_progress"}, + ) + a_args = '{"file_path":"/daaf/A.md"}' + b_args = '{"file_path":"/daaf/B.md"}' + # Both tools are ADDED before either finishes: the second opens on the wire while + # the first is still open (the shape pre-R3 rejected as a protocol failure). + builder.add( + "response.output_item.added", + output_index=0, + item={"type": "function_call", "id": "fc_interleaved_1", + "call_id": "call_interleaved_1", "name": "Read", "status": "in_progress"}, + ) + builder.add( + "response.output_item.added", + output_index=1, + item={"type": "function_call", "id": "fc_interleaved_2", + "call_id": "call_interleaved_2", "name": "Read", "status": "in_progress"}, + ) + # Interleaved argument deltas across the two open items. + builder.add("response.function_call_arguments.delta", + item_id="fc_interleaved_1", output_index=0, delta='{"file_path":"/daaf/A') + builder.add("response.function_call_arguments.delta", + item_id="fc_interleaved_2", output_index=1, delta='{"file_path":"/daaf/B') + builder.add("response.function_call_arguments.delta", + item_id="fc_interleaved_1", output_index=0, delta='.md"}') + builder.add("response.function_call_arguments.delta", + item_id="fc_interleaved_2", output_index=1, delta='.md"}') + # Live-wire shape: no `name` on arguments.done (resolved from output_item.added). + builder.add("response.function_call_arguments.done", + item_id="fc_interleaved_1", output_index=0, arguments=a_args) + builder.add("response.function_call_arguments.done", + item_id="fc_interleaved_2", output_index=1, arguments=b_args) + item_a = {"type": "function_call", "id": "fc_interleaved_1", + "call_id": "call_interleaved_1", "name": "Read", + "arguments": a_args, "status": "completed"} + item_b = {"type": "function_call", "id": "fc_interleaved_2", + "call_id": "call_interleaved_2", "name": "Read", + "arguments": b_args, "status": "completed"} + builder.add("response.output_item.done", output_index=0, item=item_a) + builder.add("response.output_item.done", output_index=1, item=item_b) + output = [item_a, item_b] + _finish_response(builder, "resp_interleaved", output) + return Scenario( + name="interleaved-two-tools", + stream_events=builder.events, + nonstream_response=_nonstream_response("resp_interleaved_ns", output), + ) + + +def unknown_wire_scenario( + name: str = "unknown-wire", + *, + unknown_event_type: Optional[str] = "response.audio.delta", + unknown_item_type: Optional[str] = "web_search_call", +) -> Scenario: + """A clean success turn that also carries unmodeled wire shapes (v1.2.14 R1). + + The turn still reduces to a well-formed Anthropic success stream (one text + block, message_start..message_stop) so downstream completion is unaffected; + the unknown SSE event type and the unmodeled ``output_item.added`` item type + are only counted for observability (``unknown_events``/``unknown_items`` on + the terminal record). Both injections are optional so a caller can isolate a + single dimension. ``response.audio.delta`` and ``web_search_call`` are real + Responses-family shapes the shim does not model, chosen so the fixture reads + as plausible forward-compat wire rather than a synthetic token. + """ + + builder = _EventBuilder() + builder.add( + "response.created", + response={"id": f"resp_{name}", "status": "in_progress"}, + ) + if unknown_event_type is not None: + # A non-load-bearing unknown event: it parses as JSON, reaches the + # reducer catch-all, is counted, and emits nothing downstream. + builder.add( + unknown_event_type, + item_id="unknown_evt_item", + output_index=0, + delta="opaque-non-text-payload", + ) + output = [_append_text_item(builder, 0, text="Clean answer despite unknown wire.")] + if unknown_item_type is not None: + # An output_item.added whose item.type is outside _KNOWN_ITEM_TYPES: it + # carries an id (so replay bookkeeping is exercised), is counted as an + # unknown item, and opens no Anthropic block. No matching + # output_item.done is emitted — the unmodeled item is not in the terminal + # output[], so the success reducer uses only the known text block. + builder.add( + "response.output_item.added", + output_index=1, + item={ + "type": unknown_item_type, + "id": "item_unknown_1", + "status": "in_progress", + }, + ) + _finish_response(builder, f"resp_{name}", output) + return Scenario( + name=name, + stream_events=builder.events, + nonstream_response=_nonstream_response(f"resp_{name}_ns", output), + ) + + +def obfuscation_tool_scenario(name: str = "obfuscation-tolerance") -> Scenario: + """A tool turn whose arguments.delta events carry the live ``obfuscation`` field. + + ``obfuscation`` is a known-but-unmodeled field (v1.2.14 R1): the shim tolerates + it silently — no ``unknown_events`` increment, no stream failure — while the + tool call still translates to a clean Anthropic ``tool_use`` block. The value + string mimics the live Codex obfuscation blob (opaque, bounded). + """ + + builder = _EventBuilder() + builder.add( + "response.created", + response={"id": f"resp_{name}", "status": "in_progress"}, + ) + output = [ + _append_tool_item( + builder, + 0, + call_id="call_obfuscation_1", + item_id="fc_obfuscation_1", + arg_delta_fields={"obfuscation": "AB12cd34EF56gh78"}, + ), + ] + _finish_response(builder, f"resp_{name}", output) + return Scenario( + name=name, + stream_events=builder.events, + nonstream_response=_nonstream_response(f"resp_{name}_ns", output), + ) + + def incomplete_response_scenario() -> Scenario: """Incomplete Responses SSE whose terminal event carries output and usage.""" @@ -1229,6 +1421,48 @@ def _send_chunked_sse( self.wfile.flush() self.close_connection = True + def _send_chunked_sse_with_gap( + self, + events: list[dict[str, Any]], + append_done: bool, + gap_s: float, + extra_headers: Optional[dict[str, str]] = None, + ) -> None: + # v1.2.14 (R6): stream the content frames, then pause `gap_s` seconds + # just before the terminal (response.completed/incomplete) event — + # WITHOUT disconnecting — so the upstream is silent while the shim's + # downstream heartbeat must keep the connection warm with pings. Each + # SSE event is one HTTP transfer chunk; the terminal is sent after the + # gap and the stream is then closed cleanly. + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.send_header("Transfer-Encoding", "chunked") + self.send_header("Connection", "close") + for name, value in (extra_headers or {}).items(): + self.send_header(name, value) + self.end_headers() + + def _write_frame(frame: bytes) -> None: + self.wfile.write(f"{len(frame):X}\r\n".encode("ascii")) + self.wfile.write(frame) + self.wfile.write(b"\r\n") + self.wfile.flush() + + for event in events: + if event.get("type") in ("response.completed", "response.incomplete"): + time.sleep(gap_s) + _write_frame( + ( + f"event: {event.get('type', 'message')}\n" + f"data: {json.dumps(event, separators=(',', ':'))}\n\n" + ).encode("utf-8") + ) + if append_done: + _write_frame(b"data: [DONE]\n\n") + self.wfile.write(b"0\r\n\r\n") + self.wfile.flush() + self.close_connection = True + def _stream_bytes(self, events: list[dict[str, Any]], append_done: bool) -> bytes: frames = [] for event in events: @@ -1419,7 +1653,14 @@ def do_POST(self) -> None: ) if owner.scenario.append_done: frames.append(b"data: [DONE]\n\n") - if owner.scenario.abrupt_eof: + if owner.scenario.heartbeat_gap_s > 0: + self._send_chunked_sse_with_gap( + owner.scenario.stream_events, + owner.scenario.append_done, + owner.scenario.heartbeat_gap_s, + attempt_headers, + ) + elif owner.scenario.abrupt_eof: self._send_abrupt_sse(frames) elif owner.scenario.preserve_raw_stream_chunks: self._send_chunked_sse(frames, attempt_headers) @@ -1692,9 +1933,11 @@ def __call__(self) -> int: "transport_failure", "upstream_retry", "upstream_headers", + "quota_snapshot", "upstream_first_event", "downstream_first_content", "backend_error", + "request_shape", "disconnect", "terminal", "cleanup", @@ -1848,6 +2091,8 @@ class RealShim(AbstractContextManager["RealShim"]): "SHIM_SANITIZE_TOOLS", "SHIM_STRIP_MODEL_PREFIX", "SHIM_TEXT_VERBOSITY", + # v1.2.14 (R6): downstream heartbeat interval knob for the heartbeat tests. + "SHIM_PING_INTERVAL_S", }) _CONTROLLED_CHILD_ENV_NAMES = frozenset({ "SHIM_PORT", diff --git a/tests/provider_shim/test_historical_regressions.py b/tests/provider_shim/test_historical_regressions.py index d4447957..11698e71 100644 --- a/tests/provider_shim/test_historical_regressions.py +++ b/tests/provider_shim/test_historical_regressions.py @@ -348,7 +348,7 @@ def test_complete_stable_health_contract_both_lanes_and_overrides(self) -> None: self.assertEqual(set(health), _HEALTH_KEYS) self.assertEqual(health["service"], "daaf-anthropic-openai-shim") self.assertEqual(health["status"], "ok") - self.assertEqual(health["version"], "1.2.13") + self.assertEqual(health["version"], "1.2.14") self.assertEqual(health["backend_mode"], mode) self.assertEqual(health["sanitize_tools"], sanitize) self.assertEqual(health["reasoning_effort"], effort) diff --git a/tests/provider_shim/test_reasoning_formatting.py b/tests/provider_shim/test_reasoning_formatting.py index 1a581086..7ca72d09 100644 --- a/tests/provider_shim/test_reasoning_formatting.py +++ b/tests/provider_shim/test_reasoning_formatting.py @@ -76,7 +76,7 @@ def test_openai_streaming_legacy_bytes_unchanged(self) -> None: ) self.assertEqual(shim.health.get("status"), "ok") self.assertEqual(shim.health.get("backend_mode"), "openai") - self.assertEqual(shim.health.get("version"), "1.2.13") + self.assertEqual(shim.health.get("version"), "1.2.14") def test_openai_nonstreaming_legacy_bytes_unchanged(self) -> None: scenario = central_multipart_scenario() diff --git a/tests/provider_shim/test_request_translation.py b/tests/provider_shim/test_request_translation.py index 78132646..d205d35b 100644 --- a/tests/provider_shim/test_request_translation.py +++ b/tests/provider_shim/test_request_translation.py @@ -520,6 +520,93 @@ def test_system_role_tool_blocks_stay_fail_closed(self) -> None: ) self.assertEqual(len(backend.responses_requests), 0) + def test_trailing_thinking_history_replay_is_accepted_and_consumed(self) -> None: + # F2 / Quality-review adjudication 2 (the R3 trailing shape): an assistant + # HISTORY turn whose content is [text, tool_use, thinking] — thinking NOT first — + # followed by a tool_result user turn. Two invariants: + # (1) the shim ACCEPTS it (no 400): the validation gauntlet checks only role + # placement of tool_use/tool_result and text-block typing, so a trailing + # thinking block passes. + # (2) the translator CONSUMES the thinking block position-independently + # (shim L2646-2650) — neither its prose nor its signature appears in the + # outbound Responses `input`. Continuity rides the encrypted reasoning + # cache, never replayed Anthropic thinking text (Claude Code replays + # history to the SHIM, the terminating API, so the documented thinking- + # first replay invariant never engages). + thinking_sentinel = "TRAILING_THINKING_PROSE_SENTINEL_5J" + signature_sentinel = "TRAILING_THINKING_SIGNATURE_SENTINEL_6K" + messages = [ + {"role": "user", "content": "kick off the tool"}, + { + "role": "assistant", + "content": [ + {"type": "text", "text": "Let me read the file."}, + { + "type": "tool_use", + "id": "call_trailing_think", + "name": "Read", + "input": {"file_path": "/daaf/README.md"}, + }, + { + "type": "thinking", + "thinking": thinking_sentinel, + "signature": signature_sentinel, + }, + ], + }, + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": "call_trailing_think", + "content": "file contents", + } + ], + }, + ] + for mode in ("openai", "chatgpt"): + with self.subTest(mode=mode): + scenario = full_response_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, mode) as shim: + result = shim.post_messages(stream=False, messages=messages) + self.assertEqual(result.status, 200, result.text) + request_input = backend.responses_requests[0].body["input"] + serialized = json.dumps(request_input) + # (2a) trailing thinking prose + signature are consumed, never + # forwarded to the backend. + self.assertNotIn(thinking_sentinel, serialized) + self.assertNotIn(signature_sentinel, serialized) + # (2b) no reasoning item is synthesized from replayed thinking + # (nothing was cached for this call_id in a fresh process). + self.assertFalse( + any( + isinstance(item, dict) + and item.get("type") == "reasoning" + for item in request_input + ) + ) + # The surrounding turn still translated: the assistant text and + # the function_call / function_call_output round-trip survive. + self.assertTrue( + any( + isinstance(item, dict) + and item.get("type") == "function_call" + for item in request_input + ) + ) + self.assertTrue( + any( + isinstance(item, dict) + and item.get("type") == "function_call_output" + for item in request_input + ) + ) + logs = shim.captured_stderr() + self.assertNotIn(thinking_sentinel, logs) + self.assertNotIn(signature_sentinel, logs) + def test_request_ids_and_transport_timeout_families_are_bounded(self) -> None: scenario = full_response_scenario() scenario.stream_headers = {"x-request-id": "provider-request-17"} diff --git a/tests/provider_shim/test_stream_hardening.py b/tests/provider_shim/test_stream_hardening.py index e3988e78..05e5bba6 100644 --- a/tests/provider_shim/test_stream_hardening.py +++ b/tests/provider_shim/test_stream_hardening.py @@ -21,6 +21,7 @@ TypedSSEFrame, abrupt_eof_scenario, backend_status_scenario, + block_starts, assert_lifecycle_log_contract, controlled_asgi_probe, delayed_body_disconnect_scenario, @@ -47,6 +48,8 @@ structured_error_scenario, terminal_contract_scenario, terminal_failure_scenario, + text_delta_values, + thinking_delta_values, ) @@ -241,19 +244,53 @@ def test_inband_error_uses_terminal_error_contract(self) -> None: marker=f"INBAND_ERROR_{prefix.upper()}", ) - def test_reasoning_while_text_open_fails_cleanly(self) -> None: - self._assert_terminal_error( - scenario=reasoning_while_text_open_scenario(), - expected_kind="text", - marker="REASONING_WHILE_TEXT_OPEN_TERMINAL_ERROR", - ) + def test_reasoning_while_text_open_defers_to_trailing_thinking(self) -> None: + # v1.2.14 (R3, adjudicated flip): out-of-order reasoning that arrives while a + # text block is open is TOLERATED — buffered and emitted as a TRAILING thinking + # block after the text block closes (ratified downstream shape), not failed as + # the pre-R3 strict contract required. + scenario = reasoning_while_text_open_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + report = lifecycle_report(frames) + self.assertEqual( + [kind for _index, kind in report.starts], + ["text", "thinking"], + "text block, then a trailing thinking block", + ) + self.assertEqual(report.open_at_end, set()) + self.assertEqual(text_delta_values(frames), ["Partial text."]) + self.assertEqual( + thinking_delta_values(frames), ["Out-of-order thinking."] + ) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) - def test_reasoning_while_tool_open_fails_cleanly(self) -> None: - self._assert_terminal_error( - scenario=reasoning_while_tool_open_scenario(), - expected_kind="tool_use", - marker="REASONING_WHILE_TOOL_OPEN_TERMINAL_ERROR", - ) + def test_reasoning_while_tool_open_defers_to_trailing_thinking(self) -> None: + # v1.2.14 (R3, adjudicated flip): out-of-order reasoning while a tool block is + # open is tolerated — the tool block closes first, then the buffered reasoning + # is emitted as a TRAILING thinking block (ratified shape). + scenario = reasoning_while_tool_open_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + report = lifecycle_report(frames) + self.assertEqual( + [kind for _index, kind in report.starts], + ["tool_use", "thinking"], + "tool_use block, then a trailing thinking block", + ) + self.assertEqual(report.open_at_end, set()) + self.assertEqual( + thinking_delta_values(frames), ["Out-of-order thinking."] + ) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) def test_missing_terminal_response_fails_cleanly(self) -> None: self._assert_terminal_error( @@ -708,7 +745,12 @@ def test_malformed_tool_output_item_done_fails_both_modes(self) -> None: marker="MALFORMED_TOOL_OUTPUT_DONE_NONSTREAM", ) - def test_sse_complete_data_line_without_blank_boundary_fails(self) -> None: + def test_sse_complete_data_line_without_blank_boundary_flushes(self) -> None: + # v1.2.14 (R5): a fully-formed terminal event whose trailing blank-line + # boundary a proxy trimmed is now flushed at EOF and finalizes SUCCESS. Before + # R5 this raised a framing failure (B7 — a completed response reported as an + # error). The payload is complete and parses cleanly, so downstream strict + # validation still applies; only the missing blank line is tolerated. terminal = { "type": "response.completed", "response": { @@ -722,11 +764,17 @@ def test_sse_complete_data_line_without_blank_boundary_fails(self) -> None: "unterminated-complete-data-line", [f"data: {json.dumps(terminal, separators=(',', ':'))}\n".encode()], ) - self._assert_stream_failure( - scenario=scenario, - marker="SSE_COMPLETE_LINE_WITHOUT_BLANK_BOUNDARY", - expected_kinds=[], - ) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True) + self.assertEqual( + result.status, + 200, + f"SSE_COMPLETE_LINE_WITHOUT_BLANK_BOUNDARY {result.text}", + ) + lifecycle_report(parse_typed_sse(result.body)) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) def test_sse_partial_line_at_eof_fails(self) -> None: scenario = raw_sse_scenario( @@ -1199,7 +1247,10 @@ def test_duplicate_output_item_added_is_protocol_failure_without_second_start(se ] self.assertEqual(len(tool_starts), 1) - def test_text_while_tool_open_is_protocol_failure(self) -> None: + def test_text_while_tool_open_defers_to_trailing_text(self) -> None: + # v1.2.14 (R3, adjudicated flip): text emitted while a tool block is open is + # TOLERATED — buffered and emitted as a TRAILING text block after the tool + # closes (ratified shape), not failed as the pre-R3 strict contract required. base = full_response_scenario() events = json.loads(json.dumps(base.stream_events)) index = next( @@ -1216,13 +1267,31 @@ def test_text_while_tool_open_is_protocol_failure(self) -> None: "delta": "overlapping text", }, ) - self._assert_stream_failure( - scenario=events_scenario("text-while-tool-open", events), - marker="TEXT_WHILE_TOOL_OPEN", - expected_kinds=["thinking", "text", "tool_use"], - ) + scenario = events_scenario("text-while-tool-open", events) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + report = lifecycle_report(frames) + self.assertEqual( + [kind for _index, kind in report.starts], + ["thinking", "text", "tool_use", "text"], + "leading thinking+text, the tool, then a trailing text block", + ) + self.assertEqual(report.open_at_end, set()) + # Leading streamed text, then the deferred out-of-order text trailing. + self.assertEqual( + text_delta_values(frames), + ["Aggregated answer.", "overlapping text"], + ) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) - def test_second_tool_while_first_open_is_protocol_failure(self) -> None: + def test_second_tool_while_first_open_defers_to_sequential_blocks(self) -> None: + # v1.2.14 (R3, adjudicated flip): a second tool added while the first is open + # is TOLERATED — DEFERRED and drained after the first closes, yielding two + # NON-OVERLAPPING tool_use blocks in added order (ratified shape), not failed. base = full_response_scenario() events = json.loads(json.dumps(base.stream_events)) index = next( @@ -1244,11 +1313,32 @@ def test_second_tool_while_first_open_is_protocol_failure(self) -> None: }, }, ) - self._assert_stream_failure( - scenario=events_scenario("second-tool-while-first-open", events), - marker="SECOND_TOOL_WHILE_FIRST_OPEN", - expected_kinds=["thinking", "text", "tool_use"], - ) + scenario = events_scenario("second-tool-while-first-open", events) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + report = lifecycle_report(frames) + self.assertEqual( + [kind for _index, kind in report.starts], + ["thinking", "text", "tool_use", "tool_use"], + "leading thinking+text, then two non-overlapping tool blocks", + ) + self.assertEqual(report.open_at_end, set()) + # Both tool_use blocks carry the correct call ids in added order; the + # deferred second tool supplied no arguments, so its input stays empty. + tool_starts = [ + start + for start in block_starts(frames) + if (start.get("content_block") or {}).get("type") == "tool_use" + ] + self.assertEqual( + [start["content_block"]["id"] for start in tool_starts], + ["call_full_fixture", "call_overlap_second"], + ) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) def test_normal_sequential_two_tools_remains_supported(self) -> None: scenario = sequential_two_tools_scenario() @@ -1611,7 +1701,7 @@ def test_v1211_request_id_headers_and_lifecycle_correlation(self) -> None: startup_records = [ line for line in shim.captured_stderr().splitlines() - if "req_id=- phase=process shim v1.2.13 starting" in line + if "req_id=- phase=process shim v1.2.14 starting" in line ] self.assertEqual(len(startup_records), 1, startup_records) self.assertIn( diff --git a/tests/provider_shim/test_v1214_heartbeat_and_quota.py b/tests/provider_shim/test_v1214_heartbeat_and_quota.py new file mode 100644 index 00000000..f6ebd621 --- /dev/null +++ b/tests/provider_shim/test_v1214_heartbeat_and_quota.py @@ -0,0 +1,211 @@ +"""Deterministic contracts for the v1.2.14 A3b work: R6 downstream heartbeat + +idle-gap telemetry, and D1 quota-window header capture / allowlist prefix. + +1. R6 heartbeat: during a held-open silent upstream phase (content already + emitted, terminal delayed without a disconnect), Anthropic ``ping`` frames are + emitted between the content and the terminal, never splitting a downstream + frame; ``SHIM_PING_INTERVAL_S=0`` disables them in an otherwise identical run. +2. R6 idle-gap telemetry: the terminal record carries ``max_idle_gap_ms`` sized to + the held-open gap. +3. D1 quota snapshot: a chatgpt-lane 2xx with synthetic ``x-codex-*`` headers emits + exactly one ``event=quota_snapshot`` line of numeric/enum fields (absent headers + render as ``-``); the openai lane and a chatgpt 4xx emit none. +4. D1 allowlist prefix: an ``x-codex-*`` header is captured in the diagnostic + header line while an unrelated header stays excluded. +""" + +from __future__ import annotations + +import unittest + +from ._loopback_harness import ( + MockResponsesServer, + RealShim, + backend_status_scenario, + full_response_scenario, + heartbeat_text_scenario, + lifecycle_for_response, + lifecycle_report, + parse_typed_sse, +) + + +# Synthetic subscription-quota header surface modeled on the live chatgpt-lane +# capture (notes/07 §8). Secondary-window headers are intentionally omitted so the +# "absent header -> field renders as -" contract is exercised. +QUOTA_HEADERS = { + "x-codex-plan-type": "pro", + "x-codex-active-limit": "premium", + "x-codex-primary-used-percent": "30", + "x-codex-primary-window-minutes": "10080", + "x-codex-primary-reset-after-seconds": "425943", + "x-codex-credits-has-credits": "true", + "x-codex-credits-balance": "0", + "x-codex-credits-unlimited": "false", +} + + +class ProviderShimV1214HeartbeatTests(unittest.TestCase): + maxDiff = 12000 + + def _terminal_fields(self, lines: list[object]) -> dict[str, str]: + matches = [line for line in lines if line.event == "terminal"] + self.assertEqual(len(matches), 1, [line.raw for line in lines]) + return matches[0].fields + + def test_pings_bridge_silent_gap_between_content_and_terminal(self) -> None: + # A single text turn whose terminal is delayed by a silent 0.8s gap; with a + # 0.1s ping interval the shim must bridge it with several ping frames. + scenario = heartbeat_text_scenario("hb-basic", gap_s=0.8) + with MockResponsesServer(scenario) as backend: + with RealShim( + backend, "chatgpt", env_overrides={"SHIM_PING_INTERVAL_S": "0.1"} + ) as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + # Frame-integrity oracle: the full success-stream shape still holds + # (one message_start/delta/stop, non-overlapping monotonic blocks, + # message_stop last) even with pings interleaved. + report = lifecycle_report(frames) + self.assertEqual([kind for _idx, kind in report.starts], ["text"]) + # A ping that split a content frame would surface as a raw (non-dict) + # parsed frame; every frame must be a well-formed dict or [DONE]. + for frame in frames: + self.assertTrue( + isinstance(frame.data, dict) or frame.data == "[DONE]", + ("split/garbled frame", frame.raw_data), + ) + types = [ + frame.data.get("type") + for frame in frames + if isinstance(frame.data, dict) + ] + # Multiple pings, all strictly between message_start and message_delta. + self.assertGreaterEqual(types.count("ping"), 2, types) + start_at = types.index("message_start") + delta_at = types.index("message_delta") + for position, kind in enumerate(types): + if kind == "ping": + self.assertGreater(position, start_at) + self.assertLess( + position, delta_at, "ping after terminal message_delta" + ) + # Every ping frame is exactly {"type": "ping"} — never a content frame. + for frame in frames: + if isinstance(frame.data, dict) and frame.data.get("type") == "ping": + self.assertEqual(frame.data, {"type": "ping"}) + terminal = self._terminal_fields(lifecycle_for_response(shim, result)) + self.assertEqual(terminal["outcome"], "success") + # max_idle_gap_ms is present and plausibly sized for the ~0.8s gap. + self.assertIn("max_idle_gap_ms", terminal) + self.assertGreaterEqual(int(terminal["max_idle_gap_ms"]), 400) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + def test_ping_interval_zero_disables_heartbeat(self) -> None: + # The identical held-open scenario with SHIM_PING_INTERVAL_S=0 must produce + # no ping frames while still completing cleanly. + scenario = heartbeat_text_scenario("hb-disabled", gap_s=0.8) + with MockResponsesServer(scenario) as backend: + with RealShim( + backend, "chatgpt", env_overrides={"SHIM_PING_INTERVAL_S": "0"} + ) as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + report = lifecycle_report(frames) + self.assertEqual([kind for _idx, kind in report.starts], ["text"]) + types = [ + frame.data.get("type") + for frame in frames + if isinstance(frame.data, dict) + ] + self.assertNotIn("ping", types) + terminal = self._terminal_fields(lifecycle_for_response(shim, result)) + self.assertEqual(terminal["outcome"], "success") + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + +class ProviderShimV1214QuotaSnapshotTests(unittest.TestCase): + maxDiff = 12000 + + def _quota_lines(self, lines: list[object]) -> list[object]: + return [line for line in lines if line.event == "quota_snapshot"] + + def test_chatgpt_2xx_emits_one_quota_snapshot_with_expected_fields(self) -> None: + scenario = full_response_scenario() + scenario.stream_headers = dict(QUOTA_HEADERS) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + parse_typed_sse(result.body) + lines = lifecycle_for_response(shim, result) + quota = self._quota_lines(lines) + self.assertEqual(len(quota), 1, [line.raw for line in lines]) + fields = quota[0].fields + self.assertEqual(fields["plan_type"], "pro") + self.assertEqual(fields["active_limit"], "premium") + self.assertEqual(fields["primary_used_pct"], "30") + self.assertEqual(fields["primary_window_min"], "10080") + self.assertEqual(fields["primary_reset_s"], "425943") + self.assertEqual(fields["credits_has"], "true") + self.assertEqual(fields["credits_balance"], "0") + self.assertEqual(fields["credits_unlimited"], "false") + # Absent secondary-window headers render as "-". + self.assertEqual(fields["secondary_used_pct"], "-") + self.assertEqual(fields["secondary_window_min"], "-") + self.assertEqual(fields["secondary_reset_s"], "-") + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + def test_openai_lane_emits_no_quota_snapshot(self) -> None: + scenario = full_response_scenario() + scenario.stream_headers = dict(QUOTA_HEADERS) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "openai") as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + parse_typed_sse(result.body) + lines = lifecycle_for_response(shim, result) + self.assertEqual(self._quota_lines(lines), []) + shim.assert_offline_contract() + + def test_chatgpt_4xx_emits_no_quota_snapshot(self) -> None: + scenario = backend_status_scenario("quota-4xx", 400) + scenario.stream_headers = dict(QUOTA_HEADERS) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True) + # In-band error stream: HTTP 200, error surfaced as an SSE event. + self.assertEqual(result.status, 200, result.text) + lines = lifecycle_for_response(shim, result) + self.assertEqual(self._quota_lines(lines), []) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + def test_allowlist_prefix_captures_x_codex_excludes_unrelated(self) -> None: + # A chatgpt 4xx routes its headers through _diag_headers. The x-codex-* + # header must be captured by the new prefix rule; an unrelated header (not + # exact- or prefix-allowlisted) must stay excluded from the diagnostic line. + scenario = backend_status_scenario("prefix-4xx", 400) + scenario.stream_headers = { + "x-codex-anything": "capturedval", + "x-unrelated-header": "excludedval", + } + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + lifecycle_for_response(shim, result) + logs = shim.captured_stderr() + self.assertIn("x-codex-anything=capturedval", logs) + self.assertNotIn("x-unrelated-header", logs) + self.assertNotIn("excludedval", logs) + shim.assert_offline_contract() + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/provider_shim/test_v1214_observability.py b/tests/provider_shim/test_v1214_observability.py new file mode 100644 index 00000000..28bfe0a5 --- /dev/null +++ b/tests/provider_shim/test_v1214_observability.py @@ -0,0 +1,428 @@ +"""Deterministic contracts for the v1.2.14 A1 observability/classification work. + +Covers the four turnkey test families handed off from Dispatch A1 (shim frozen): + +1. R1 unknown-wire counting: an unknown SSE event type and an unmodeled + ``output_item`` type are counted on the terminal record while the turn still + completes cleanly downstream; a clean stream reports zero. +2. R1 ``obfuscation`` tolerance: a known-but-unmodeled ``obfuscation`` field on + ``arguments.delta`` is silently accepted (no unknown_events, no failure). +3. R2 unified code-driven classification: each recognized backend code maps to its + Anthropic error type across the four captured error-envelope shapes; the two + status-only rows and the adjudicated bare-503 -> api_error behavior are pinned. +4. D5 request-shape diagnosis line: an invalid_request_error emits exactly one + ``event=request_shape`` line of key NAMES only, never request values. + +Retry expectations honor the A1 build: retries are still RETRY_STATUSES-gated +(A2 lands the classification-driven gating later), so a retryable 5xx is attempted +MAX_RETRIES+1 = 4 times and a 4xx is attempted once. +""" + +from __future__ import annotations + +import json +import unittest +import urllib.parse + +from ._loopback_harness import ( + READ_TOOL, + MockResponsesServer, + RealShim, + backend_status_scenario, + failure_lifecycle_report, + full_response_scenario, + lifecycle_for_response, + lifecycle_report, + obfuscation_tool_scenario, + parse_typed_sse, + structured_error_scenario, + unknown_wire_scenario, +) + + +class ProviderShimV1214ObservabilityTests(unittest.TestCase): + maxDiff = 12000 + + # --- shared lifecycle-line helpers (mirror the stream-hardening suite) --- + + def _event_line(self, lines: list[object], event: str) -> object: + matches = [line for line in lines if line.event == event] + self.assertEqual(len(matches), 1, (event, [line.raw for line in lines])) + return matches[0] + + def _terminal_fields(self, lines: list[object]) -> dict[str, str]: + return self._event_line(lines, "terminal").fields + + def _inband_error( + self, + name: str, + error_payload: dict[str, object], + expected_type: str, + ) -> tuple[object, dict[str, str]]: + """Drive one in-band SSE ``error`` event through real classification. + + An in-band error carries no HTTP status, so classification is code/type + driven; it also arrives after ``message_start`` and is therefore never + retried (responses == 1). + """ + + scenario = structured_error_scenario(name, "error", error_payload) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + failure = failure_lifecycle_report( + parse_typed_sse(result.body), expected_type + ) + lines = lifecycle_for_response(shim, result) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + return failure, self._terminal_fields(lines) + + def _http_failure( + self, + scenario: object, + expected_type: str, + expected_requests: int, + ) -> tuple[object, dict[str, str]]: + """Drive one pre-content backend HTTP failure through status classification.""" + + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + failure = failure_lifecycle_report( + parse_typed_sse(result.body), expected_type + ) + lines = lifecycle_for_response(shim, result) + shim.assert_offline_contract() + backend.assert_request_counts(responses=expected_requests, oauth=0) + return failure, self._terminal_fields(lines) + + # --- Group 1: R1 unknown-wire counting --- + + def test_unknown_event_and_item_counted_while_turn_completes(self) -> None: + # One unknown SSE event type (response.audio.delta) and one unmodeled + # output_item.added type (web_search_call). The downstream Anthropic stream + # must still be a well-formed success (single text block), and the terminal + # record must surface exactly one unknown event and one unknown item. + scenario = unknown_wire_scenario("unknown-both") + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + report = lifecycle_report(frames) + self.assertEqual( + [kind for _index, kind in report.starts], + ["text"], + "unknown wire must not disturb the clean text block", + ) + self.assertEqual(report.open_at_end, set()) + terminal = self._terminal_fields(lifecycle_for_response(shim, result)) + self.assertEqual(terminal["outcome"], "success") + self.assertEqual(terminal["unknown_events"], "1") + self.assertEqual(terminal["unknown_items"], "1") + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + def test_clean_stream_reports_zero_unknown_counts(self) -> None: + # A fully modeled reasoning+text+tool turn must report unknown_events=0 and + # unknown_items=0 — the "0 when clean" invariant the allowlists protect. + scenario = full_response_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + lifecycle_report(parse_typed_sse(result.body)) + terminal = self._terminal_fields(lifecycle_for_response(shim, result)) + self.assertEqual(terminal["outcome"], "success") + self.assertEqual(terminal["unknown_events"], "0") + self.assertEqual(terminal["unknown_items"], "0") + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + def test_unknown_event_only_and_item_only_isolated(self) -> None: + # Isolate each dimension so a future regression cannot cross-contaminate the + # event vs item counters. + for label, kwargs, expect in ( + ("event-only", {"unknown_item_type": None}, ("1", "0")), + ("item-only", {"unknown_event_type": None}, ("0", "1")), + ): + with self.subTest(dimension=label): + scenario = unknown_wire_scenario(f"unknown-{label}", **kwargs) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + lifecycle_report(parse_typed_sse(result.body)) + terminal = self._terminal_fields( + lifecycle_for_response(shim, result) + ) + self.assertEqual(terminal["unknown_events"], expect[0], label) + self.assertEqual(terminal["unknown_items"], expect[1], label) + shim.assert_offline_contract() + + # --- Group 2: R1 obfuscation tolerance --- + + def test_obfuscation_field_on_arguments_delta_is_tolerated(self) -> None: + # A live `obfuscation` field on arguments.delta is a known-but-unmodeled + # field: tolerated silently (unknown_events stays 0) while the tool call + # still translates to a clean tool_use block. + scenario = obfuscation_tool_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + report = lifecycle_report(parse_typed_sse(result.body)) + self.assertEqual( + [kind for _index, kind in report.starts], + ["tool_use"], + ) + self.assertEqual(report.open_at_end, set()) + terminal = self._terminal_fields(lifecycle_for_response(shim, result)) + self.assertEqual(terminal["outcome"], "success") + self.assertEqual(terminal["unknown_events"], "0") + self.assertEqual(terminal["unknown_items"], "0") + self.assertEqual(terminal["tools_called"], "1") + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + # --- Group 3: R2 code/status classification + envelope shapes --- + + def test_inband_recognized_code_rows_across_envelope_shapes(self) -> None: + # Each recognized backend code maps to its Anthropic error type, and each is + # carried by a DIFFERENT captured envelope shape so both dimensions (code + # classification + envelope tolerance) are proven together. A recognized + # envelope never increments the unknown-envelope counter (unknown_events=0). + cases = ( + # (label, error_payload, expected anthropic type) + # Shape 3: flat root {code, message} + ( + "insufficient_quota/flat-root", + {"code": "insufficient_quota", "message": "plan exhausted"}, + "invalid_request_error", + ), + # Shape 2: {status, error:{...}} (chatgpt lane); top-level status is + # metadata only — classification is code-driven. + ( + "rate_limit_exceeded/status-wrapped", + {"status": 429, "error": {"code": "rate_limit_exceeded"}}, + "rate_limit_error", + ), + # Shape 1: bare {error:{...}} (openai lane). + ( + "server_is_overloaded/bare-error", + {"error": {"code": "server_is_overloaded"}}, + "overloaded_error", + ), + # Shape 3 again, distinct code -> authentication_error. + ( + "token_invalidated/flat-root", + {"code": "token_invalidated", "message": "session expired"}, + "authentication_error", + ), + ) + for label, payload, expected_type in cases: + with self.subTest(row=label): + failure, terminal = self._inband_error( + f"classify-{label.split('/')[0]}", payload, expected_type + ) + self.assertEqual(failure.error.get("type"), expected_type, label) + self.assertIsInstance(failure.error.get("message"), str, label) + self.assertLessEqual(len(failure.error.get("message", "")), 200, label) + self.assertEqual(terminal["outcome"], "error", label) + # Recognized envelope: no unknown-envelope increment. + self.assertEqual(terminal["unknown_events"], "0", label) + + def test_http_status_only_rows(self) -> None: + # Unknown backend type + no code -> pure HTTP-status classification. 422 is a + # 4xx (non-retryable, one request); 500 is a retryable 5xx (RETRY_STATUSES, + # exhausted after MAX_RETRIES -> four requests). + with self.subTest(row="unknown-code-4xx"): + failure, terminal = self._http_failure( + backend_status_scenario("status-422", 422), + "invalid_request_error", + expected_requests=1, + ) + self.assertEqual(failure.error.get("type"), "invalid_request_error") + self.assertEqual(terminal["outcome"], "error") + + with self.subTest(row="unknown-code-5xx"): + failure, terminal = self._http_failure( + backend_status_scenario("status-500", 500, retry_after="0"), + "api_error", + expected_requests=4, + ) + self.assertEqual(failure.error.get("type"), "api_error") + self.assertEqual(terminal["retries"], "3") + + def test_bare_503_classifies_as_api_error(self) -> None: + # Adjudicated (2026-07-20): a bare/unknown-code 503 -> api_error, NOT + # overloaded_error. overloaded_error is reserved for the recognized overload + # codes (server_is_overloaded/slow_down) and status 529. Both classes stay + # retryable, so client behavior is equivalent; this pins the type explicitly. + failure, terminal = self._http_failure( + backend_status_scenario("status-503-bare", 503, retry_after="0"), + "api_error", + expected_requests=4, + ) + self.assertEqual(failure.error.get("type"), "api_error") + self.assertEqual(terminal["outcome"], "error") + self.assertEqual(terminal["retries"], "3") + + def test_detail_envelope_is_recognized_and_status_classified(self) -> None: + # Codex model-rejection shape {detail: "..."} is a RECOGNIZED envelope (no + # unknown-envelope increment) even though it is code-less; it classifies by + # HTTP status (400 -> invalid_request_error). + scenario = backend_status_scenario("detail-400", 400) + scenario.stream_error_body = json.dumps( + {"detail": "The requested model is not available."}, + separators=(",", ":"), + ).encode("utf-8") + failure, terminal = self._http_failure( + scenario, "invalid_request_error", expected_requests=1 + ) + self.assertEqual(failure.error.get("type"), "invalid_request_error") + self.assertEqual(terminal["outcome"], "error") + # {detail} is recognized shape -> NOT counted as an unknown envelope. + self.assertEqual(terminal["unknown_events"], "0") + + def test_unstructured_envelope_is_counted(self) -> None: + # Contrast to {detail}: a dict with no error-ish fields is UNRECOGNIZED, so + # R1 counts it (unknown_events=1) before classifying it by HTTP status. + scenario = backend_status_scenario("unstructured-400", 400) + scenario.stream_error_body = json.dumps( + {"foo": "bar"}, separators=(",", ":") + ).encode("utf-8") + failure, terminal = self._http_failure( + scenario, "invalid_request_error", expected_requests=1 + ) + self.assertEqual(failure.error.get("type"), "invalid_request_error") + self.assertEqual(terminal["unknown_events"], "1") + + # --- Group 4: D5 request-shape diagnosis line --- + + def test_request_shape_line_on_invalid_request_leaks_no_values(self) -> None: + # On a backend invalid_request_error, exactly one `event=request_shape` line + # is emitted carrying only sorted key NAMES (keys=, and text_keys=/ + # reasoning_keys= where applicable). No request VALUE may appear on that line + # or anywhere in the captured logs (sentinel-absence, mirroring + # test_v1211_reflected_backend_prose...). + system_sentinel = "FABRICATED_SYSTEM_VALUE_D5_A1B" + tool_desc_sentinel = "FABRICATED_TOOL_DESC_VALUE_D5_A1B" + tool_input_sentinel = "FABRICATED_TOOL_INPUT_VALUE_D5_A1B" + prompt_sentinel = "FABRICATED_PROMPT_VALUE_D5_A1B" + sentinels = ( + system_sentinel, + tool_desc_sentinel, + tool_input_sentinel, + prompt_sentinel, + ) + request_body = { + "model": "gpt-fixture", + "max_tokens": 256, + "stream": True, + "system": system_sentinel, + "tools": [ + { + "name": "ShapeFixture", + "description": tool_desc_sentinel, + "input_schema": { + "type": "object", + "properties": {"payload": {"type": "string"}}, + }, + } + ], + "messages": [ + { + "role": "assistant", + "content": [ + { + "type": "tool_use", + "id": "call_shape_fixture", + "name": "ShapeFixture", + "input": {"payload": tool_input_sentinel}, + } + ], + }, + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": "call_shape_fixture", + "content": prompt_sentinel, + } + ], + }, + ], + } + # insufficient_quota -> invalid_request_error triggers the request_shape line. + scenario = structured_error_scenario( + "d5-request-shape", + "error", + {"code": "insufficient_quota", "message": "plan exhausted"}, + ) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_raw_messages( + json.dumps(request_body, separators=(",", ":")).encode("utf-8") + ) + self.assertEqual(result.status, 200, result.text) + failure_lifecycle_report( + parse_typed_sse(result.body), + expected_error_type="invalid_request_error", + ) + lines = lifecycle_for_response(shim, result) + + shape_lines = [line for line in lines if line.event == "request_shape"] + self.assertEqual( + len(shape_lines), + 1, + ("exactly one request_shape line", [ln.raw for ln in lines]), + ) + shape = shape_lines[0] + # keys= present, non-empty, and includes the always-present outbound + # Responses `input` key. The comma-joined name list is percent-encoded + # by the v1.2.11 injection-safe machine-field encoder (commas -> %2C), + # so decode with the stdlib oracle before splitting — production + # helpers are intentionally not imported. + keys_field = shape.fields.get("keys") + self.assertIsInstance(keys_field, str) + self.assertTrue(keys_field, "request_shape keys= is empty") + decoded_keys = urllib.parse.unquote(keys_field).split(",") + self.assertIn("input", decoded_keys) + # Names only — never a request value. + self.assertEqual(decoded_keys, sorted(decoded_keys)) + # text_keys/reasoning_keys, if present, are also name-lists only. + for optional_field in ("text_keys", "reasoning_keys"): + if optional_field in shape.fields: + self.assertTrue(shape.fields[optional_field]) + sub_keys = urllib.parse.unquote( + shape.fields[optional_field] + ).split(",") + self.assertEqual(sub_keys, sorted(sub_keys)) + + # No request VALUE may appear on the request_shape line, in any + # lifecycle line, or anywhere in the captured stderr. + logs = shim.captured_stderr() + for sentinel in sentinels: + self.assertNotIn(sentinel, shape.raw, sentinel) + for line in lines: + self.assertNotIn(sentinel, line.raw, (sentinel, line.event)) + self.assertNotIn(sentinel, logs, sentinel) + self.assertNotIn(sentinel, result.text, sentinel) + # The sentinels DID reach the backend request (proving non-reflection, + # not mere absence from an unsent payload). + backend_request = json.dumps( + backend.responses_requests[0].body, separators=(",", ":") + ) + for sentinel in (tool_desc_sentinel, tool_input_sentinel, prompt_sentinel): + self.assertIn(sentinel, backend_request, sentinel) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/provider_shim/test_v1214_retry_and_tail.py b/tests/provider_shim/test_v1214_retry_and_tail.py new file mode 100644 index 00000000..6a51891c --- /dev/null +++ b/tests/provider_shim/test_v1214_retry_and_tail.py @@ -0,0 +1,172 @@ +"""v1.2.14 R4 (classification-driven retry policy) + R5 (SSE tail tolerance). + +These deterministic red/green contracts exercise the two behaviors added in +Dispatch A2 against the real shim subprocess via the loopback harness: + +- R4: retry gating is driven by the unified error classifier. A recognized + deterministic code overrides the bare-status fallback (insufficient_quota-coded + 429 fails fast; rate_limit_exceeded is retried with a parsed/advertised delay), + while a bare status still reproduces today's RETRY_STATUSES behavior. A + rate-limit-class delay beyond the 60s cap fails fast instead of stalling. +- R5: a fully-formed terminal event at EOF whose trailing blank line was trimmed + is flushed (see test_stream_hardening for the clean-tail pin); a malformed tail + AFTER a terminal frame is counted and ignored (this file). +""" + +from __future__ import annotations + +import json +import time +import unittest + +from ._loopback_harness import ( + USAGE, + MockResponsesServer, + RealShim, + backend_status_scenario, + lifecycle_for_response, + lifecycle_report, + parse_typed_sse, + raw_sse_scenario, + sequenced_attempt_scenario, +) + + +class ProviderShimRetryAndTailTests(unittest.TestCase): + maxDiff = 12000 + + def _terminal(self, shim: object, result: object) -> dict[str, str]: + lines = lifecycle_for_response(shim, result) + terminals = [line for line in lines if line.event == "terminal"] + self.assertEqual(len(terminals), 1, [line.raw for line in lines]) + return terminals[0].fields + + # --- R4: classification-driven retry gating --- + + def test_insufficient_quota_429_fails_fast_no_retry(self) -> None: + # A recognized deterministic code OVERRIDES the retryable 429 status: an + # insufficient_quota-coded 429 is non-retryable and surfaces as + # invalid_request_error, with exactly one upstream request (no retry). + scenario = backend_status_scenario("quota-429", 429) + scenario.stream_error_body = ( + b'{"error":{"type":"insufficient_quota","code":"insufficient_quota",' + b'"message":"You exceeded your current quota."}}' + ) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "openai") as shim: + result = shim.post_messages(stream=False) + self.assertEqual(result.status, 429, result.text) + self.assertEqual( + result.json()["error"]["type"], "invalid_request_error") + backend.assert_request_counts(responses=1, oauth=0) + terminal = self._terminal(shim, result) + self.assertEqual(terminal["retries"], "0") + self.assertEqual(terminal["retry_delay_source"], "-") + + def test_rate_limit_exceeded_429_parsed_delay_retries(self) -> None: + # A rate_limit_exceeded code with an embedded "try again in s" hint is + # retried; the delay is parsed from the message NUMBER only and the source + # is recorded as `parsed`. A tiny 0.05s delay keeps the suite fast. + scenario = sequenced_attempt_scenario("rl-parsed", [429, 200]) + scenario.stream_error_body = ( + b'{"error":{"type":"rate_limit_exceeded","code":"rate_limit_exceeded",' + b'"message":"Rate limit reached. Please try again in 0.05s. Contact us."}}' + ) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "openai") as shim: + result = shim.post_messages(stream=False) + self.assertEqual(result.status, 200, result.text) + backend.assert_request_counts(responses=2, oauth=0) + terminal = self._terminal(shim, result) + self.assertEqual(terminal["retries"], "1") + self.assertEqual(terminal["retry_delay_source"], "parsed") + + def test_bare_429_retries_with_header_delay_source(self) -> None: + # A bare 429 (no recognized code) still retries via the RETRY_STATUSES + # fallback. With a Retry-After header present the delay source is `header` + # (value 0 keeps the retry instant). + scenario = sequenced_attempt_scenario( + "bare-429", [429, 200], headers=[{"Retry-After": "0"}, {}]) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "openai") as shim: + result = shim.post_messages(stream=False) + self.assertEqual(result.status, 200, result.text) + backend.assert_request_counts(responses=2, oauth=0) + terminal = self._terminal(shim, result) + self.assertEqual(terminal["retries"], "1") + self.assertEqual(terminal["retry_delay_source"], "header") + + def test_rate_limit_delay_beyond_cap_fails_fast(self) -> None: + # A rate-limit-class advertised delay beyond the 60s cap fails fast with + # rate_limit_error rather than sleeping — the client owns the long wait. One + # upstream request, no multi-minute internal stall. + scenario = backend_status_scenario("rl-over-cap", 429, retry_after="120") + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "openai") as shim: + start = time.monotonic() + result = shim.post_messages(stream=False) + elapsed = time.monotonic() - start + self.assertEqual(result.status, 429, result.text) + self.assertEqual( + result.json()["error"]["type"], "rate_limit_error") + backend.assert_request_counts(responses=1, oauth=0) + self.assertLess(elapsed, 30.0, "must not sleep the 120s advertised delay") + terminal = self._terminal(shim, result) + self.assertEqual(terminal["retries"], "0") + self.assertEqual(terminal["retry_delay_source"], "header") + + def test_oversized_error_body_is_bounded_and_classified_by_status(self) -> None: + # F3: a >=400 error body far larger than any real envelope (2 MiB of junk on a + # 503) must not drive an unbounded parse in _open_backend_stream's retry path. + # The body is truncated to MAX_ERROR_BODY_BYTES (1 MiB) before _plan_retry; the + # truncated junk fails envelope parse and falls back to HTTP-status gating (503 + # is in RETRY_STATUSES), so the attempt is retried and the next (200) succeeds. + # Retry-After: 0 keeps the retry instant and pins the delay source deterministically. + scenario = sequenced_attempt_scenario( + "f3-oversized-error-body", [503, 200], headers=[{"Retry-After": "0"}, {}]) + scenario.stream_error_body = b"x" * (2 * 1024 * 1024) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "openai") as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + lifecycle_report(parse_typed_sse(result.body)) + shim.assert_offline_contract() + backend.assert_request_counts(responses=2, oauth=0) + terminal = self._terminal(shim, result) + self.assertEqual(terminal["retries"], "1") + self.assertEqual(terminal["retry_delay_source"], "header") + + # --- R5: SSE tail tolerance --- + + def test_eof_malformed_tail_after_terminal_flushes_and_counts(self) -> None: + # A complete terminal event missing its blank-line boundary, followed by a + # truncated non-data field at EOF: the terminal event is flushed (success) + # and the trailing malformed bytes are counted for observability, not raised. + terminal = { + "type": "response.completed", + "response": { + "id": "resp_tail", + "status": "completed", + "output": [], + "usage": dict(USAGE), + }, + } + frame = ( + b"data: " + + json.dumps(terminal, separators=(",", ":")).encode() + + b"\nevent: response.trailing-truncated" + ) + scenario = raw_sse_scenario("eof-malformed-tail-after-terminal", [frame]) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + lifecycle_report(parse_typed_sse(result.body)) + shim.assert_offline_contract() + terminal_fields = self._terminal(shim, result) + self.assertGreaterEqual(int(terminal_fields["unknown_events"]), 1) + backend.assert_request_counts(responses=1, oauth=0) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/provider_shim/test_v1214_tolerant_reducer.py b/tests/provider_shim/test_v1214_tolerant_reducer.py new file mode 100644 index 00000000..9ae3bfea --- /dev/null +++ b/tests/provider_shim/test_v1214_tolerant_reducer.py @@ -0,0 +1,510 @@ +"""Deterministic contracts for the v1.2.14 R3 tolerant stream reducer (Dispatch A3a). + +The R3 doctrine is STRICT EMIT, TOLERANT ACCEPT: the reducer accepts interleaved / +out-of-order / null / malformed-non-load-bearing upstream shapes that pre-R3 failed +on, while the DOWNSTREAM Anthropic SSE stream stays strictly non-overlapping and +serialized. These tests pin: + +1. Serialized wire is BYTE-IDENTICAL to the pre-R3 single-slot reducer (the + non-negotiable regression gate — the tolerant scheduler must not perturb the + common serialized path). +2. Interleaved two-tool wire (second tool added while first open, alternating arg + deltas) → two non-overlapping tool_use blocks in added order, inputs correct. +3. `response.completed` with `output: null` / `output: []` after a tool turn is + tolerated (streamed state is the fallback source). +4. A text turn with NO `content_part.added` opens the text block on the first + `output_text.delta` (R3.4, pinned by test only — no shim change). +5. Malformed-JSON tolerance is TYPE-GATED: a malformed non-load-bearing status frame + (`response.in_progress`) is counted and skipped (success); a malformed + load-bearing frame (`output_text.delta`) still fails the stream strictly. +6. Reasoning-summary deltas BETWEEN two completed tools reduce to a clean thinking + block between two tool_use blocks. +""" + +from __future__ import annotations + +import json +import unittest + +from ._loopback_harness import ( + USAGE, + READ_TOOL, + MockResponsesServer, + RealShim, + Scenario, + _EventBuilder, + _append_reasoning_item, + _append_tool_item, + _finish_response, + _nonstream_response, + block_starts, + events_scenario, + failure_lifecycle_report, + interleaved_two_tools_scenario, + lifecycle_for_response, + lifecycle_report, + parse_typed_sse, + sequential_two_tools_scenario, + text_delta_values, + thinking_delta_values, +) + + +# Golden captured from the serialized-two-tools path (msg id normalized). This is the +# byte-identical baseline the R3 scheduler must not perturb: two non-overlapping +# tool_use blocks at monotonic indices 0 and 1, each with one sanitized input delta. +_SERIALIZED_TWO_TOOL_NAMES = [ + "message_start", + "content_block_start", + "content_block_delta", + "content_block_stop", + "content_block_start", + "content_block_delta", + "content_block_stop", + "message_delta", + "message_stop", +] +_SERIALIZED_TWO_TOOL_PROJECTION = [ + { + "type": "message_start", + "message": { + "id": "", + "type": "message", + "role": "assistant", + "model": "gpt-fixture", + "content": [], + "stop_reason": None, + "stop_sequence": None, + "usage": {"input_tokens": 0, "output_tokens": 0}, + }, + }, + { + "type": "content_block_start", + "index": 0, + "content_block": { + "type": "tool_use", + "id": "call_sequential_1", + "name": "Read", + "input": {}, + }, + }, + { + "type": "content_block_delta", + "index": 0, + "delta": { + "type": "input_json_delta", + "partial_json": '{"file_path": "/daaf/README.md"}', + }, + }, + {"type": "content_block_stop", "index": 0}, + { + "type": "content_block_start", + "index": 1, + "content_block": { + "type": "tool_use", + "id": "call_sequential_2", + "name": "Read", + "input": {}, + }, + }, + { + "type": "content_block_delta", + "index": 1, + "delta": { + "type": "input_json_delta", + "partial_json": '{"file_path": "/daaf/README.md"}', + }, + }, + {"type": "content_block_stop", "index": 1}, + { + "type": "message_delta", + "delta": {"stop_reason": "tool_use", "stop_sequence": None}, + "usage": { + "input_tokens": USAGE["input_tokens"], + "output_tokens": USAGE["output_tokens"], + }, + }, + {"type": "message_stop"}, +] + + +def _normalize_message_id(frames): + """Project frames to (event, data) with the dynamic message id normalized.""" + + projected = [] + for frame in frames: + data = json.loads(json.dumps(frame.data)) if isinstance(frame.data, dict) else frame.data + if isinstance(data, dict) and data.get("type") == "message_start": + message = data.get("message") or {} + if "id" in message: + message["id"] = "" + projected.append(data) + return projected + + +def _tool_input_by_index(frames): + """Reconstruct each tool block's input JSON from its input_json_delta stream.""" + + partials: dict[int, str] = {} + for event in (f.data for f in frames if isinstance(f.data, dict)): + if event.get("type") != "content_block_delta": + continue + delta = event.get("delta") or {} + if delta.get("type") == "input_json_delta": + index = event.get("index") + partials[index] = partials.get(index, "") + delta.get("partial_json", "") + return {index: json.loads(raw or "{}") for index, raw in partials.items()} + + +class ProviderShimV1214TolerantReducerTests(unittest.TestCase): + maxDiff = 20000 + + def _event_line(self, lines, event): + matches = [line for line in lines if line.event == event] + self.assertEqual(len(matches), 1, (event, [line.raw for line in lines])) + return matches[0] + + def _terminal_fields(self, lines): + return self._event_line(lines, "terminal").fields + + # --- 1. serialized byte-identical pin (non-negotiable regression gate) --- + + def test_serialized_two_tools_is_byte_identical(self) -> None: + scenario = sequential_two_tools_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + # Success lifecycle still holds (non-overlap, monotonic, clean stop). + lifecycle_report(frames) + self.assertEqual( + [frame.event for frame in frames], + _SERIALIZED_TWO_TOOL_NAMES, + ) + self.assertEqual( + _normalize_message_id(frames), + _SERIALIZED_TWO_TOOL_PROJECTION, + ) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + # --- 2. interleaved two-tool success --- + + def test_interleaved_two_tools_yields_ordered_blocks(self) -> None: + scenario = interleaved_two_tools_scenario() + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + report = lifecycle_report(frames) + self.assertEqual( + [kind for _index, kind in report.starts], + ["tool_use", "tool_use"], + ) + self.assertEqual(report.open_at_end, set()) + # Non-overlapping blocks in ADDED order, each with the correct input. + tool_starts = [ + start + for start in block_starts(frames) + if (start.get("content_block") or {}).get("type") == "tool_use" + ] + self.assertEqual( + [start["content_block"]["id"] for start in tool_starts], + ["call_interleaved_1", "call_interleaved_2"], + ) + self.assertEqual( + _tool_input_by_index(frames), + {0: {"file_path": "/daaf/A.md"}, 1: {"file_path": "/daaf/B.md"}}, + ) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + # --- 3. completed output null / [] after a tool turn --- + + def _tool_turn_with_terminal_output(self, name, terminal_output): + args = '{"file_path":"/daaf/README.md"}' + events = [ + {"type": "response.created", + "response": {"id": f"resp_{name}", "status": "in_progress"}}, + {"type": "response.output_item.added", "output_index": 0, + "item": {"type": "function_call", "id": "fc_null_1", + "call_id": "call_null_1", "name": "Read", "status": "in_progress"}}, + {"type": "response.function_call_arguments.delta", + "item_id": "fc_null_1", "output_index": 0, "delta": args}, + {"type": "response.function_call_arguments.done", + "item_id": "fc_null_1", "output_index": 0, "arguments": args}, + {"type": "response.output_item.done", "output_index": 0, + "item": {"type": "function_call", "id": "fc_null_1", "call_id": "call_null_1", + "name": "Read", "arguments": args, "status": "completed"}}, + {"type": "response.completed", + "response": {"id": f"resp_{name}", "status": "completed", + "output": terminal_output, "usage": dict(USAGE)}}, + ] + return events_scenario(name, events) + + def test_completed_null_or_empty_output_after_tool_is_tolerated(self) -> None: + for label, terminal_output in (("null", None), ("empty", [])): + with self.subTest(output=label): + scenario = self._tool_turn_with_terminal_output( + f"tool-{label}-output", terminal_output + ) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + report = lifecycle_report(frames) + self.assertEqual( + [kind for _index, kind in report.starts], + ["tool_use"], + label, + ) + self.assertEqual(report.open_at_end, set()) + self.assertEqual( + _tool_input_by_index(frames), + {0: {"file_path": "/daaf/README.md"}}, + label, + ) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + # --- 4. content_part-omitted text turn (R3.4 pin) --- + + def test_text_turn_without_content_part_opens_on_first_delta(self) -> None: + text = "No content_part first." + events = [ + {"type": "response.created", + "response": {"id": "resp_cp", "status": "in_progress"}}, + {"type": "response.output_item.added", "output_index": 0, + "item": {"type": "message", "id": "msg_cp", "role": "assistant", + "status": "in_progress", "content": []}}, + # NOTE: no response.content_part.added is emitted before the delta. + {"type": "response.output_text.delta", "item_id": "msg_cp", + "output_index": 0, "content_index": 0, "delta": text}, + {"type": "response.output_item.done", "output_index": 0, + "item": {"type": "message", "id": "msg_cp", "role": "assistant", + "status": "completed", + "content": [{"type": "output_text", "text": text}]}}, + {"type": "response.completed", + "response": {"id": "resp_cp", "status": "completed", + "output": [{"type": "message", "id": "msg_cp", + "role": "assistant", "status": "completed", + "content": [{"type": "output_text", "text": text}]}], + "usage": dict(USAGE)}}, + ] + scenario = events_scenario("content-part-omitted", events) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + report = lifecycle_report(frames) + self.assertEqual( + [kind for _index, kind in report.starts], ["text"] + ) + self.assertEqual(text_delta_values(frames), [text]) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + # --- 4b. fully-realistic no-reasoning text turn (F1 completeness pin) --- + + def test_no_reasoning_turn_is_tolerated(self) -> None: + # A live-capture-realistic streamed text turn that carries NO reasoning items + # or summary deltas at all: created -> in_progress -> output_item.added(message) + # -> content_part.added -> output_text.delta -> output_text.done -> + # content_part.done -> output_item.done -> completed. The reducer must produce a + # clean single text block (no synthesized thinking block), the text must survive, + # and every event is a known type so unknown_events stays 0. + text = "A plain answer with no reasoning at all." + events = [ + {"type": "response.created", + "response": {"id": "resp_noreason", "status": "in_progress"}}, + {"type": "response.in_progress", + "response": {"id": "resp_noreason", "status": "in_progress"}}, + {"type": "response.output_item.added", "output_index": 0, + "item": {"type": "message", "id": "msg_noreason", "role": "assistant", + "status": "in_progress", "content": []}}, + {"type": "response.content_part.added", "item_id": "msg_noreason", + "output_index": 0, "content_index": 0, + "part": {"type": "output_text", "text": ""}}, + {"type": "response.output_text.delta", "item_id": "msg_noreason", + "output_index": 0, "content_index": 0, "delta": text}, + {"type": "response.output_text.done", "item_id": "msg_noreason", + "output_index": 0, "content_index": 0, "text": text}, + {"type": "response.content_part.done", "item_id": "msg_noreason", + "output_index": 0, "content_index": 0, + "part": {"type": "output_text", "text": text}}, + {"type": "response.output_item.done", "output_index": 0, + "item": {"type": "message", "id": "msg_noreason", "role": "assistant", + "status": "completed", + "content": [{"type": "output_text", "text": text}]}}, + {"type": "response.completed", + "response": {"id": "resp_noreason", "status": "completed", + "output": [{"type": "message", "id": "msg_noreason", + "role": "assistant", "status": "completed", + "content": [{"type": "output_text", "text": text}]}], + "usage": dict(USAGE)}}, + ] + scenario = events_scenario("no-reasoning-turn", events) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + report = lifecycle_report(frames) + # Exactly one text block, no thinking block anywhere. + self.assertEqual([kind for _index, kind in report.starts], ["text"]) + self.assertEqual(report.open_at_end, set()) + self.assertEqual(text_delta_values(frames), [text]) + # No reasoning arrived, so the observability counter stays clean. + terminal_fields = self._terminal_fields( + lifecycle_for_response(shim, result) + ) + self.assertEqual(terminal_fields["outcome"], "success") + self.assertEqual(int(terminal_fields["unknown_events"]), 0) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + # --- 5. malformed-JSON tolerance is type-gated --- + + def _raw_frame(self, event_type, obj): + return ( + f"event: {event_type}\n" + f"data: {json.dumps(obj, separators=(',', ':'))}\n\n" + ).encode("utf-8") + + def _malformed_prefix_frames(self): + return [ + self._raw_frame( + "response.created", + {"type": "response.created", + "response": {"id": "resp_mal", "status": "in_progress"}}, + ), + self._raw_frame( + "response.output_item.added", + {"type": "response.output_item.added", "output_index": 0, + "item": {"type": "message", "id": "msg_mal", "role": "assistant", + "status": "in_progress", "content": []}}, + ), + self._raw_frame( + "response.output_text.delta", + {"type": "response.output_text.delta", "item_id": "msg_mal", + "output_index": 0, "content_index": 0, "delta": "Prefix."}, + ), + ] + + def test_malformed_non_load_bearing_frame_is_tolerated(self) -> None: + # A malformed `response.in_progress` frame: json.loads fails, but the probe + # recovers a non-load-bearing type, so it is counted (R1) and skipped and the + # stream still completes via the following valid terminal frame. + frames = self._malformed_prefix_frames() + frames.append( + b'event: response.in_progress\n' + b'data: {"type":"response.in_progress","x":}\n\n' + ) + terminal = { + "type": "response.completed", + "response": {"id": "resp_mal", "status": "completed", + "output": [{"type": "message", "id": "msg_mal", + "role": "assistant", "status": "completed", + "content": [{"type": "output_text", "text": "Prefix."}]}], + "usage": dict(USAGE)}, + } + frames.append(self._raw_frame("response.completed", terminal)) + frames.append(b"data: [DONE]\n\n") + scenario = Scenario( + name="malformed-in-progress-tolerated", + stream_events=[], + nonstream_response=_nonstream_response("resp_mal_ns", []), + append_done=False, + raw_stream_frames=frames, + ) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + sse = parse_typed_sse(result.body) + report = lifecycle_report(sse) + self.assertEqual([kind for _index, kind in report.starts], ["text"]) + self.assertEqual(text_delta_values(sse), ["Prefix."]) + terminal_fields = self._terminal_fields( + lifecycle_for_response(shim, result) + ) + self.assertEqual(terminal_fields["outcome"], "success") + self.assertGreaterEqual(int(terminal_fields["unknown_events"]), 1) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + def test_malformed_load_bearing_frame_still_fails(self) -> None: + # Contrast: a malformed `output_text.delta` frame carries a load-bearing type, + # so even though the probe recovers the type it is NOT in the tolerant set — + # the stream fails strictly (terminal error), never presenting partial success. + frames = self._malformed_prefix_frames() + frames.append( + b'event: response.output_text.delta\n' + b'data: {"type":"response.output_text.delta","delta":}\n\n' + ) + frames.append(b"data: [DONE]\n\n") + scenario = Scenario( + name="malformed-output-text-fails", + stream_events=[], + nonstream_response=_nonstream_response("resp_mal_fail_ns", []), + append_done=False, + raw_stream_frames=frames, + ) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True) + self.assertEqual(result.status, 200, result.text) + failure = failure_lifecycle_report(parse_typed_sse(result.body)) + self.assertEqual(failure.error.get("type"), "api_error") + self.assertEqual(failure.open_at_end, set()) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + # --- 6. reasoning-summary deltas between two completed tools --- + + def test_reasoning_between_completed_tools_is_a_thinking_block(self) -> None: + builder = _EventBuilder() + builder.add( + "response.created", + response={"id": "resp_between", "status": "in_progress"}, + ) + output = [ + _append_tool_item( + builder, 0, call_id="call_between_1", item_id="fc_between_1" + ), + _append_reasoning_item( + builder, "rs_between", 1, + [{"summary_index": 0, "deltas": ["Between the tools."]}], + ), + _append_tool_item( + builder, 2, call_id="call_between_2", item_id="fc_between_2" + ), + ] + _finish_response(builder, "resp_between", output) + scenario = Scenario( + name="reasoning-between-tools", + stream_events=builder.events, + nonstream_response=_nonstream_response("resp_between_ns", output), + ) + with MockResponsesServer(scenario) as backend: + with RealShim(backend, "chatgpt") as shim: + result = shim.post_messages(stream=True, tools=[READ_TOOL]) + self.assertEqual(result.status, 200, result.text) + frames = parse_typed_sse(result.body) + report = lifecycle_report(frames) + self.assertEqual( + [kind for _index, kind in report.starts], + ["tool_use", "thinking", "tool_use"], + ) + self.assertEqual(report.open_at_end, set()) + self.assertEqual(thinking_delta_values(frames), ["Between the tools."]) + shim.assert_offline_contract() + backend.assert_request_counts(responses=1, oauth=0) + + +if __name__ == "__main__": + unittest.main() From 8cc958f48711b792ea0385044d60bb9d403d3515 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 19:45:27 +0000 Subject: [PATCH 12/66] feat(shim): delegate ChatGPT-lane auth refresh to the codex CLI (A1-i) Make the codex CLI the single writer of CODEX_HOME/auth.json. The shim now only reads tokens; on near-expiry (proactive) or a backend 401 (reactive) it spawns codex login status and re-reads auth.json, judging success solely by the re-read result. Deletes the Python OAuth refresh path (token POST, rotated-token persistence, manager.rs-mirror reload guard) and the SHIM_OAUTH_TOKEN_URL/SHIM_OAUTH_CLIENT_ID seams. Adds SHIM_CODEX_BIN and SHIM_CODEX_TIMEOUT_S; the proactive window now mirrors codex 5-min refresh window. Every auth-failure surface carries the literal recovery command codex login --device-auth. SHIM_VERSION stays 1.2.14 - version bump, changelog, /health auth block, and start_shim.sh land in A1-ii. Implements A1-R1/R2/R3/R5/R7. Adds tests/provider_shim/fake_codex.py stub and test_v130_auth_delegation.py (12 tests); deletes the obsolete rotating-OAuth persistence test. Full suite: Ran 169 tests OK (independently verified). Co-Authored-By: Claude Fable 5 --- .../provider_shim/anthropic_openai_shim.py | 426 +++++++----------- tests/provider_shim/_loopback_harness.py | 16 +- tests/provider_shim/fake_codex.py | 97 ++++ tests/provider_shim/test_stream_hardening.py | 80 +--- .../test_v130_auth_delegation.py | 267 +++++++++++ 5 files changed, 549 insertions(+), 337 deletions(-) create mode 100755 tests/provider_shim/fake_codex.py create mode 100644 tests/provider_shim/test_v130_auth_delegation.py diff --git a/scripts/provider_shim/anthropic_openai_shim.py b/scripts/provider_shim/anthropic_openai_shim.py index 2c2be411..a6493ad6 100644 --- a/scripts/provider_shim/anthropic_openai_shim.py +++ b/scripts/provider_shim/anthropic_openai_shim.py @@ -664,21 +664,23 @@ # CODEX_HOME (chatgpt mode only) directory holding auth.json (the # codex OAuth token store, mode 0600). Compose sets it to # /home/appuser/.claude/codex-daaf. In chatgpt mode the -# shim reads $CODEX_HOME/auth.json for the access_token -# and refreshes it in place when near expiry or on a 401. -# Never logged as a path-of-secrets; /health reports only -# a codex_home_present boolean (True iff auth.json is +# shim READS $CODEX_HOME/auth.json for the access_token; +# when the token is near expiry or a backend 401 rejects +# it, the shim DELEGATES the refresh to the codex CLI +# (`codex login status`, CODEX_HOME passed through) and +# re-reads the result — codex is the single writer, the +# shim never writes auth.json. Never logged as a +# path-of-secrets; /health reports only a +# codex_home_present boolean (True iff auth.json is # readable). -# SHIM_OAUTH_TOKEN_URL (chatgpt mode only) TEST/STAGING OVERRIDE for the OAuth -# token endpoint. Default (production) is the hardcoded -# https://auth.openai.com/oauth/token. Set only to point -# the refresh POST at a mock/staging token endpoint (the -# mock rig uses this); leave unset in production. -# SHIM_OAUTH_CLIENT_ID (chatgpt mode only) TEST/STAGING OVERRIDE for the OAuth -# client_id sent on refresh. Default (production) is the -# hardcoded codex first-party client_id. Set only for a -# mock/staging token endpoint that expects a different -# client_id; leave unset in production. +# SHIM_CODEX_BIN (chatgpt mode only) codex binary invoked for delegated +# token refresh. Default "codex" (on PATH in the DAAF +# image). Overridable for portability and for test +# injection of a fake-codex stub. +# SHIM_CODEX_TIMEOUT_S (chatgpt mode only) wall-clock bound (float seconds) on +# the `codex login status` subprocess. Default 30; an +# unparseable value falls back to 30. On timeout the child +# is killed and the auth.json re-read decides success. # SHIM_STRIP_MODEL_PREFIX default "" (e.g. "openai/" to strip for api.openai.com) # SHIM_SANITIZE_TOOLS default ON ("0"/"false"/"no" to disable). Strips # known GPT "fill-every-optional" tool-call quirks @@ -1898,51 +1900,56 @@ def _populate_reasoning_cache(output_items): _client = httpx.AsyncClient(timeout=httpx.Timeout(600.0, connect=30.0)) -# --- v1.2.5: ChatGPT-subscription OAuth token layer (chatgpt mode ONLY) --- +# --- v1.3.0: ChatGPT-subscription auth layer — delegated to the codex CLI --- # INTENT: in chatgpt mode the Bearer is the OAuth access_token from # $CODEX_HOME/auth.json, not an api.openai.com API key. The access_token TTL is -# ~10 days (notes/09 A), so within any DAAF session the shim refreshes ~never: -# this layer is READ-MOSTLY. It reads the access_token, sends it as Bearer, and -# refreshes ONLY when the JWT `exp` is within a 30-min safety margin OR on a -# backend 401 (lazy). Refresh rotates the refresh_token (notes/10) and MUST -# persist atomically; a guarded reload-before-refresh (notes/09 B2, codex -# manager.rs:2388) skips the refresh when another writer already produced a -# newer on-disk token. All own-refreshes are serialized by an asyncio.Lock. -# CREDENTIAL SAFETY: no token value is ever logged. Writing token values INTO -# auth.json (0600) is the intended credential-store operation, not a leak. -# Presence is checked with `if not tok:` guards, never by printing. - -# OAuth refresh endpoint + client_id (first-party, notes/09 B). Both are the codex -# constants; env-overridable purely for testing (the mock rig points them at a -# local mock token endpoint). Real production leaves them unset -> these defaults. -_OAUTH_TOKEN_URL = os.environ.get( - "SHIM_OAUTH_TOKEN_URL", "https://auth.openai.com/oauth/token" -) -_OAUTH_CLIENT_ID = os.environ.get( - "SHIM_OAUTH_CLIENT_ID", "app_EMoamEEZ73f0CkXaXp7hrann" -) -# Refresh when the access_token is within this many seconds of its `exp` (notes/09 -# F: 30-min safety margin — deliberately wider than codex's 5-min window because a -# DAAF session is long and a mid-turn expiry is worse than a slightly-early refresh). -_TOKEN_REFRESH_MARGIN_S = 30 * 60 -# Actionable message surfaced on a permanent refresh failure or a missing/unreadable -# auth store. No secret content — just the recovery instruction (notes/09 B/F). -_RELOGIN_MSG = ("ChatGPT OAuth token refresh failed permanently; run " - "'codex login --device-auth' inside the container to re-authenticate") - -# In-process serialization of our OWN refreshes (the shim is async; concurrent -# requests must not each fire a refresh). Per-process only — cross-process -# coordination is the guarded reload-before-refresh below (notes/09 B2). +# ~10 days (notes/09 A), so within any DAAF session refreshes are rare: this layer +# is READ-MOSTLY. It reads the access_token, sends it as Bearer, and — when the +# token is near expiry (proactive) or a backend 401 rejects it (reactive) — +# DELEGATES the refresh to the codex CLI. The codex CLI is the SINGLE WRITER of +# auth.json; the shim NEVER writes it. The shim invokes `codex login status`, then +# re-reads auth.json and judges validity from the JWT `exp` claim. Delegating the +# write structurally eliminates the refresh-token-rotation race a Python-side +# refresh had with any other codex-based tool sharing the same CODEX_HOME (the +# live-confirmed 2026-07-20 auth-lockout cause). +# CREDENTIAL SAFETY: no token value is ever logged. The shim only READS auth.json; +# codex owns every write. Presence is checked with `if not tok:` guards, never by +# printing. The codex subprocess's stdout/stderr are captured (to keep them off the +# shim's own streams) but NEVER logged — codex may echo account detail. Only its +# exit code and a coarse event label are logged. + +# The codex binary used for delegated refresh. Default `codex` (on PATH in the DAAF +# image); overridable for test injection (a fake-codex stub) and portability. +SHIM_CODEX_BIN = os.environ.get("SHIM_CODEX_BIN", "").strip() or "codex" +# Wall-clock bound on the `codex login status` subprocess. Float-tolerant: an +# unparseable value falls back to the 30s default (mirrors the SHIM_PING_INTERVAL_S +# convention). A hung codex must not stall a chatgpt-lane request indefinitely; on +# timeout the child is killed and the re-read decides success/failure. +try: + SHIM_CODEX_TIMEOUT_S = float(os.environ.get("SHIM_CODEX_TIMEOUT_S", "30")) +except (ValueError, TypeError): + SHIM_CODEX_TIMEOUT_S = 30.0 +# Refresh when the access_token is within this many seconds of its `exp`. Mirrors +# codex's own CHATGPT_ACCESS_TOKEN_REFRESH_WINDOW_MINUTES (5 min, notes/05) so that +# when the shim decides to refresh, codex — keying refresh on the same window — +# actually performs it rather than treating `login status` as a no-op. +_TOKEN_REFRESH_MARGIN_S = 5 * 60 +# Actionable message surfaced on ANY auth failure (a failed delegated refresh, a +# post-retry 401, a missing/unreadable auth store, or codex reporting "Not logged +# in"). No secret content — just the recovery instruction. It MUST literally contain +# `codex login --device-auth` (A1-R5); every auth-failure surface reuses this string. +_RELOGIN_MSG = ("ChatGPT OAuth token is invalid or expired and the codex CLI could " + "not refresh it; run 'codex login --device-auth' inside the container " + "to re-authenticate") + +# In-process single-flight serialization of delegated refreshes (the shim is async; +# concurrent requests that all observe a stale/rejected token must not each spawn a +# codex subprocess — the first refresh wins and the rest adopt its result). +# Cross-process coordination is now codex's own domain as the single writer. _token_refresh_lock = asyncio.Lock() # The access_token currently held in memory. Seeded lazily from disk on first use. # Value is a secret string; never logged. _token_state = {"access_token": None, "exp": None} -# Explicit ownership for the rotating-token commit. The task exists only while the -# holder of _token_refresh_lock supervises one refresh exchange through persistence. -# A bounded failure is retained for the process lifetime to prevent accidental reuse -# of a possibly consumed refresh token. Neither field ever contains token values. -_refresh_commit_task = None -_refresh_commit_failure = None def _b64url_decode(seg): @@ -2009,233 +2016,130 @@ def _read_auth_json(): return data -def _codex_last_refresh_now(): - # INTENT: produce a `last_refresh` timestamp string matching codex's EXACT - # format so an external codex reload treats our write as well-formed. - # REASONING: codex writes RFC3339 UTC with 9-digit (nanosecond) fractional - # seconds + trailing "Z" (chrono's DateTime::to_rfc3339() default, - # notes/10). Python's datetime sources only microseconds (6 digits), so we - # format to microseconds and zero-pad the fractional field to 9 digits. - # ASSUMES: nanosecond-precision beyond microseconds is not required for - # correctness — codex only parses the field; the padding matches its STRING - # shape (notes/10 wrote 2026-07-15T00:46:57.074803000Z this exact way). - from datetime import datetime, timezone - now = datetime.now(timezone.utc) - # e.g. 2026-07-15T00:46:57.074803 -> pad microseconds (6) to nanoseconds (9). - return now.strftime("%Y-%m-%dT%H:%M:%S.") + ("%06d000" % now.microsecond) + "Z" - - -def _atomic_write_auth_json(new_data): - # INTENT: persist the updated auth.json ATOMICALLY, mode 0600, in the SAME - # directory as the real file (same-fs rename requirement for os.replace). - # REASONING: temp file in the same dir -> chmod 0600 on the temp -> os.replace() - # (atomic same-fs rename). This fixes the torn-write hazard codex itself has - # (notes/09 B2) — a reader never sees a partial file. Live-validated write - # path (notes/10). The temp is uniquely named to avoid colliding with a - # concurrent writer's temp. - # CREDENTIAL SAFETY: writing token values into the 0600 store is the INTENDED - # operation, not a leak. Nothing is logged here. - import tempfile - d = os.path.dirname(_CODEX_AUTH_PATH) - fd, tmp = tempfile.mkstemp(prefix=".auth.json.tmp.", dir=d) +async def _run_codex_login_status(): + # INTENT: invoke `{SHIM_CODEX_BIN} login status` so the codex CLI — the SINGLE + # writer of auth.json — performs any needed token refresh in its own store. + # REASONING: async create_subprocess_exec keeps the event loop unblocked; the + # subprocess inherits the shim's environ (CODEX_HOME already exported) so codex + # targets the same per-install store the shim reads. Bounded by + # SHIM_CODEX_TIMEOUT_S. This call's exit code and output are DIAGNOSTIC ONLY — + # the authoritative refresh outcome is the auth.json re-read the caller performs + # afterward, NEVER parsed from this subprocess's stdout/stderr (A1-R3). A spawn + # failure (missing binary) or a timeout is non-fatal here; the re-read decides. + # CREDENTIAL SAFETY: stdout/stderr are captured (to keep them off the shim's own + # streams) but NEVER logged — codex may echo account detail. Only the exit code + # and a coarse event label are logged. try: - with os.fdopen(fd, "w", encoding="utf-8") as f: - json.dump(new_data, f) - os.chmod(tmp, 0o600) - os.replace(tmp, _CODEX_AUTH_PATH) # atomic same-fs rename - except Exception: - # Best-effort cleanup of the temp on any failure so we never leave a - # partial credential file behind. + proc = await asyncio.create_subprocess_exec( + SHIM_CODEX_BIN, "login", "status", + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + env=os.environ.copy(), + ) + except (OSError, ValueError) as e: + # Missing binary (FileNotFoundError), permission error, bad argv, etc. Log the + # exception TYPE only; the caller's re-read surfaces the actionable error. + log.warning("event=codex_spawn_failed err=%s", type(e).__name__) + return + try: + await asyncio.wait_for(proc.communicate(), timeout=SHIM_CODEX_TIMEOUT_S) + except asyncio.TimeoutError: + log.warning("event=codex_login_status_timeout timeout_s=%s", + SHIM_CODEX_TIMEOUT_S) try: - os.unlink(tmp) - except OSError: + proc.kill() + except ProcessLookupError: pass - raise + # Reap the killed child so it does not linger as a zombie or leak its pipes. + try: + await proc.communicate() + except Exception: + pass + return + log.info("event=codex_login_status rc=%s", proc.returncode) + + +async def delegated_refresh(rejected_token=None): + # INTENT: the single delegated-refresh primitive. Serialize (single-flight), + # invoke the codex CLI to refresh, then re-read auth.json and judge validity by + # the JWT `exp` — returning a currently-valid access_token or raising + # RuntimeError(_RELOGIN_MSG). Success/failure is determined ONLY by the re-read + # result, never by the subprocess's exit code or output (A1-R3). + # REASONING: the asyncio.Lock makes concurrent callers single-flight — the first + # waiter refreshes; the rest, after acquiring the lock, adopt the now-fresh + # in-memory token WITHOUT re-invoking codex (the short-circuit below). + # rejected_token: on the reactive (401) path this is the token the backend just + # rejected. A token that merely LOOKS exp-valid but EQUALS rejected_token must + # NOT short-circuit the codex invocation — the server has already rejected it, + # so its exp margin cannot vouch for it. On the proactive path it is None. + # CREDENTIAL SAFETY: token values live in memory / auth.json only; never logged. + async with _token_refresh_lock: + now = time.time() + # Single-flight short-circuit: a concurrent waiter may have already refreshed + # to a comfortably-valid token. Adopt it without re-invoking codex — unless it + # is the very token we were told the backend rejected. + cached = _token_state["access_token"] + cached_exp = _token_state["exp"] + if (cached and cached_exp is not None + and cached_exp - now > _TOKEN_REFRESH_MARGIN_S + and (rejected_token is None or cached != rejected_token)): + return cached + # Delegate the refresh to codex (the single writer). Diagnostic only. + await _run_codex_login_status() -async def _refresh_tokens(current): - # INTENT: exchange the current refresh_token for a fresh token set, persist the - # rotated tokens atomically, and update in-memory state. Returns the new - # access_token. - # REASONING: POST _OAUTH_TOKEN_URL with client_id + grant_type=refresh_token + - # the current refresh_token. On HTTP 200 the response carries a ROTATED - # refresh_token (notes/10) plus new access_token/id_token; we MUST persist the - # rotation or a later reuse of the consumed refresh_token triggers a permanent - # refresh_token_reused lockout (notes/09 B). Deserialize LENIENTLY — the live - # response also carries undocumented fields (earliest_refresh_at, oai_is) that - # we tolerate/ignore (notes/10). We do NOT honor earliest_refresh_at for - # scheduling; the shim keys refresh decisions on the live JWT `exp` claim. - # FAST-FAIL: a non-200, or a 200 missing access_token, is a permanent failure - # for this session — raise RuntimeError(_RELOGIN_MSG). We do NOT spin. - # CREDENTIAL SAFETY: the refresh_token/access_token/id_token are handled - # in-memory and written into auth.json only; never logged. The request body - # contains the refresh_token — httpx does not log bodies, and we never print it. - tokens = current.get("tokens") or {} - refresh_token = tokens.get("refresh_token") - if not refresh_token: - raise RuntimeError("auth.json has no refresh_token; " + _RELOGIN_MSG) - payload = { - "client_id": _OAUTH_CLIENT_ID, - "grant_type": "refresh_token", - "refresh_token": refresh_token, - } - try: - # TIMEOUT: this single OAuth POST gets a DEDICATED short timeout, NOT the - # shared _client's 600s read window. - # REASONING: _refresh_tokens runs inside _token_refresh_lock; a hung - # auth.openai.com would otherwise hold that lock for up to 600s and stall - # every concurrent chatgpt-lane request behind it. The OAuth token - # endpoint is a fast, small JSON exchange — 10s connect / 30s read is - # generous for it while capping the worst-case lock hold. On timeout httpx - # raises httpx.HTTPError (TimeoutException is a subclass), which the - # existing except-clause converts to a clean RuntimeError(_RELOGIN_MSG), - # releasing the lock via the `async with` in _get_access_token. - r = await _client.post(_OAUTH_TOKEN_URL, json=payload, - headers={"Content-Type": "application/json"}, - timeout=httpx.Timeout(30.0, connect=10.0)) - except httpx.HTTPError as e: - # A transport error is not necessarily permanent, but we do not retry-spin - # a refresh (rare path); surface it as a clean failure for this attempt. - raise RuntimeError("token refresh transport error (%s); %s" - % (type(e).__name__, _RELOGIN_MSG)) - if r.status_code != 200: - # Permanent: refresh_token_reused, invalid_grant, etc. Do NOT log the body - # (may echo token fragments); log only the status. - log.error("token refresh failed status=%d (permanent); %s", - r.status_code, _RELOGIN_MSG) - raise RuntimeError(_RELOGIN_MSG) - try: - resp = r.json() - except ValueError: - raise RuntimeError("token refresh returned non-JSON; " + _RELOGIN_MSG) - new_access = resp.get("access_token") - new_refresh = resp.get("refresh_token") # ROTATED — differs from the input - new_id = resp.get("id_token") - if not new_access: - raise RuntimeError("token refresh response missing access_token; " + _RELOGIN_MSG) - - # Build the new auth.json preserving the fields codex owns, updating only the - # rotated tokens + last_refresh. Lenient: unknown response fields are ignored. - new_data = dict(current) # shallow copy preserves OPENAI_API_KEY/auth_mode/etc. - new_tokens = dict(tokens) # preserves tokens.account_id (not returned by refresh) - new_tokens["access_token"] = new_access - if new_refresh: - new_tokens["refresh_token"] = new_refresh - if new_id: - new_tokens["id_token"] = new_id - new_data["tokens"] = new_tokens - new_data["last_refresh"] = _codex_last_refresh_now() - _atomic_write_auth_json(new_data) - - _token_state["access_token"] = new_access - _token_state["exp"] = _jwt_exp(new_access) - # SAFETY: log ONLY the fact of a refresh and the new expiry (a unix ts / ISO is - # NOT a secret); never the token itself. - log.info("chatgpt token refreshed (new exp=%s)", _token_state["exp"]) - return new_access - - -async def _await_refresh_commit(current): - # INTENT: supervise one rotating-token refresh from POST launch through atomic - # persistence and in-memory update, regardless of request disconnect/cancellation. - # REASONING: the OAuth server may consume the old refresh token before the response - # arrives. Cancelling at that point could strand auth.json permanently stale. The - # token-lock holder therefore creates one explicit task and awaits it to settlement - # with _settle_owned_task; request cancellation is remembered and propagated only - # after response validation, rotated-token extraction, atomic replace, and state - # update finish. A failed exchange is retained as a bounded process-lifetime - # failure so no waiter can retry a refresh token that might already be consumed. - # CREDENTIAL SAFETY: supervision stores only the task and a generic exception; - # token values remain solely inside _refresh_tokens/auth.json and are never logged. - global _refresh_commit_task, _refresh_commit_failure - - if _refresh_commit_failure is not None: + # Authoritative outcome: re-read auth.json and judge validity from the JWT + # exp. A missing/unreadable/malformed store raises RuntimeError(_RELOGIN_MSG). + current = _read_auth_json() + disk_access = (current.get("tokens") or {}).get("access_token") + disk_exp = _jwt_exp(disk_access) + if disk_access and disk_exp is not None and disk_exp - now > _TOKEN_REFRESH_MARGIN_S: + _token_state["access_token"] = disk_access + _token_state["exp"] = disk_exp + log.info("chatgpt token refreshed via codex (new exp=%s)", disk_exp) + return disk_access + # Still invalid/absent after delegation -> actionable failure (A1-R5). raise RuntimeError(_RELOGIN_MSG) - if _refresh_commit_task is not None: - # _get_access_token owns _token_refresh_lock across this call, so a second live - # task would mean lock ownership was violated rather than useful deduplication. - raise RuntimeError("token refresh commit ownership conflict; " + _RELOGIN_MSG) - - refresh_task = asyncio.create_task(_refresh_tokens(current)) - _refresh_commit_task = refresh_task - kind, outcome, owner_cancellation = await _settle_owned_task(refresh_task) - _refresh_commit_task = None - - if kind == "exception": - _refresh_commit_failure = outcome - elif kind == "cancelled": - _refresh_commit_failure = RuntimeError(_RELOGIN_MSG) - - # Cancellation remains cancellation even when the commit itself failed. The task - # outcome has already been retrieved and the bounded failure remains for waiters. - if owner_cancellation is not None: - raise owner_cancellation - if kind == "result": - return outcome - if kind == "exception": - raise outcome - raise asyncio.CancelledError async def _get_access_token(force_refresh=False, rejected_token=None): # INTENT: return a currently-valid access_token for the chatgpt-lane Bearer. - # Refresh (guarded reload -> POST -> rotate -> atomic persist) only when the - # token is within _TOKEN_REFRESH_MARGIN_S of exp, OR when force_refresh is set - # (the lazy-401 path forces one refresh regardless of exp). - # REASONING (read-mostly): the common path reads auth.json (or the cached - # in-memory token), checks exp, and returns the Bearer with NO network call — - # the ~10-day TTL means this is the path ~always taken. Only near-expiry or a - # 401 triggers the refresh branch, which is serialized by _token_refresh_lock. - # GUARDED RELOAD-BEFORE-REFRESH (notes/09 B2, codex manager.rs:2388): inside the - # lock we re-read auth.json; if another writer (codex CLI/plugin) already - # rotated the on-disk token, adopt it and SKIP our own refresh — this avoids - # consuming (and rotating away) a refresh_token another process just replaced. - # The comparison differs by path (see below): the proactive path keys on exp - # margin; the lazy-401 path keys on TOKEN IDENTITY vs. the rejected token, - # because a 401'd token can still be far from its nominal exp (it was rejected - # server-side, so exp margin cannot decide whether a refresh is needed). + # The common path is a pure READ (no subprocess): the ~10-day TTL means the + # token is almost always comfortably valid. A refresh — DELEGATED to the codex + # CLI — happens only proactively (token within _TOKEN_REFRESH_MARGIN_S of exp) + # or reactively (force_refresh, the lazy-401 path). + # REASONING: the proactive path checks the cached token, then the authoritative + # on-disk token; only if that is missing or near-expiry does it delegate. The + # reactive path always delegates (passing the rejected token) because a 401'd + # token can still be far from its nominal exp — exp margin cannot decide. # CREDENTIAL SAFETY: token values live in memory / auth.json only; never logged. now = time.time() - # Fast path (no lock): a cached in-memory token that is comfortably valid and no - # forced refresh -> return it directly. + # Fast path (no lock, no subprocess): a cached in-memory token that is comfortably + # valid and no forced refresh -> return it directly. if not force_refresh: cached = _token_state["access_token"] cached_exp = _token_state["exp"] if cached and cached_exp is not None and cached_exp - now > _TOKEN_REFRESH_MARGIN_S: return cached - async with _token_refresh_lock: - # Re-read from disk under the lock (this IS the guarded reload). This picks - # up any external writer's refresh and is the authoritative current state. - current = _read_auth_json() - disk_access = (current.get("tokens") or {}).get("access_token") - disk_exp = _jwt_exp(disk_access) - - # Adopt the on-disk token into memory (it is at least as fresh as ours). - if disk_access: - _token_state["access_token"] = disk_access - _token_state["exp"] = disk_exp - - if force_refresh: - # Lazy-401 path. Guarded reload by TOKEN IDENTITY (not exp margin): if the - # on-disk token DIFFERS from the token the backend just rejected, another - # writer already rotated it — adopt it and skip our refresh, retrying with - # the fresh on-disk token first. If the on-disk token is the SAME one that - # got 401'd (the common single-writer case), we MUST refresh — its exp - # margin is irrelevant because the server has already rejected it. - if (disk_access and rejected_token is not None - and disk_access != rejected_token): - log.info("chatgpt 401 refresh skipped: on-disk token rotated by another writer (guarded reload)") - return disk_access - return await _await_refresh_commit(current) - - # Proactive path: refresh only if the (authoritative on-disk) token is - # missing or within the safety margin of exp. - if disk_access and disk_exp is not None and disk_exp - now > _TOKEN_REFRESH_MARGIN_S: - # Guarded reload already adopted a fresh on-disk token — no refresh. - return disk_access - return await _await_refresh_commit(current) + if force_refresh: + # Reactive (lazy-401): delegate unconditionally, passing the rejected token so + # the single-flight short-circuit cannot hand back the just-rejected token. + return await delegated_refresh(rejected_token=rejected_token) + + # Proactive: re-read the authoritative on-disk token (cheap). If it is present and + # comfortably valid, adopt and return it with no subprocess. Otherwise delegate. + # A missing/unreadable store raises RuntimeError(_RELOGIN_MSG) (A1-R5). + current = _read_auth_json() + disk_access = (current.get("tokens") or {}).get("access_token") + disk_exp = _jwt_exp(disk_access) + if disk_access: + _token_state["access_token"] = disk_access + _token_state["exp"] = disk_exp + if disk_access and disk_exp is not None and disk_exp - now > _TOKEN_REFRESH_MARGIN_S: + return disk_access + return await delegated_refresh() # --- Helpers: request translation (Anthropic -> OpenAI Responses) --- diff --git a/tests/provider_shim/_loopback_harness.py b/tests/provider_shim/_loopback_harness.py index 0391ac10..aec21285 100644 --- a/tests/provider_shim/_loopback_harness.py +++ b/tests/provider_shim/_loopback_harness.py @@ -37,6 +37,9 @@ DAAF_ROOT = Path("/daaf") PRODUCTION_SHIM = DAAF_ROOT / "scripts/provider_shim/anthropic_openai_shim.py" +# v1.3.0: the fake-codex stub the shim's delegated refresh spawns instead of the +# real codex CLI. Injected via SHIM_CODEX_BIN. Behavior is driven by FAKE_CODEX_MODE. +FAKE_CODEX_BIN = DAAF_ROOT / "tests/provider_shim/fake_codex.py" SCRATCH_ROOT = DAAF_ROOT / "scripts/scratch" SCRATCH_PREFIX = "provider-shim-unittest-" FAKE_OPENAI_KEY = "sk-FAKE_PROVIDER_SHIM_UNITTEST_OPENAI_000000000000" @@ -2101,8 +2104,11 @@ class RealShim(AbstractContextManager["RealShim"]): "SHIM_BACKEND_API_KEY", "OPENAI_API_KEY", "CODEX_HOME", - "SHIM_OAUTH_TOKEN_URL", - "SHIM_OAUTH_CLIENT_ID", + # v1.3.0: delegated-refresh binary (replaces the deleted SHIM_OAUTH_* seams). + # SHIM_CODEX_TIMEOUT_S is intentionally not set by RealShim (the shim's 30s + # default suffices for subprocess fixtures); the in-process auth-delegation + # tests exercise it via their own env patch. + "SHIM_CODEX_BIN", "HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY", @@ -2208,8 +2214,10 @@ def __enter__(self) -> "RealShim": "SHIM_BACKEND_API_KEY": FAKE_OPENAI_KEY, "OPENAI_API_KEY": FAKE_OPENAI_KEY, "CODEX_HOME": str(self.scratch_dir), - "SHIM_OAUTH_TOKEN_URL": self.oauth_url, - "SHIM_OAUTH_CLIENT_ID": "app_FAKE_PROVIDER_SHIM_UNITTEST", + # Point delegated refresh at the fake-codex stub. Existing chatgpt + # fixtures seed a far-future token, so the stub is not actually + # invoked; it defaults to a benign no-op if it ever is. + "SHIM_CODEX_BIN": str(FAKE_CODEX_BIN), "HTTP_PROXY": self.proxy_url, "HTTPS_PROXY": self.proxy_url, "ALL_PROXY": self.proxy_url, diff --git a/tests/provider_shim/fake_codex.py b/tests/provider_shim/fake_codex.py new file mode 100755 index 00000000..55f803a7 --- /dev/null +++ b/tests/provider_shim/fake_codex.py @@ -0,0 +1,97 @@ +#!/usr/bin/env python3 +"""Fake `codex` CLI stub for provider-shim auth-delegation tests. + +The production shim (v1.3.0+) no longer refreshes ChatGPT OAuth tokens itself; +it DELEGATES refresh to the codex CLI by spawning ``codex login status`` and then +re-reading ``$CODEX_HOME/auth.json``. This stub stands in for that binary so the +tests can drive every delegated-refresh outcome deterministically, entirely +offline, with no real network or real credentials. + +It is injected via the ``SHIM_CODEX_BIN`` environment variable and behaves per the +``FAKE_CODEX_MODE`` environment variable: + + refresh-success Rewrite auth.json's access_token with a fresh (far-future exp) + fabricated JWT, preserving every other field; print the + logged-in banner; exit 0. Models codex performing the refresh. + noop Print the logged-in banner; leave auth.json untouched; exit 0. + Models codex deciding no refresh is needed (or being unable to + change the on-disk token). This is the default when unset. + not-logged-in Print "Not logged in"; exit 1. Models a logged-out store. + hang Sleep far longer than any test timeout. Exercises the shim's + SHIM_CODEX_TIMEOUT_S kill path. + +Every invocation appends one line to ``$CODEX_HOME/.codex_calls`` so tests can +assert single-flight behavior (the delegated refresh spawns codex exactly once +under concurrency). All tokens/markers are transparently fabricated test-only +values — no real secret material is ever produced or read. +""" + +import base64 +import json +import os +import sys +import time + + +def _b64url(raw): + return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii") + + +def _make_fresh_jwt(ttl_seconds): + # Minimal JWT the shim's _jwt_exp() can decode: header.payload.signature with a + # base64url payload carrying a future `exp`. Signature is inert fabricated bytes. + header = _b64url(json.dumps({"alg": "none", "typ": "JWT"}).encode("utf-8")) + payload = _b64url( + json.dumps( + { + "exp": int(time.time()) + int(ttl_seconds), + "marker": "FABRICATED_FAKE_CODEX_STUB_ONLY", + } + ).encode("utf-8") + ) + signature = _b64url(b"fabricated-fake-codex-signature") + return f"{header}.{payload}.{signature}" + + +codex_home = os.environ.get("CODEX_HOME", "") +mode = os.environ.get("FAKE_CODEX_MODE", "noop").strip() or "noop" + +# Invocation counter (single-flight assertions). Best-effort; never fatal. +if codex_home: + try: + with open(os.path.join(codex_home, ".codex_calls"), "a", encoding="utf-8") as f: + f.write(f"{mode}\n") + except OSError: + pass + +if mode == "hang": + time.sleep(3600) + sys.exit(0) + +if mode == "not-logged-in": + sys.stderr.write("Not logged in\n") + sys.exit(1) + +if mode == "refresh-success": + auth_path = os.path.join(codex_home, "auth.json") + try: + with open(auth_path, "r", encoding="utf-8") as f: + data = json.load(f) + tokens = data.get("tokens") + if not isinstance(tokens, dict): + tokens = {} + data["tokens"] = tokens + tokens["access_token"] = _make_fresh_jwt(3600) + # Codex writes atomically; a plain write is sufficient for the test store. + with open(auth_path, "w", encoding="utf-8") as f: + json.dump(data, f) + os.chmod(auth_path, 0o600) + except (OSError, ValueError) as exc: + sys.stderr.write(f"fake-codex refresh-success failed: {type(exc).__name__}\n") + sys.exit(1) + sys.stdout.write("Logged in using ChatGPT\n") + sys.exit(0) + +# Default / "noop": codex reports logged-in but changes nothing on disk. +sys.stdout.write("Logged in using ChatGPT\n") +sys.exit(0) diff --git a/tests/provider_shim/test_stream_hardening.py b/tests/provider_shim/test_stream_hardening.py index 05e5bba6..a8de588d 100644 --- a/tests/provider_shim/test_stream_hardening.py +++ b/tests/provider_shim/test_stream_hardening.py @@ -1471,78 +1471,14 @@ def test_client_disconnect_cancels_stream_body_read(self) -> None: backend.assert_request_counts(responses=1, oauth=0) self.assertEqual(provider_scratch_residue(), before) - def test_disconnect_during_rotating_oauth_refresh_persists_new_token(self) -> None: - before = provider_scratch_residue() - scenario = full_response_scenario() - with MockResponsesServer(scenario) as backend: - backend.delay_oauth_response = True - with RealShim(backend, "chatgpt") as shim: - shim.expire_auth_access_token() - client_errors: list[BaseException] = [] - - def disconnecting_client() -> None: - try: - raw_disconnect_messages( - shim, - close_after_event=backend.oauth_request_received, - timeout=4.0, - ) - except BaseException as error: - client_errors.append(error) - - client = threading.Thread( - target=disconnecting_client, - name="provider-shim-oauth-disconnect-client", - daemon=True, - ) - client.start() - self.assertTrue( - backend.oauth_request_received.wait(4.0), - "ROTATING_OAUTH_REQUEST_NOT_RECEIVED", - ) - client.join(timeout=2.0) - self.assertFalse(client.is_alive(), "ROTATING_OAUTH_CLIENT_DID_NOT_DISCONNECT") - self.assertEqual(client_errors, []) - backend.oauth_response_release.set() - self.assertTrue( - backend.oauth_response_sent.wait(2.0), - "ROTATING_OAUTH_RESPONSE_NOT_SENT_AFTER_DISCONNECT", - ) - - persisted = False - persisted_deadline = time.monotonic() + 3.0 - while time.monotonic() < persisted_deadline: - auth = json.loads(shim.auth_path.read_text(encoding="utf-8")) - tokens = auth.get("tokens") or {} - if ( - tokens.get("access_token") == backend.rotated_access_token - and tokens.get("refresh_token") == FAKE_REFRESH_TOKEN + "_ROTATED" - ): - persisted = True - break - time.sleep(0.02) - self.assertTrue(persisted, "ROTATING_OAUTH_TOKEN_PAIR_NOT_PERSISTED") - self.assertEqual(len(backend.oauth_requests), 1) - self.assertTrue( - backend.oauth_requests[0].body.get("refresh_token") - == FAKE_REFRESH_TOKEN, - "ROTATING_OAUTH_SERVER_DID_NOT_CONSUME_ORIGINAL_REFRESH_TOKEN", - ) - - result = shim.post_messages(stream=True) - self.assertEqual(result.status, 200, result.text) - lifecycle_report(parse_typed_sse(result.body)) - self.assertEqual(len(backend.responses_requests), 1) - self.assertTrue( - backend.responses_requests[0].headers.get("authorization") - == "Bearer " + backend.rotated_access_token, - "NEXT_REQUEST_DID_NOT_USE_ROTATED_ACCESS_TOKEN", - ) - backend.assert_request_counts(responses=1, oauth=1) - health = shim.get_health() - self.assertEqual(health.status, 200, health.text) - shim.assert_offline_contract() - self.assertEqual(provider_scratch_residue(), before) + # v1.3.0 (A1): test_disconnect_during_rotating_oauth_refresh_persists_new_token + # was DELETED here. It pinned the deleted Python OAuth-refresh path — the shim + # POSTing to /oauth/token, atomically persisting the ROTATED refresh/access token + # pair into auth.json, and reusing it on the next request. Under the delegation + # design (A1-R2) the shim never writes auth.json and never performs an OAuth POST; + # codex is the single writer. The disconnect-mid-refresh persistence hazard this + # test guarded is structurally gone (no shim-side write to strand). Delegated + # refresh is covered in test_v130_auth_delegation.py. def test_outer_task_cancel_after_stream_enter_closes_context(self) -> None: report = outer_cancel_after_stream_enter_report() diff --git a/tests/provider_shim/test_v130_auth_delegation.py b/tests/provider_shim/test_v130_auth_delegation.py new file mode 100644 index 00000000..4a674edb --- /dev/null +++ b/tests/provider_shim/test_v130_auth_delegation.py @@ -0,0 +1,267 @@ +"""v1.3.0 (Tier 3 A1) — ChatGPT-lane auth delegation to the codex CLI. + +The shim no longer refreshes ChatGPT OAuth tokens itself. It READS +``$CODEX_HOME/auth.json``, and when the access_token is near expiry (proactive) or +a backend 401 rejects it (reactive), it DELEGATES the refresh to the codex CLI by +spawning ``codex login status`` and then re-reading auth.json — judging success +ONLY by the re-read result, never by the subprocess's exit code or output. Codex is +the single writer; the shim never writes auth.json. + +These are in-process unit contracts: each test imports a fresh copy of the +production shim module under a controlled environment (chatgpt mode, an isolated +CODEX_HOME, and SHIM_CODEX_BIN pointed at the fake-codex stub), then drives the auth +primitives directly. The env patch stays live across the async calls so the codex +subprocess the shim spawns inherits CODEX_HOME + FAKE_CODEX_MODE. No network, no real +credentials, no real codex binary is involved. + +Coverage: +- proactive trigger refreshes a near-expiry token (and a valid token does not spawn); +- reactive 401 refresh returns the fresh token; +- an unrecoverable refresh (codex no-op on an expired token) raises an error whose + message literally contains ``codex login --device-auth``; +- the codex binary missing, and codex hanging past SHIM_CODEX_TIMEOUT_S, both surface + the actionable error rather than crashing or hanging; +- single-flight: concurrent refreshes spawn codex exactly once; +- an absent or unreadable auth.json surfaces the actionable error. +""" + +from __future__ import annotations + +import asyncio +import importlib.util +import json +import os +import shutil +import time +import unittest +import uuid +from pathlib import Path +from unittest import mock + +from ._loopback_harness import ( + FAKE_CODEX_BIN, + PRODUCTION_SHIM, + SCRATCH_ROOT, + _make_fake_jwt, +) + +RECOVERY_COMMAND = "codex login --device-auth" +FAKE_OPENAI_KEY = "sk-FAKE_PROVIDER_SHIM_AUTHTEST_000000000000" + + +def _load_fresh_shim(): + """Import a pristine copy of the production shim under the current environment.""" + + module_name = f"provider_shim_authtest_{uuid.uuid4().hex}" + spec = importlib.util.spec_from_file_location(module_name, PRODUCTION_SHIM) + if spec is None or spec.loader is None: + raise RuntimeError("could not load production shim for auth-delegation test") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +class AuthDelegationTest(unittest.TestCase): + def setUp(self) -> None: + SCRATCH_ROOT.mkdir(parents=True, exist_ok=True) + self.codex_home = SCRATCH_ROOT / f"provider-shim-authtest-{uuid.uuid4().hex}" + self.codex_home.mkdir(mode=0o700) + self.auth_path = self.codex_home / "auth.json" + + def tearDown(self) -> None: + # Restore readability so an unreadable-store test can still be cleaned up. + if self.auth_path.exists(): + try: + self.auth_path.chmod(0o600) + except OSError: + pass + if self.codex_home.exists(): + shutil.rmtree(self.codex_home, ignore_errors=True) + + # --- helpers --- + + def _seed_auth(self, exp_offset_s: int) -> str: + """Write an auth.json whose access_token expires `exp_offset_s` from now.""" + + token = _make_fake_jwt(int(time.time()) + exp_offset_s) + auth = { + "OPENAI_API_KEY": FAKE_OPENAI_KEY, + "auth_mode": "chatgpt", + "last_refresh": "2099-01-01T00:00:00.000000000Z", + "tokens": { + "access_token": token, + "account_id": "acct_authtest", + "id_token": "FAKE_ID_TOKEN", + "refresh_token": "FAKE_REFRESH_TOKEN", + }, + } + self.auth_path.write_text(json.dumps(auth), encoding="utf-8") + self.auth_path.chmod(0o600) + return token + + def _disk_token(self): + auth = json.loads(self.auth_path.read_text(encoding="utf-8")) + return (auth.get("tokens") or {}).get("access_token") + + def _codex_call_count(self) -> int: + marker = self.codex_home / ".codex_calls" + if not marker.exists(): + return 0 + return len( + [line for line in marker.read_text(encoding="utf-8").splitlines() if line] + ) + + def _env(self, *, mode="noop", timeout=None, codex_bin=None): + env = { + "SHIM_BACKEND_MODE": "chatgpt", + "SHIM_BACKEND_BASE_URL": "http://127.0.0.1:1", + "OPENAI_API_KEY": FAKE_OPENAI_KEY, + "CODEX_HOME": str(self.codex_home), + "SHIM_CODEX_BIN": codex_bin if codex_bin is not None else str(FAKE_CODEX_BIN), + "FAKE_CODEX_MODE": mode, + } + if timeout is not None: + env["SHIM_CODEX_TIMEOUT_S"] = str(timeout) + return env + + # --- proactive path --- + + def test_proactive_trigger_refreshes_near_expiry_token(self) -> None: + seeded = self._seed_auth(exp_offset_s=60) # inside the 5-min margin + with mock.patch.dict(os.environ, self._env(mode="refresh-success"), clear=False): + module = _load_fresh_shim() + returned = asyncio.run(module._get_access_token()) + fresh = self._disk_token() + self.assertNotEqual(fresh, seeded, "codex should have rewritten the token") + self.assertEqual(returned, fresh, "shim must return the freshly re-read token") + self.assertEqual(self._codex_call_count(), 1, "codex must be invoked exactly once") + + def test_proactive_valid_token_does_not_spawn_codex(self) -> None: + seeded = self._seed_auth(exp_offset_s=365 * 86400) # far future + with mock.patch.dict(os.environ, self._env(mode="refresh-success"), clear=False): + module = _load_fresh_shim() + returned = asyncio.run(module._get_access_token()) + self.assertEqual(returned, seeded, "a comfortably-valid token is returned as-is") + self.assertEqual(self._codex_call_count(), 0, "no refresh should be delegated") + + # --- reactive (401) path --- + + def test_reactive_401_refresh_returns_fresh_token(self) -> None: + seeded = self._seed_auth(exp_offset_s=365 * 86400) # exp-valid but rejected + with mock.patch.dict(os.environ, self._env(mode="refresh-success"), clear=False): + module = _load_fresh_shim() + returned = asyncio.run( + module._get_access_token(force_refresh=True, rejected_token=seeded) + ) + fresh = self._disk_token() + self.assertNotEqual(fresh, seeded) + self.assertEqual(returned, fresh) + self.assertEqual(self._codex_call_count(), 1) + + def test_reactive_401_unrecoverable_raises_actionable_error(self) -> None: + # Expired on disk + codex that cannot refresh (no-op) -> the re-read is still + # invalid, so delegated refresh fails with the actionable recovery command. + seeded = self._seed_auth(exp_offset_s=-3600) + with mock.patch.dict(os.environ, self._env(mode="noop"), clear=False): + module = _load_fresh_shim() + with self.assertRaises(RuntimeError) as ctx: + asyncio.run( + module._get_access_token(force_refresh=True, rejected_token=seeded) + ) + self.assertIn(RECOVERY_COMMAND, str(ctx.exception)) + self.assertEqual(self._codex_call_count(), 1) + + def test_reactive_401_codex_not_logged_in_raises_actionable_error(self) -> None: + seeded = self._seed_auth(exp_offset_s=-3600) + with mock.patch.dict(os.environ, self._env(mode="not-logged-in"), clear=False): + module = _load_fresh_shim() + with self.assertRaises(RuntimeError) as ctx: + asyncio.run( + module._get_access_token(force_refresh=True, rejected_token=seeded) + ) + self.assertIn(RECOVERY_COMMAND, str(ctx.exception)) + + # --- resilience: missing binary, timeout --- + + def test_codex_binary_missing_surfaces_actionable_error(self) -> None: + self._seed_auth(exp_offset_s=-3600) + missing = str(self.codex_home / "nonexistent-codex-binary") + with mock.patch.dict( + os.environ, self._env(mode="noop", codex_bin=missing), clear=False + ): + module = _load_fresh_shim() + with self.assertRaises(RuntimeError) as ctx: + asyncio.run(module.delegated_refresh()) + self.assertIn(RECOVERY_COMMAND, str(ctx.exception)) + + def test_codex_hang_is_killed_and_surfaces_actionable_error(self) -> None: + self._seed_auth(exp_offset_s=-3600) + with mock.patch.dict( + os.environ, self._env(mode="hang", timeout=0.5), clear=False + ): + module = _load_fresh_shim() + self.assertEqual(module.SHIM_CODEX_TIMEOUT_S, 0.5) + start = time.monotonic() + with self.assertRaises(RuntimeError) as ctx: + asyncio.run(module.delegated_refresh()) + elapsed = time.monotonic() - start + self.assertIn(RECOVERY_COMMAND, str(ctx.exception)) + # Bounded by the timeout knob, not the stub's 3600s sleep. + self.assertLess(elapsed, 20.0, "hung codex must be killed near the timeout") + + def test_timeout_env_unparseable_falls_back_to_default(self) -> None: + self._seed_auth(exp_offset_s=365 * 86400) + env = self._env(mode="noop") + env["SHIM_CODEX_TIMEOUT_S"] = "not-a-number" + with mock.patch.dict(os.environ, env, clear=False): + module = _load_fresh_shim() + self.assertEqual(module.SHIM_CODEX_TIMEOUT_S, 30.0) + + # --- single-flight --- + + def test_single_flight_concurrent_refreshes_spawn_codex_once(self) -> None: + self._seed_auth(exp_offset_s=60) # near expiry -> all callers want a refresh + + async def _race(module): + return await asyncio.gather( + *[module.delegated_refresh() for _ in range(8)] + ) + + with mock.patch.dict(os.environ, self._env(mode="refresh-success"), clear=False): + module = _load_fresh_shim() + results = asyncio.run(_race(module)) + fresh = self._disk_token() + self.assertTrue(all(token == fresh for token in results)) + self.assertEqual( + self._codex_call_count(), 1, "single-flight must spawn codex exactly once" + ) + + # --- absent / unreadable store --- + + def test_auth_json_absent_surfaces_actionable_error(self) -> None: + # No auth.json is written at all. + with mock.patch.dict(os.environ, self._env(mode="refresh-success"), clear=False): + module = _load_fresh_shim() + with self.assertRaises(RuntimeError) as ctx: + asyncio.run(module._get_access_token()) + self.assertIn(RECOVERY_COMMAND, str(ctx.exception)) + + def test_auth_json_unreadable_surfaces_actionable_error(self) -> None: + self._seed_auth(exp_offset_s=365 * 86400) + self.auth_path.chmod(0o000) + with mock.patch.dict(os.environ, self._env(mode="refresh-success"), clear=False): + module = _load_fresh_shim() + with self.assertRaises(RuntimeError) as ctx: + asyncio.run(module._get_access_token()) + self.assertIn(RECOVERY_COMMAND, str(ctx.exception)) + + # --- invariant: the recovery command is always in the canonical message --- + + def test_relogin_message_contains_recovery_command(self) -> None: + with mock.patch.dict(os.environ, self._env(mode="noop"), clear=False): + module = _load_fresh_shim() + self.assertIn(RECOVERY_COMMAND, module._RELOGIN_MSG) + + +if __name__ == "__main__": + unittest.main() From 3c3075e92ddda6d04319c7aa9a2b663ee2e173e2 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 20:24:42 +0000 Subject: [PATCH 13/66] feat(shim): v1.3.0 - /health auth block, readiness auth warnings, deploy-smoke T0.9 (A1-ii) Complete the ChatGPT-lane auth-delegation work (Tier 3 A1) and cut v1.3.0. - /health gains a read-only auth block {state, expires_at, days_left, recovery?} on the chatgpt lane (n/a on openai), derived from auth.json presence + JWT exp only - never token material. Expiring = exp within 48h (A1-R4). - start_shim.sh readiness + --status print an auth line: expiring -> expires-in-N- days warning naming codex login --device-auth; expired/absent/unreadable -> is-dead phrasing. D4-sanitized output (A1-R6a). - deploy-smoke T0.9 extends from auth.json-readable to the /health auth block: FAIL on expired/absent/unreadable, WARN on expiring, PASS on valid (A1-R6b). - SHIM_CODEX_BIN/SHIM_CODEX_TIMEOUT_S documented in start_shim.sh; the deleted SHIM_OAUTH_* seams removed from its config docs (A1-R7). - SHIM_VERSION 1.2.14 -> 1.3.0; cumulative v1.3.0 changelog covering A1-i + A1-ii; the two historical v1.2.5 OAuth comments carry superseded-in-v1.3.0 markers. - Version pins bumped in provider-shim tests; +8 /health auth-block tests, +5 bats auth tests. Gates independently verified: Ran 177 tests OK; bats 46/46; deploy-smoke TD.0 38 OK. Co-Authored-By: Claude Fable 5 --- scripts/deploy_smoke/smoke_probes.py | 73 ++++++++-- .../provider_shim/anthropic_openai_shim.py | 126 ++++++++++++++++- scripts/provider_shim/start_shim.sh | 53 +++++++- tests/bash/start_shim.bats | 80 +++++++++++ .../test_historical_regressions.py | 17 ++- .../test_reasoning_formatting.py | 2 +- tests/provider_shim/test_stream_hardening.py | 2 +- .../test_v130_auth_delegation.py | 128 ++++++++++++++++++ 8 files changed, 459 insertions(+), 22 deletions(-) diff --git a/scripts/deploy_smoke/smoke_probes.py b/scripts/deploy_smoke/smoke_probes.py index c8b4a840..473dd70d 100644 --- a/scripts/deploy_smoke/smoke_probes.py +++ b/scripts/deploy_smoke/smoke_probes.py @@ -566,12 +566,17 @@ def probe_shim_health(route_info: RouteInfo) -> ProbeResult: def probe_auth_json(route_info: RouteInfo, env) -> ProbeResult: - """chatgpt route only: confirm $CODEX_HOME/auth.json exists and is readable - (presence/readability only — never read or emit its contents).""" - r = ProbeResult(probe_id="T0.9", name="ChatGPT auth.json readable", tier="0") + """chatgpt route only: assert the shim /health auth block reports usable + subscription auth. v1.3.0 (A1-R6b) extends this from a bare "auth.json readable" + filesystem check to the authoritative /health auth state (derived by the shim from + auth.json presence + the access_token's JWT exp — never token material): + FAIL on expired|absent|unreadable, WARN on expiring, PASS on valid. The filesystem + readability check is retained as supplementary evidence (presence/readability only — + contents never read). Non-chatgpt routes SKIP (auth validity N/A there).""" + r = ProbeResult(probe_id="T0.9", name="ChatGPT auth validity (/health)", tier="0") if route_info.detected_route != ROUTE_CHATGPT: r.verdict = Verdict.SKIP - r.detail = f"Not the chatgpt-subscription route ({route_info.detected_route}); auth.json N/A." + r.detail = f"Not the chatgpt-subscription route ({route_info.detected_route}); auth validity N/A." return r codex_home = env.get("CODEX_HOME") if not codex_home: @@ -579,15 +584,65 @@ def probe_auth_json(route_info: RouteInfo, env) -> ProbeResult: r.detail = "CODEX_HOME unset — cannot locate auth.json for the chatgpt route." r.add_evidence("env: CODEX_HOME", output="") return r + + # Supplementary filesystem evidence (presence/readability only; contents never read). auth_path = Path(codex_home) / "auth.json" - readable = auth_path.exists() and os.access(auth_path, os.R_OK) - r.add_evidence(f"os.access({auth_path}, R_OK)", output=str(readable), note="presence/readability only; contents never read") - if readable: + fs_readable = auth_path.exists() and os.access(auth_path, os.R_OK) + r.add_evidence( + f"os.access({auth_path}, R_OK)", output=str(fs_readable), + note="supplementary; presence/readability only, contents never read", + ) + + # Authoritative: the /health auth block (JWT-exp-derived validity state). + try: + with urlopen(SHIM_HEALTH_URL, timeout=10) as resp: + body = resp.read().decode("utf-8", "replace") + health = json.loads(body) + if not isinstance(health, dict): + raise ValueError("shim /health JSON must be an object") + except (URLError, socket.timeout, json.JSONDecodeError, OSError, ValueError) as e: + r.verdict = Verdict.FAIL + r.detail = f"shim /health unreachable/invalid, cannot assess auth: {type(e).__name__}: {str(e)[:200]}" + r.add_evidence(f"GET {SHIM_HEALTH_URL}", note=str(e)[:200]) + return r + + auth = health.get("auth") + if not isinstance(auth, dict): + r.verdict = Verdict.FAIL + r.detail = "shim /health carries no auth block — a pre-v1.3.0 shim or a broken build." + r.add_evidence("GET /health .auth", output="") + return r + + _KNOWN_STATES = {"valid", "expiring", "expired", "absent", "unreadable", "n/a"} + state = auth.get("state") + state_marker = state if state in _KNOWN_STATES else "" + days_left = auth.get("days_left") + days_marker = days_left if isinstance(days_left, (int, float)) else "" + r.add_evidence( + "GET /health .auth", + output=scrub_secret_values(json.dumps(auth)[:500]), + note=f"state={state_marker} days_left={days_marker}", + ) + + if state == "valid": r.verdict = Verdict.PASS - r.detail = "auth.json present and readable (contents not inspected)." + r.detail = f"ChatGPT subscription auth valid (days_left={days_marker})." + elif state == "expiring": + r.verdict = Verdict.WARN + r.detail = ( + f"ChatGPT subscription auth expires soon (days_left={days_marker}) — " + "run 'codex login --device-auth' before it lapses." + ) + elif state in ("expired", "absent", "unreadable"): + r.verdict = Verdict.FAIL + r.detail = ( + f"ChatGPT subscription auth is dead (state={state_marker}) — " + "run 'codex login --device-auth' to re-authenticate." + ) else: + # "n/a" on a chatgpt route, or an unrecognized state — both unexpected here. r.verdict = Verdict.FAIL - r.detail = f"auth.json missing or unreadable at {auth_path} — run 'codex login --device-auth'." + r.detail = f"unexpected auth state on the chatgpt route: state={state_marker}." return r diff --git a/scripts/provider_shim/anthropic_openai_shim.py b/scripts/provider_shim/anthropic_openai_shim.py index a6493ad6..e2897d8b 100644 --- a/scripts/provider_shim/anthropic_openai_shim.py +++ b/scripts/provider_shim/anthropic_openai_shim.py @@ -63,6 +63,50 @@ # * GET /health endpoint for the manager's idempotency/status checks. # # Changelog: +# v1.3.0 (2026-07-20): ChatGPT-lane auth refresh DELEGATED to the codex CLI +# (Tier 3 A1; minor bump — behavior + config-surface change). The shim is now a +# pure READER of $CODEX_HOME/auth.json; codex is the SINGLE WRITER. This deletes +# the shim's Python OAuth reimplementation and structurally eliminates the +# refresh-rotation race that caused the 2026-07-20 auth lockout. +# A1-i (delegation core, committed separately): +# * DELETED the Python refresh path: the OAuth token POST to +# auth.openai.com/oauth/token, rotated-refresh-token atomic persistence, the +# manager.rs-mirror reload-before-refresh guard, and the SHIM_OAUTH_TOKEN_URL/ +# SHIM_OAUTH_CLIENT_ID env seams (their only consumer was the deleted path). +# No legacy fallback flag — a retained Python path would preserve the race. +# * ADDED delegated_refresh(): a single-flight (asyncio.Lock) primitive that +# spawns `{SHIM_CODEX_BIN} login status` (CODEX_HOME passthrough, bounded by +# SHIM_CODEX_TIMEOUT_S) then RE-READS auth.json and judges success SOLELY by +# the re-read result — the subprocess exit code/output are diagnostic-only. +# Proactive trigger: token within a 5-min exp margin (was 30 min) mirroring +# codex's CHATGPT_ACCESS_TOKEN_REFRESH_WINDOW_MINUTES so codex actually +# refreshes when asked. Reactive trigger: backend 401 -> one delegated_refresh +# + single retry (existing retry-guard shape preserved). +# * NEW config: SHIM_CODEX_BIN (default "codex") and SHIM_CODEX_TIMEOUT_S +# (float, default 30, unparseable->30). CODEX_HOME passthrough unchanged. +# * Every auth-failure surface (failed delegation, post-retry 401, absent/ +# unreadable store, codex "Not logged in"/missing/timeout) raises the actionable +# _RELOGIN_MSG that literally contains `codex login --device-auth` (A1-R5). +# A1-ii (auth surfaces, this dispatch): +# * A1-R4: /health gains a read-only `auth` block {state, expires_at, days_left, +# recovery?} on the chatgpt lane ("n/a" on the openai lane). state is +# valid|expiring|expired|absent|unreadable, derived from auth.json presence + +# JWT exp decode only — NEVER token material. `recovery` (the literal re-login +# command) is present only for the four actionable states. "expiring" = exp +# within _AUTH_EXPIRING_WINDOW_S (48h) — a USER heads-up horizon, deliberately +# wider than the internal 5-min refresh margin. +# * A1-R6a: start_shim.sh readiness + --status query /health and print an auth +# line; expiring -> prominent "expires in N days" warning naming +# `codex login --device-auth`; expired/absent/unreadable -> the "is dead" phrasing. +# * A1-R6b: deploy-smoke T0.9 extends from "auth.json readable" to asserting the +# /health auth block on shim routes: FAIL on expired|absent|unreadable, WARN on +# expiring, PASS on valid (SKIP on non-shim routes). +# * A1-R7: SHIM_CODEX_BIN/SHIM_CODEX_TIMEOUT_S documented in the Config block and +# start_shim.sh help/env passthrough. +# The two historical v1.2.5 comments describing the now-deleted Python OAuth path +# carry inline "[superseded in v1.3.0 ...]" markers so version history stays +# accurate without asserting present-tense behavior that no longer exists. +# SHIM_VERSION -> 1.3.0. # v1.2.14 (2026-07-20): Robustness (R1-R6) + diagnostics (D1-D5) hardening pass, # driven by the v1.2.8/v1.2.13 self-inflicted-outage retrospective. Doctrine: # "strict emit, tolerant accept" — every change moves an unexpected upstream @@ -415,7 +459,12 @@ # strict:false->true) and returns STANDARD Responses SSE. So chatgpt mode emits # ONLY the 3-header floor: authorization (Bearer ), # content-type: application/json, accept: text/event-stream. -# TOKEN LAYER (the only genuinely new component; notes/09 + notes/10, live- +# TOKEN LAYER [SUPERSEDED in v1.3.0 — the Python OAuth refresh described in +# present tense below (the auth.openai.com token POST, rotated-refresh-token +# persistence, and the manager.rs-mirror reload guard) was DELETED in v1.3.0; +# auth refresh is now delegated to the codex CLI (see the v1.3.0 entry). The +# paragraph is preserved verbatim as v1.2.5 history — read it in the past tense] +# (the only genuinely new component; notes/09 + notes/10, live- # verified): the access_token TTL is ~10 days, so within any DAAF session the # shim refreshes ~never — it is read-mostly. It decodes the access_token JWT # PAYLOAD ONLY (never the signature) for the `exp` claim and refreshes only @@ -449,9 +498,11 @@ # hung auth.openai.com cannot hold _token_refresh_lock (and block concurrent # chatgpt-lane requests) for up to 600s; httpx timeout -> the existing clean # RuntimeError(_RELOGIN_MSG). (b) The two OAuth test/staging override env vars -# (SHIM_OAUTH_TOKEN_URL, SHIM_OAUTH_CLIENT_ID) are now documented in the Config -# block here and in start_shim.sh (production leaves them unset -> hardcoded -# codex defaults). (c) codex_home_present is now HONEST — it reflects actual +# (SHIM_OAUTH_TOKEN_URL, SHIM_OAUTH_CLIENT_ID) were documented in the Config +# block and in start_shim.sh (production leaves them unset -> hardcoded +# codex defaults). [SUPERSEDED in v1.3.0 — both env seams were REMOVED with the +# Python refresh path; they no longer exist in the Config block or start_shim.sh.] +# (c) codex_home_present is now HONEST — it reflects actual # auth.json readability (os.access R_OK), not merely CODEX_HOME being set, # matching the "resolvable auth.json" docstring; applied to BOTH /health and # the startup log field so they agree. Still a presence-only boolean (no @@ -745,7 +796,7 @@ import uvicorn # --- Config --- -SHIM_VERSION = "1.2.14" +SHIM_VERSION = "1.3.0" SHIM_SERVICE_ID = "daaf-anthropic-openai-shim" SHIM_PORT = int(os.environ.get("SHIM_PORT", "4141")) @@ -5864,6 +5915,67 @@ async def _handle_models(send): ]}) +# A1-R4 (v1.3.0): read-only auth-validity snapshot for /health. Derived ENTIRELY from +# auth.json presence + the access_token's JWT `exp` claim (payload decoded, never +# verified, never the signature). It NEVER returns token material — only a coarse +# state, the decoded expiry, a day count, and (for actionable states) the literal +# recovery command. This is a REPORTING mirror of the auth store: it never writes, +# never refreshes, and never touches the delegation path (delegated_refresh's domain). +# The "expiring" horizon is deliberately WIDER than the internal 5-min refresh margin +# (_TOKEN_REFRESH_MARGIN_S): that margin governs when the shim asks codex to refresh; +# this window is a USER heads-up ("subscription auth expires in N days") surfaced by +# start_shim.sh readiness and deploy-smoke, so it warns 48h ahead — enough lead time to +# run `codex login --device-auth` before work is blocked. +_AUTH_EXPIRING_WINDOW_S = 48 * 3600 +_AUTH_RECOVERY_CMD = "codex login --device-auth" + + +def _auth_health_block(): + # INTENT: return the /health "auth" block: {state, expires_at, days_left, recovery?}. + # state is valid|expiring|expired|absent|unreadable on the chatgpt lane, and "n/a" + # on the openai (API-key) lane, which does not use the codex OAuth store. + # REASONING: absent = no readable auth.json (CODEX_HOME unset, or the file missing/ + # unreadable); unreadable = the file reads but yields no decodable access_token exp + # (malformed JSON, missing tokens/access_token, or a non-JWT/undecodable token); + # otherwise the exp decides expired (exp <= now) / expiring (within + # _AUTH_EXPIRING_WINDOW_S) / valid. `recovery` (the literal re-login command) is + # present ONLY for the four actionable states, never for valid or n/a. + # CREDENTIAL SAFETY: never returns token material — only the coarse state, the + # decoded expiry timestamp, a day count, and a static command string. Never raises + # (a /health probe must not 500); every failure path degrades to absent/unreadable. + if SHIM_BACKEND_MODE != "chatgpt": + return {"state": "n/a"} + if not _CODEX_AUTH_PATH or not os.access(_CODEX_AUTH_PATH, os.R_OK): + return {"state": "absent", "expires_at": None, "days_left": None, + "recovery": _AUTH_RECOVERY_CMD} + try: + with open(_CODEX_AUTH_PATH, "r", encoding="utf-8") as f: + data = json.load(f) + access = (data.get("tokens") or {}).get("access_token") if isinstance(data, dict) else None + except (OSError, ValueError): + access = None + exp = _jwt_exp(access) if access else None + if exp is None: + return {"state": "unreadable", "expires_at": None, "days_left": None, + "recovery": _AUTH_RECOVERY_CMD} + now = time.time() + remaining = exp - now + if remaining <= 0: + state = "expired" + elif remaining <= _AUTH_EXPIRING_WINDOW_S: + state = "expiring" + else: + state = "valid" + block = { + "state": state, + "expires_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(exp)), + "days_left": round(remaining / 86400, 1), + } + if state != "valid": + block["recovery"] = _AUTH_RECOVERY_CMD + return block + + async def _handle_health(send): # HARDENING: health endpoint for the manager's idempotency + --status checks. await _send_json(send, 200, { @@ -5888,6 +6000,10 @@ async def _handle_health(send): "sanitize_tools": SHIM_SANITIZE_TOOLS, "reasoning_effort": SHIM_REASONING_EFFORT, "text_verbosity": SHIM_TEXT_VERBOSITY, + # A1-R4 (v1.3.0): read-only auth-validity snapshot (chatgpt lane; "n/a" on the + # openai lane). Derived from auth.json presence + JWT exp only — never token + # material. start_shim.sh readiness/--status and deploy-smoke T0.9 consume it. + "auth": _auth_health_block(), }) diff --git a/scripts/provider_shim/start_shim.sh b/scripts/provider_shim/start_shim.sh index 3b0def9e..978ba63e 100755 --- a/scripts/provider_shim/start_shim.sh +++ b/scripts/provider_shim/start_shim.sh @@ -36,11 +36,15 @@ # ignored in chatgpt mode) # CODEX_HOME (chatgpt mode) directory holding auth.json, the codex # OAuth token store (mode 0600). Compose sets it to -# /home/appuser/.claude/codex-daaf. -# SHIM_OAUTH_TOKEN_URL (chatgpt mode) TEST/STAGING OVERRIDE only. Production -# default is the hardcoded auth.openai.com/oauth/token. -# SHIM_OAUTH_CLIENT_ID (chatgpt mode) TEST/STAGING OVERRIDE only. Production -# default is the hardcoded codex first-party client_id. +# /home/appuser/.claude/codex-daaf. As of v1.3.0 the shim +# only READS this store; codex is the single writer and +# performs any token refresh (see SHIM_CODEX_BIN). +# SHIM_CODEX_BIN (chatgpt mode) codex binary the shim spawns to delegate +# token refresh (`codex login status`). Default "codex" +# (on PATH in the DAAF image). v1.3.0. +# SHIM_CODEX_TIMEOUT_S (chatgpt mode) wall-clock bound (float seconds) on the +# delegated `codex login status` subprocess. Default 30; +# an unparseable value falls back to 30. v1.3.0. # SHIM_STRIP_MODEL_PREFIX default "" # SHIM_SANITIZE_TOOLS default "1" (enabled); set to 0 and restart for # DAAFBench runs of shim-routed models @@ -456,6 +460,42 @@ emit_readiness_failure() { "$EXPECTED_BACKEND_MODE" "$SHIM_PORT" >&2 } +# A1-R6a (v1.3.0): surface the /health auth block's validity as a human-readable line +# so an expiring/dead ChatGPT subscription is visible BEFORE work starts (and via +# --status). Reads the auth block already captured in HEALTH_JSON by probe_health — no +# extra request. Prints nothing when the block is absent or "n/a" (the openai API-key +# lane does not use the codex OAuth store). Both fields are sanitized to bounded, +# single-line tokens (the D4 injection convention, mirroring do_auto's `observed`) +# before reaching the terminal: they originate from our own /health but are DERIVED +# from the on-disk token store, so they are treated as untrusted for output. +print_auth_line() { + command -v jq >/dev/null 2>&1 || return 0 + [ -n "$HEALTH_JSON" ] || return 0 + local state days + state="$(printf '%s' "$HEALTH_JSON" | jq -r '.auth.state // empty' 2>/dev/null \ + | tr -cd 'a-z/' | cut -c1-16)" + [ -n "$state" ] && [ "$state" != "n/a" ] || return 0 + days="$(printf '%s' "$HEALTH_JSON" | jq -r '.auth.days_left // empty' 2>/dev/null \ + | tr -cd '0-9.-' | cut -c1-16)" + case "$state" in + valid) + printf 'AUTH: ChatGPT subscription auth is valid (expires in %s days).\n' \ + "${days:-?}" >&2 + ;; + expiring) + printf 'WARNING: ChatGPT subscription auth expires in %s days. Run `codex login --device-auth` inside the container to re-authenticate before it lapses.\n' \ + "${days:-?}" >&2 + ;; + expired|absent|unreadable) + printf 'WARNING: ChatGPT subscription auth is dead (%s) — run `codex login --device-auth` inside the container to re-authenticate.\n' \ + "$state" >&2 + ;; + *) + printf 'AUTH: ChatGPT subscription auth state: %s.\n' "$state" >&2 + ;; + esac +} + # --- Process identity helpers ---------------------------------------------- is_decimal_pid() { case "${1:-}" in @@ -816,6 +856,7 @@ do_start_locked() { if is_healthy; then START_FAILURE_KIND="none" printf 'Shim already running and strictly ready on port %s.\n' "$SHIM_PORT" >&2 + print_auth_line return 0 fi if supervisor_running; then @@ -851,6 +892,7 @@ do_start_locked() { START_FAILURE_KIND="none" ACTIVE_LAUNCH_PID="" printf 'Shim started and strictly ready on port %s.\n' "$SHIM_PORT" >&2 + print_auth_line return 0 fi # The detached process needs a scheduling turn to create supervisor.pid. @@ -1103,6 +1145,7 @@ do_status() { if is_healthy; then printf 'STATUS: running and strictly ready (port %s)\n' "$SHIM_PORT" printf 'HEALTH: %s\n' "$HEALTH_JSON" + print_auth_line return 0 fi if supervisor_running; then diff --git a/tests/bash/start_shim.bats b/tests/bash/start_shim.bats index 5a71d6b7..efc15bb5 100644 --- a/tests/bash/start_shim.bats +++ b/tests/bash/start_shim.bats @@ -64,6 +64,16 @@ class Handler(BaseHTTPRequestHandler): "reasoning_effort": None, "text_verbosity": "high", } + # v1.3.0 (A1-R4): optional auth block, driven by FAKE_HEALTH_AUTH_STATE. + auth_state = os.environ.get("FAKE_HEALTH_AUTH_STATE") + if auth_state: + auth = {"state": auth_state} + days = os.environ.get("FAKE_HEALTH_AUTH_DAYS") + if days: + auth["days_left"] = json.loads(days) + if auth_state not in ("valid", "n/a"): + auth["recovery"] = "codex login --device-auth" + payload["auth"] = auth raw = json.dumps(payload) body = raw.encode("utf-8") self.send_response(int(os.environ.get("FAKE_HEALTH_HTTP_STATUS", "200"))) @@ -97,6 +107,7 @@ PY unset FAKE_SHIM_BEHAVIOR FAKE_SHIM_CRASH_CODE FAKE_SHIM_START_DELAY FAKE_HEALTH_RAW unset FAKE_HEALTH_SERVICE FAKE_HEALTH_STATUS FAKE_HEALTH_VERSION unset FAKE_HEALTH_BACKEND_MODE FAKE_HEALTH_HTTP_STATUS DAAF_SHIM_TEST_NO_SETSID + unset FAKE_HEALTH_AUTH_STATE FAKE_HEALTH_AUTH_DAYS unset DAAF_SHIM_TEST_STOP_FAILURE DAAF_SHIM_TEST_RESULT_APPEND_FAILURE unset DAAF_SHIM_TEST_SIGNAL_RESULT_IN_PROGRESS SHIM_BACKEND_MODE } @@ -336,6 +347,75 @@ start_external_fixture() { assert_output --partial '"backend_mode": "chatgpt"' } +@test "readiness renders an expiring auth warning naming the recovery command" { + export SHIM_BACKEND_MODE=chatgpt + export FAKE_HEALTH_BACKEND_MODE=chatgpt + export FAKE_HEALTH_AUTH_STATE=expiring + export FAKE_HEALTH_AUTH_DAYS=1.5 + run "$MANAGER" --start + assert_manager_success + assert_output --partial "strictly ready" + assert_output --partial "ChatGPT subscription auth expires in 1.5 days" + assert_output --partial "codex login --device-auth" + # The same line is surfaced by --status. + run "$MANAGER" --status + assert_success + assert_output --partial "expires in 1.5 days" + assert_output --partial "codex login --device-auth" +} + +@test "readiness renders a dead auth warning for expired absent and unreadable" { + export SHIM_BACKEND_MODE=chatgpt + export FAKE_HEALTH_BACKEND_MODE=chatgpt + export FAKE_HEALTH_AUTH_STATE=expired + run "$MANAGER" --start + assert_manager_success + assert_output --partial "ChatGPT subscription auth is dead (expired)" + assert_output --partial "codex login --device-auth" +} + +@test "a valid auth state prints an informational line without a warning" { + export SHIM_BACKEND_MODE=chatgpt + export FAKE_HEALTH_BACKEND_MODE=chatgpt + export FAKE_HEALTH_AUTH_STATE=valid + export FAKE_HEALTH_AUTH_DAYS=280 + run "$MANAGER" --start + assert_manager_success + run "$MANAGER" --status + assert_success + assert_output --partial "ChatGPT subscription auth is valid (expires in 280 days)" + refute_output --partial "codex login --device-auth" + refute_output --partial "is dead" +} + +@test "the openai lane emits no auth line" { + export SHIM_BACKEND_MODE=openai + export FAKE_HEALTH_AUTH_STATE=n/a + run "$MANAGER" --start + assert_manager_success + run "$MANAGER" --status + assert_success + refute_output --partial "ChatGPT subscription auth" + refute_output --partial "codex login --device-auth" +} + +@test "auth line sanitizes an injecting days value into a bounded token" { + export SHIM_BACKEND_MODE=chatgpt + # A /health body whose auth.days_left carries an embedded newline + shell + # metacharacters (a forged-line / injection attempt). print_auth_line strips it + # to the bounded [0-9.-] token before output (the D4 injection convention), so no + # forged line and no metacharacters reach the terminal. The `\n` is a JSON string + # escape that jq -r decodes to a real newline in the shell value. + export FAKE_HEALTH_RAW='{"service":"daaf-anthropic-openai-shim","status":"ok","version":"1.2.12","backend_mode":"chatgpt","backend":"x","codex_home_present":true,"sanitize_tools":true,"reasoning_effort":null,"text_verbosity":"high","auth":{"state":"expiring","days_left":"2\nFORGED: fake auth line rm -rf slash"}}' + # Drive the readiness path (not --status, which additionally dumps the raw HEALTH + # JSON): the only auth output here is print_auth_line's own sanitized line. + run "$MANAGER" --start + assert_manager_success + assert_output --partial "expires in 2" + refute_output --partial "FORGED" + refute_output --partial "rm -rf" +} + @test "non-200 health response is reachable unexpected and blocks manager launch" { export FAKE_HEALTH_HTTP_STATUS=404 start_external_fixture diff --git a/tests/provider_shim/test_historical_regressions.py b/tests/provider_shim/test_historical_regressions.py index 11698e71..6245789c 100644 --- a/tests/provider_shim/test_historical_regressions.py +++ b/tests/provider_shim/test_historical_regressions.py @@ -28,6 +28,8 @@ "sanitize_tools", "reasoning_effort", "text_verbosity", + # v1.3.0 (A1-R4): read-only auth-validity snapshot block. + "auth", } @@ -348,12 +350,25 @@ def test_complete_stable_health_contract_both_lanes_and_overrides(self) -> None: self.assertEqual(set(health), _HEALTH_KEYS) self.assertEqual(health["service"], "daaf-anthropic-openai-shim") self.assertEqual(health["status"], "ok") - self.assertEqual(health["version"], "1.2.14") + self.assertEqual(health["version"], "1.3.0") self.assertEqual(health["backend_mode"], mode) self.assertEqual(health["sanitize_tools"], sanitize) self.assertEqual(health["reasoning_effort"], effort) self.assertEqual(health["text_verbosity"], verbosity) self.assertIs(health["codex_home_present"], True) + # v1.3.0 (A1-R4): the auth block. The openai lane does not use + # the codex OAuth store -> "n/a" (no token material). The chatgpt + # lane seeds a far-future fabricated token -> "valid", with an + # expiry/day-count and NO recovery command (recovery is present + # only for the four actionable states). + auth = health["auth"] + if mode == "openai": + self.assertEqual(auth, {"state": "n/a"}) + else: + self.assertEqual(auth["state"], "valid") + self.assertNotIn("recovery", auth) + self.assertIsInstance(auth["expires_at"], str) + self.assertGreater(auth["days_left"], 1) expected_backend = ( f"{backend.base_url}/v1" if mode == "openai" diff --git a/tests/provider_shim/test_reasoning_formatting.py b/tests/provider_shim/test_reasoning_formatting.py index 7ca72d09..348b0ca4 100644 --- a/tests/provider_shim/test_reasoning_formatting.py +++ b/tests/provider_shim/test_reasoning_formatting.py @@ -76,7 +76,7 @@ def test_openai_streaming_legacy_bytes_unchanged(self) -> None: ) self.assertEqual(shim.health.get("status"), "ok") self.assertEqual(shim.health.get("backend_mode"), "openai") - self.assertEqual(shim.health.get("version"), "1.2.14") + self.assertEqual(shim.health.get("version"), "1.3.0") def test_openai_nonstreaming_legacy_bytes_unchanged(self) -> None: scenario = central_multipart_scenario() diff --git a/tests/provider_shim/test_stream_hardening.py b/tests/provider_shim/test_stream_hardening.py index a8de588d..9121f708 100644 --- a/tests/provider_shim/test_stream_hardening.py +++ b/tests/provider_shim/test_stream_hardening.py @@ -1637,7 +1637,7 @@ def test_v1211_request_id_headers_and_lifecycle_correlation(self) -> None: startup_records = [ line for line in shim.captured_stderr().splitlines() - if "req_id=- phase=process shim v1.2.14 starting" in line + if "req_id=- phase=process shim v1.3.0 starting" in line ] self.assertEqual(len(startup_records), 1, startup_records) self.assertIn( diff --git a/tests/provider_shim/test_v130_auth_delegation.py b/tests/provider_shim/test_v130_auth_delegation.py index 4a674edb..75442271 100644 --- a/tests/provider_shim/test_v130_auth_delegation.py +++ b/tests/provider_shim/test_v130_auth_delegation.py @@ -263,5 +263,133 @@ def test_relogin_message_contains_recovery_command(self) -> None: self.assertIn(RECOVERY_COMMAND, module._RELOGIN_MSG) +class AuthHealthBlockTest(unittest.TestCase): + """A1-R4: the read-only `auth` block surfaced by /health. + + Exercises `_auth_health_block()` directly in-process across all five chatgpt-lane + states (valid | expiring | expired | absent | unreadable) plus the openai-lane + "n/a". The function is a pure reader of auth.json presence + the JWT exp; it never + returns token material, never writes, and never raises. Driving it in-process + (rather than through a RealShim subprocess) gives deterministic control of the + crafted exp and the store's readability. The far-future "valid" case is additionally + asserted end-to-end through /health by test_historical_regressions. + """ + + def setUp(self) -> None: + SCRATCH_ROOT.mkdir(parents=True, exist_ok=True) + self.codex_home = SCRATCH_ROOT / f"provider-shim-healthtest-{uuid.uuid4().hex}" + self.codex_home.mkdir(mode=0o700) + self.auth_path = self.codex_home / "auth.json" + + def tearDown(self) -> None: + if self.auth_path.exists(): + try: + self.auth_path.chmod(0o600) + except OSError: + pass + if self.codex_home.exists(): + shutil.rmtree(self.codex_home, ignore_errors=True) + + def _seed_auth(self, exp_offset_s: int) -> None: + auth = { + "OPENAI_API_KEY": FAKE_OPENAI_KEY, + "auth_mode": "chatgpt", + "tokens": { + "access_token": _make_fake_jwt(int(time.time()) + exp_offset_s), + "account_id": "acct_healthtest", + }, + } + self.auth_path.write_text(json.dumps(auth), encoding="utf-8") + self.auth_path.chmod(0o600) + + def _chatgpt_env(self): + return { + "SHIM_BACKEND_MODE": "chatgpt", + "SHIM_BACKEND_BASE_URL": "http://127.0.0.1:1", + "OPENAI_API_KEY": FAKE_OPENAI_KEY, + "CODEX_HOME": str(self.codex_home), + "SHIM_CODEX_BIN": str(FAKE_CODEX_BIN), + } + + def _block(self): + module = _load_fresh_shim() + return module._auth_health_block() + + def test_state_valid_far_future_token(self) -> None: + self._seed_auth(exp_offset_s=365 * 86400) + with mock.patch.dict(os.environ, self._chatgpt_env(), clear=False): + block = self._block() + self.assertEqual(block["state"], "valid") + self.assertNotIn("recovery", block) # recovery only for actionable states + self.assertIsInstance(block["expires_at"], str) + self.assertGreater(block["days_left"], 300) + + def test_state_expiring_within_48h_window(self) -> None: + self._seed_auth(exp_offset_s=24 * 3600) # inside the 48h expiring horizon + with mock.patch.dict(os.environ, self._chatgpt_env(), clear=False): + block = self._block() + self.assertEqual(block["state"], "expiring") + self.assertEqual(block["recovery"], RECOVERY_COMMAND) + self.assertLessEqual(block["days_left"], 2) + self.assertGreater(block["days_left"], 0) + + def test_state_expired_past_token(self) -> None: + self._seed_auth(exp_offset_s=-3600) + with mock.patch.dict(os.environ, self._chatgpt_env(), clear=False): + block = self._block() + self.assertEqual(block["state"], "expired") + self.assertEqual(block["recovery"], RECOVERY_COMMAND) + self.assertLessEqual(block["days_left"], 0) + + def test_state_absent_no_auth_file(self) -> None: + # No auth.json written; CODEX_HOME exists but the store is missing. + with mock.patch.dict(os.environ, self._chatgpt_env(), clear=False): + block = self._block() + self.assertEqual(block["state"], "absent") + self.assertEqual(block["recovery"], RECOVERY_COMMAND) + self.assertIsNone(block["expires_at"]) + self.assertIsNone(block["days_left"]) + + def test_state_unreadable_malformed_json(self) -> None: + # File present + readable, but its content yields no decodable access_token exp. + self.auth_path.write_text("this is not json{", encoding="utf-8") + self.auth_path.chmod(0o600) + with mock.patch.dict(os.environ, self._chatgpt_env(), clear=False): + block = self._block() + self.assertEqual(block["state"], "unreadable") + self.assertEqual(block["recovery"], RECOVERY_COMMAND) + self.assertIsNone(block["expires_at"]) + + def test_state_unreadable_non_jwt_token(self) -> None: + # Valid JSON, but the access_token is opaque (not a decodable JWT) -> unreadable. + auth = {"tokens": {"access_token": "opaque-not-a-jwt"}} + self.auth_path.write_text(json.dumps(auth), encoding="utf-8") + self.auth_path.chmod(0o600) + with mock.patch.dict(os.environ, self._chatgpt_env(), clear=False): + block = self._block() + self.assertEqual(block["state"], "unreadable") + + def test_openai_lane_reports_na(self) -> None: + # The API-key lane does not use the codex OAuth store -> "n/a", no token surface. + env = { + "SHIM_BACKEND_MODE": "openai", + "SHIM_BACKEND_BASE_URL": "http://127.0.0.1:1", + "OPENAI_API_KEY": FAKE_OPENAI_KEY, + } + with mock.patch.dict(os.environ, env, clear=False): + block = self._block() + self.assertEqual(block, {"state": "n/a"}) + + def test_block_never_contains_token_material(self) -> None: + # Credential-safety invariant: no state exposes the access_token value. + self._seed_auth(exp_offset_s=365 * 86400) + token = json.loads(self.auth_path.read_text(encoding="utf-8"))["tokens"][ + "access_token" + ] + with mock.patch.dict(os.environ, self._chatgpt_env(), clear=False): + block = self._block() + self.assertNotIn(token, json.dumps(block)) + + if __name__ == "__main__": unittest.main() From 92c3d3897ef9d6baeb63025e452dec7206e04f9a Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 20:27:37 +0000 Subject: [PATCH 14/66] test(deploy-smoke): add T0.9 auth-block verdict coverage for probe_auth_json (A1-iii) Covers PASS/WARN/FAIL state transitions, non-shim-route SKIP, and malformed/missing auth-block edge cases introduced in commit 3c3075e. Harness suite: Ran 50 tests OK (38 + 12 new), independently verified. Co-Authored-By: Claude Fable 5 --- tests/python/test_deploy_smoke.py | 145 ++++++++++++++++++++++++++++++ 1 file changed, 145 insertions(+) diff --git a/tests/python/test_deploy_smoke.py b/tests/python/test_deploy_smoke.py index de282c79..21dd7725 100644 --- a/tests/python/test_deploy_smoke.py +++ b/tests/python/test_deploy_smoke.py @@ -22,6 +22,7 @@ import uuid from pathlib import Path from unittest import mock +from urllib.error import URLError # --- Import path guard: mirror run_deploy_smoke.py so route_detection / # smoke_probes / run_deploy_smoke and the benchmarks harness all import @@ -216,6 +217,150 @@ def test_nonobject_and_malformed_health_json_fail_cleanly(self): self.assertIn("invalid", result.detail) +class AuthJsonProbeTests(unittest.TestCase): + """T0.9 (probe_auth_json): the shim /health `auth` block is authoritative for + chatgpt-route auth validity (v1.3.0 / A1-R6b). FAIL on + expired|absent|unreadable, WARN on expiring, PASS on valid, route-appropriate + SKIP off shim routes.""" + + def setUp(self): + # Supplementary filesystem evidence target (presence/readability only, + # never blocking on its own) — a real scratch dir keeps os.access() honest + # without requiring an actual auth.json to exist. + self.codex_home = _scratch_dir("codex_home") + + def tearDown(self): + import shutil + shutil.rmtree(self.codex_home, ignore_errors=True) + + def _route(self, route_name): + return route_detection.RouteInfo(route_name) + + def _env(self): + return {"CODEX_HOME": str(self.codex_home)} + + def _health(self, **auth_overrides): + auth = {"state": "valid", "days_left": 30} + auth.update(auth_overrides) + return {"auth": auth} + + def _probe(self, route_name, env, payload): + route = self._route(route_name) + with mock.patch.object( + smoke_probes, "urlopen", return_value=_HealthResponse(payload) + ): + return smoke_probes.probe_auth_json(route, env) + + def test_non_chatgpt_route_skips_without_hitting_health(self): + # SKIP fires on route alone; env/health are never consulted. + result = smoke_probes.probe_auth_json( + self._route(route_detection.ROUTE_OPENAI_API), {} + ) + self.assertEqual(result.verdict, Verdict.SKIP) + self.assertIn("Not the chatgpt-subscription route", result.detail) + + def test_missing_codex_home_fails_before_any_health_call(self): + result = smoke_probes.probe_auth_json( + self._route(route_detection.ROUTE_CHATGPT), {} + ) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("CODEX_HOME", result.detail) + + def test_valid_state_passes(self): + result = self._probe( + route_detection.ROUTE_CHATGPT, self._env(), + self._health(state="valid", days_left=45), + ) + self.assertEqual(result.verdict, Verdict.PASS, result.detail) + self.assertIn("days_left=45", result.detail) + + def test_expiring_state_warns(self): + result = self._probe( + route_detection.ROUTE_CHATGPT, self._env(), + self._health(state="expiring", days_left=2), + ) + self.assertEqual(result.verdict, Verdict.WARN) + self.assertIn("expires soon", result.detail) + self.assertIn("days_left=2", result.detail) + + def test_expired_state_fails(self): + result = self._probe( + route_detection.ROUTE_CHATGPT, self._env(), + self._health(state="expired", days_left=None), + ) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("auth is dead", result.detail) + self.assertIn("state=expired", result.detail) + + def test_absent_state_fails(self): + result = self._probe( + route_detection.ROUTE_CHATGPT, self._env(), + self._health(state="absent", days_left=None), + ) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("auth is dead", result.detail) + self.assertIn("state=absent", result.detail) + + def test_unreadable_state_fails(self): + result = self._probe( + route_detection.ROUTE_CHATGPT, self._env(), + self._health(state="unreadable", days_left=None), + ) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("auth is dead", result.detail) + self.assertIn("state=unreadable", result.detail) + + def test_missing_auth_block_fails_cleanly(self): + # Pre-v1.3.0 shim / broken build: no "auth" key at all in /health. + result = self._probe( + route_detection.ROUTE_CHATGPT, self._env(), + {"service": "daaf-anthropic-openai-shim", "status": "ok"}, + ) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("no auth block", result.detail) + + def test_auth_block_wrong_type_fails_cleanly(self): + # Malformed shape: "auth" present but not an object. + result = self._probe( + route_detection.ROUTE_CHATGPT, self._env(), + {"auth": "not-an-object"}, + ) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("no auth block", result.detail) + + def test_unrecognized_state_value_fails_without_crashing(self): + # Garbage state string outside the known vocabulary must not crash the + # probe and must be bounded (never reflected verbatim) in the detail. + garbage_state = "totally-bogus-state" + result = self._probe( + route_detection.ROUTE_CHATGPT, self._env(), + self._health(state=garbage_state, days_left=None), + ) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("unexpected auth state", result.detail) + self.assertIn("state=", result.detail) + self.assertNotIn(garbage_state, result.detail) + + def test_health_reported_na_state_on_chatgpt_route_fails(self): + # "n/a" is a KNOWN state string, but is unexpected specifically on the + # chatgpt route (it's the non-shim-route marker) — must still FAIL, not + # crash, and the known-but-wrong-here value is reflected verbatim. + result = self._probe( + route_detection.ROUTE_CHATGPT, self._env(), + self._health(state="n/a", days_left=None), + ) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("unexpected auth state", result.detail) + self.assertIn("state=n/a", result.detail) + + def test_health_endpoint_unreachable_fails(self): + route = self._route(route_detection.ROUTE_CHATGPT) + with mock.patch.object(smoke_probes, "urlopen", side_effect=URLError("refused")): + result = smoke_probes.probe_auth_json(route, self._env()) + self.assertEqual(result.verdict, Verdict.FAIL) + self.assertIn("cannot assess auth", result.detail) + + class TierDEnvSanitizationTests(unittest.TestCase): def test_removes_exactly_the_two_contaminants_and_preserves_the_rest(self): with mock.patch.dict(os.environ, { From cf052043e2e7e7e91a0b5686ddfce28bc8b76a69 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 20:57:05 +0000 Subject: [PATCH 15/66] fix(shim): v1.3.0 A1 review fix-it (health exp-overflow guard, bounded reap, doc sweep) Cycle-1 fixes from the three-angle A1 auth-delegation review: - _auth_health_block(): guard the JWT-exp computation against OverflowError/ OSError/ValueError (pathological numeric exp), classify as unreadable rather than raising - upholds the /health never-raises contract; +1 test - _run_codex_login_status(): bound the post-kill reap with wait_for(5s) so an unkillable child cannot stall the request path - host template + install guide + technical FAQ: drop the deleted SHIM_OAUTH_* seams and reframe the rotation-race caution around codex-delegated refresh (shim is now a pure reader; keep per-tool CODEX_HOME isolation advice); also align the Codex-plugin warning block mechanism wording (install guide ~L1236) with the v1.3.0 delegation model Version unchanged (1.3.0). Gates independently verified: Ran 178 tests OK, bats 46/46, deploy-smoke harness 50/50. Co-Authored-By: Claude Fable 5 --- scripts/host/environment_settings_example.txt | 13 ++++-- .../provider_shim/anthropic_openai_shim.py | 46 +++++++++++++------ .../test_v130_auth_delegation.py | 14 ++++++ .../01_installation_and_quickstart.md | 4 +- user_reference/07_faq_technical.md | 2 +- 5 files changed, 58 insertions(+), 21 deletions(-) diff --git a/scripts/host/environment_settings_example.txt b/scripts/host/environment_settings_example.txt index 283583ab..a15b95f6 100644 --- a/scripts/host/environment_settings_example.txt +++ b/scripts/host/environment_settings_example.txt @@ -477,12 +477,15 @@ # --- To activate, remove the leading '#': --- #CODEX_HOME=/home/appuser/.claude/codex-daaf # -# The two OAuth override vars below are TEST / STAGING overrides only -- the -# production defaults are correct (auth.openai.com/oauth/token and the codex -# first-party client_id). Leave them unset for normal use. +# CODEX DELEGATION (shim v1.3.0+, chatgpt lane): the shim delegates token refresh +# to the codex CLI (the single writer of auth.json). The two knobs below tune that +# delegated call; the defaults are correct for the standard image. +# SHIM_CODEX_BIN: path to the codex binary the shim invokes for the delegated auth +# refresh (default "codex" -- resolved on PATH). +# SHIM_CODEX_TIMEOUT_S: seconds to wait for that invocation (default 30). # --- To activate, remove the leading '#' on the lines you need: --- -#SHIM_OAUTH_TOKEN_URL= -#SHIM_OAUTH_CLIENT_ID= +#SHIM_CODEX_BIN= +#SHIM_CODEX_TIMEOUT_S= # # Troubleshooting (inside the container): # codex --version (confirm the standard-image Codex CLI is present) diff --git a/scripts/provider_shim/anthropic_openai_shim.py b/scripts/provider_shim/anthropic_openai_shim.py index e2897d8b..d2ac248d 100644 --- a/scripts/provider_shim/anthropic_openai_shim.py +++ b/scripts/provider_shim/anthropic_openai_shim.py @@ -103,6 +103,13 @@ # expiring, PASS on valid (SKIP on non-shim routes). # * A1-R7: SHIM_CODEX_BIN/SHIM_CODEX_TIMEOUT_S documented in the Config block and # start_shim.sh help/env passthrough. +# Review fix-it (post-A1 three-angle review; version unchanged): guard +# _auth_health_block() against a pathological JWT exp overflowing platform time_t +# (classify "unreadable" rather than raising, upholding the never-raises +# contract); bound the post-kill codex reap with a 5s wait_for so an unkillable +# child cannot stall the request path; and swept the deleted SHIM_OAUTH_* seams +# out of the host settings template and the user docs (install guide + technical +# FAQ), which still framed the now-delegated refresh as a shim-side rotation race. # The two historical v1.2.5 comments describing the now-deleted Python OAuth path # carry inline "[superseded in v1.3.0 ...]" markers so version history stays # accurate without asserting present-tense behavior that no longer exists. @@ -2102,8 +2109,12 @@ async def _run_codex_login_status(): except ProcessLookupError: pass # Reap the killed child so it does not linger as a zombie or leak its pipes. + # Bound the reap: an unkillable child (e.g. stuck in uninterruptible sleep) + # must not stall the request path — log a coarse event and keep going. try: - await proc.communicate() + await asyncio.wait_for(proc.communicate(), timeout=5) + except asyncio.TimeoutError: + log.warning("event=codex_login_status_reap_timeout timeout_s=5") except Exception: pass return @@ -5959,18 +5970,27 @@ def _auth_health_block(): return {"state": "unreadable", "expires_at": None, "days_left": None, "recovery": _AUTH_RECOVERY_CMD} now = time.time() - remaining = exp - now - if remaining <= 0: - state = "expired" - elif remaining <= _AUTH_EXPIRING_WINDOW_S: - state = "expiring" - else: - state = "valid" - block = { - "state": state, - "expires_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(exp)), - "days_left": round(remaining / 86400, 1), - } + try: + remaining = exp - now + if remaining <= 0: + state = "expired" + elif remaining <= _AUTH_EXPIRING_WINDOW_S: + state = "expiring" + else: + state = "valid" + block = { + "state": state, + "expires_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(exp)), + "days_left": round(remaining / 86400, 1), + } + except (OverflowError, OSError, ValueError): + # A pathological numeric exp (corrupt/partial-write auth.json, or a non-codex + # writer yielding a garbage-but-numeric exp) can overflow the platform time_t + # inside time.gmtime/strftime. The file reached us and parsed, but the auth + # data is unusable -> classify "unreadable" (an actionable state, recovery + # command included), preserving this function's "never raises" contract. + return {"state": "unreadable", "expires_at": None, "days_left": None, + "recovery": _AUTH_RECOVERY_CMD} if state != "valid": block["recovery"] = _AUTH_RECOVERY_CMD return block diff --git a/tests/provider_shim/test_v130_auth_delegation.py b/tests/provider_shim/test_v130_auth_delegation.py index 75442271..42e32029 100644 --- a/tests/provider_shim/test_v130_auth_delegation.py +++ b/tests/provider_shim/test_v130_auth_delegation.py @@ -369,6 +369,20 @@ def test_state_unreadable_non_jwt_token(self) -> None: block = self._block() self.assertEqual(block["state"], "unreadable") + def test_state_unreadable_overflow_exp(self) -> None: + # A garbage-but-numeric exp (corrupt/partial-write auth.json, or a non-codex + # writer) can overflow platform time_t inside time.gmtime/strftime. The block + # must classify "unreadable" rather than raise -- the /health probe must never + # 500. (_jwt_exp returns int(10**19); time.gmtime(10**19) raises OverflowError.) + auth = {"tokens": {"access_token": _make_fake_jwt(10 ** 19)}} + self.auth_path.write_text(json.dumps(auth), encoding="utf-8") + self.auth_path.chmod(0o600) + with mock.patch.dict(os.environ, self._chatgpt_env(), clear=False): + block = self._block() # must not raise + self.assertEqual(block["state"], "unreadable") + self.assertEqual(block["recovery"], RECOVERY_COMMAND) + self.assertIsNone(block["expires_at"]) + def test_openai_lane_reports_na(self) -> None: # The API-key lane does not use the codex OAuth store -> "n/a", no token surface. env = { diff --git a/user_reference/01_installation_and_quickstart.md b/user_reference/01_installation_and_quickstart.md index 8ab8be7e..5b1cc7cd 100644 --- a/user_reference/01_installation_and_quickstart.md +++ b/user_reference/01_installation_and_quickstart.md @@ -1201,7 +1201,7 @@ A healthy ChatGPT lane logs `backend_mode=chatgpt` at startup and then healthy ` | `codex: command not found` | The image predates the universal Codex installation or was built from a stale Dockerfile. Update DAAF and rebuild (`rebuild_daaf.sh` / `rebuild_daaf.ps1`) so the current Dockerfile is used; `DAAF_DEV=1` is not required. | | Shim log tells you to re-login | The OAuth token refresh failed permanently — run `codex login --device-auth` again inside the container. | -**Running more than one container.** Each DAAF container does its **own** `codex login`, which creates an **independent refresh-token grant** — there is no credential collision between containers. They share only your ChatGPT **usage pool** (the 5-hour and weekly caps), so running two in parallel simply draws that pool down faster. This is the clean way to run parallel DAAF instances. (The subtler case is running several codex-based tools — the `codex` CLI, the [Codex plugin](#using-the-codex-plugin-for-claude-code), the shim — inside a *single* container off the *same* login: that can rarely trigger a refresh-token-rotation race. To isolate them, give each its own `codex login` under a separate `CODEX_HOME`; `CODEX_HOME` and the `SHIM_OAUTH_TOKEN_URL` / `SHIM_OAUTH_CLIENT_ID` override variables in `environment_settings_example.txt` are the hooks for that. None of this is needed for a normal single-tool setup.) +**Running more than one container.** Each DAAF container does its **own** `codex login`, which creates an **independent refresh-token grant** — there is no credential collision between containers. They share only your ChatGPT **usage pool** (the 5-hour and weekly caps), so running two in parallel simply draws that pool down faster. This is the clean way to run parallel DAAF instances. (The subtler case is running several codex-based tools — the `codex` CLI, the [Codex plugin](#using-the-codex-plugin-for-claude-code), the shim — inside a *single* container off the *same* login. As of shim v1.3.0 the shim is a pure *reader* of `auth.json`: it never rewrites the store, delegating every token refresh to the `codex` CLI — the single writer per `CODEX_HOME` — so the old shim-versus-other-tool rotation race is structurally eliminated. What stays unverified is *codex-versus-codex*: two independent codex invocations against one `CODEX_HOME` (for instance the plugin and the shim's delegated refresh firing at nearly the same moment). The safe posture is therefore still to give each codex-based tool its own `codex login` under a separate `CODEX_HOME`. None of this is needed for a normal single-tool setup.) See the [technical FAQ entry on the ChatGPT subscription lane](07_faq_technical.md#q-can-i-use-my-chatgpt-subscription-instead-of-an-openai-api-key-option-f). @@ -1233,7 +1233,7 @@ As with the shim's ChatGPT lane, this requires **device-code login enabled in yo 1. **Two containers (the natural pairing with the ChatGPT lane).** Run the plugin in a container on the **Anthropic route** (Claude subscription/API), and keep the ChatGPT-subscription **shim lane** in a *separate* container. Because each container has its own config volume, each has its own Codex login — the plugin and the shim never touch the same token. This pairs naturally with the [shared-workspace setup above](#sharing-one-research-workspace-across-two-installs-advanced). 2. **One Anthropic-route container plus the plugin.** If the container is *not* running the ChatGPT shim lane, nothing else is consuming the Codex login, so the plugin is the sole consumer and there is no contention. -> **Do not add the Codex plugin to a container that is already using the ChatGPT-subscription shim lane.** In that lane the shim is *itself* a consumer of your Codex login, continuously reading and refreshing the OAuth token in `auth.json`. Adding a second consumer of the same login — the plugin, or for that matter running the `codex` CLI by hand — means two processes can rotate the refresh token at nearly the same moment, and a lost rotation **permanently invalidates the login** (a `refresh_token_reused` lockout that forces a fresh `codex login`). If you genuinely must run both in one container, give each tool its own **separate `CODEX_HOME`** with its own **separate `codex login --device-auth`**, so they never share a token file. Either way, remember that all logins on one ChatGPT account draw from the **same usage pool** (the 5-hour and weekly caps), so adding a second Codex consumer spends that pool faster. +> **Do not add the Codex plugin to a container that is already using the ChatGPT-subscription shim lane.** In that lane the shim is *itself* a consumer of your Codex login: it reads the OAuth token from `auth.json` and (as of shim v1.3.0) triggers refreshes by invoking the `codex` CLI against its `CODEX_HOME` rather than rewriting the file itself. Adding a second consumer of the same login — the plugin, or for that matter running the `codex` CLI by hand — is exactly the unverified *codex-versus-codex* case described above: two codex invocations can still rotate the refresh token at nearly the same moment, and a lost rotation **permanently invalidates the login** (a `refresh_token_reused` lockout that forces a fresh `codex login`). If you genuinely must run both in one container, give each tool its own **separate `CODEX_HOME`** with its own **separate `codex login --device-auth`**, so they never share a token file. Either way, remember that all logins on one ChatGPT account draw from the **same usage pool** (the 5-hour and weekly caps), so adding a second Codex consumer spends that pool faster. #### Known limitations of GPT sessions (both lanes) diff --git a/user_reference/07_faq_technical.md b/user_reference/07_faq_technical.md index 6cc3d544..7580fb68 100644 --- a/user_reference/07_faq_technical.md +++ b/user_reference/07_faq_technical.md @@ -355,7 +355,7 @@ The shim reads the OAuth token from `auth.json` and **refreshes it automatically **Context ceiling on this lane.** The Codex subscription backend caps the usable input window much lower than the model's true 1M window — measured at ~370,000 tokens for `gpt-5.6-sol` (2026-07-16). This is a hard backend limit on the Codex route, not a client setting: no `[1m]` hint or local declaration can raise it. Set `CLAUDE_CODE_MAX_CONTEXT_TOKENS=370000` on this lane (not the API-key lane's `1050000`), so Claude Code and DAAF measure usage against the real ceiling. DAAF applies this automatically and keeps automatic compaction off; the backend itself remains the ultimate hard limit. If you exceed it, you'll get a clean "context length exceeded" error rather than a confusing retry loop. -**Parallel use.** Running more than one container is the clean way to parallelize: each container does its own `codex login` (an independent token grant — no collision), and they share only your ChatGPT usage pool, so running them in parallel just draws that pool down faster. For several codex tools inside a *single* container, give each its own login under a separate `CODEX_HOME` to avoid a rare token-rotation race. +**Parallel use.** Running more than one container is the clean way to parallelize: each container does its own `codex login` (an independent token grant — no collision), and they share only your ChatGPT usage pool, so running them in parallel just draws that pool down faster. For several codex-based tools inside a *single* container, the shim itself is no longer a factor — as of v1.3.0 it never writes `auth.json`, delegating every refresh to the `codex` CLI (the single writer) — but two independent *codex* invocations sharing one `CODEX_HOME` stay unverified, so give each tool its own login under a separate `CODEX_HOME` to stay safe. **Optional reading — under the hood.** Both lanes handle tools and message content the same way; only the login method and endpoint differ. If the backend ever sends something malformed or out of order, the shim fails loudly rather than quietly passing along corrupted output. A few small formatting and logging details differ between the two lanes, but nothing you need to configure or worry about. From 69eeee07cc2a39b906479a87d7bb044747e58639 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 22:24:33 +0000 Subject: [PATCH 16/66] feat(shim+statusline): v1.3.1 - Codex plan-usage quota in the statusline Shim caches the latest chatgpt-lane quota snapshot to an install-shared quota_state.json (atomic, 0600, fail-open); context-bar.sh renders it as the Plan-usage segment on shim-lane sessions (window labels from window-minutes, stale-window drop rule, zero-secondary omitted). Co-Authored-By: Claude Fable 5 --- .claude/scripts/context-bar.sh | 126 +++++++++++ .../provider_shim/anthropic_openai_shim.py | 70 +++++- tests/bash/context_bar.bats | 107 ++++++++- .../test_historical_regressions.py | 2 +- .../test_reasoning_formatting.py | 2 +- tests/provider_shim/test_stream_hardening.py | 2 +- tests/provider_shim/test_v131_quota_state.py | 213 ++++++++++++++++++ 7 files changed, 517 insertions(+), 5 deletions(-) create mode 100644 tests/provider_shim/test_v131_quota_state.py diff --git a/.claude/scripts/context-bar.sh b/.claude/scripts/context-bar.sh index f3e8b3dc..7955f124 100755 --- a/.claude/scripts/context-bar.sh +++ b/.claude/scripts/context-bar.sh @@ -39,6 +39,21 @@ is_canonical_positive_decimal() { [[ "$value" == "$max_value" || "$value" < "$max_value" ]] } +# As above, but ALSO accepts a canonical zero. The codex quota fields +# used_percent and *-reset-after-seconds are legitimately 0 (e.g. an unused +# window or an all-zero secondary), which is_canonical_positive_decimal rejects. +# Same length/lexical bounds run before any arithmetic so oversized or +# non-canonical values can never reach an arithmetic context. +is_canonical_nonneg_decimal() { + local value="${1:-}" + local max_value="9223372036854775807" + [[ "$value" =~ ^(0|[1-9][0-9]*)$ ]] || return 1 + [[ ${#value} -lt ${#max_value} ]] && return 0 + [[ ${#value} -gt ${#max_value} ]] && return 1 + # shellcheck disable=SC2071 + [[ "$value" == "$max_value" || "$value" < "$max_value" ]] +} + input=$(cat) # One jq pass validates and normalizes every field before joining it with an @@ -466,6 +481,16 @@ fmt_reset() { printf '(%dh%dm)' $((remain / 3600)) $(((remain % 3600) / 60)) fi } +window_label_for() { + # $1 = window length in whole minutes (a validated positive integer). Echoes a + # compact window label: divisible by 1440 -> "d" (10080 -> 7d), else divisible + # by 60 -> "h" (300 -> 5h), else "m". Pure arithmetic on a pre-validated int. + local min="$1" + if [[ $((min % 1440)) -eq 0 ]]; then printf '%dd' $((min / 1440)) + elif [[ $((min % 60)) -eq 0 ]]; then printf '%dh' $((min / 60)) + else printf '%dm' "$min" + fi +} if [[ -n "$rl_5h" || -n "$rl_7d" ]]; then rl_body="" if [[ -n "$rl_5h" ]]; then @@ -488,6 +513,107 @@ if [[ -n "$rl_5h" || -n "$rl_7d" ]]; then [[ -n "$rl_body" ]] && rl_seg=" ${C_GRAY}|${C_RESET} ${C_GRAY}Plan usage:${C_RESET} ${rl_body}" fi +# --- Codex (ChatGPT-subscription) Plan-usage fallback --- +# On shim-lane sessions the native Anthropic rate-limit payload fields are absent; +# instead the provider shim caches its latest ChatGPT-subscription quota snapshot to +# scripts/provider_shim/logs/quota_state.json (see anthropic_openai_shim.py +# _write_quota_state — install-shared under the shared-/daaf assumption). Render that +# file as the SAME unchanged "Plan usage:" segment, reusing rl_color_for/fmt_reset. +# Gate (belt-and-braces): the exact shim lane AND no native rate limits in the payload +# (if the payload somehow carries native limits, they win and this read is skipped). +# Everything here is pure string/arithmetic + one local file read; any parse or +# validation failure yields no segment and leaves the rest of the statusline untouched. +if [[ -z "$rl_seg" && \ + "${DAAF_PROVIDER_SHIM:-}" == "openai" && \ + "${SHIM_BACKEND_MODE:-}" == "chatgpt" && \ + -z "$rl_5h" && -z "$rl_7d" ]]; then + # State-file path: default derived from this script's own location + # (.claude/scripts -> repo root -> scripts/provider_shim/logs/quota_state.json). + # DAAF_QUOTA_STATE_FILE is a deterministic-test seam mirroring + # DAAF_CONTEXT_BAR_CACHE_DIR: Bats points it at project scratch so tests never + # depend on a live shim's log directory. + quota_state_file="${DAAF_QUOTA_STATE_FILE:-}" + if [[ -z "$quota_state_file" ]]; then + cb_scripts_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P 2>/dev/null)" || cb_scripts_dir="" + if [[ -n "$cb_scripts_dir" ]]; then + quota_state_file="${cb_scripts_dir%/.claude/scripts}/scripts/provider_shim/logs/quota_state.json" + fi + fi + + if [[ -n "$quota_state_file" && -f "$quota_state_file" ]] && command -v jq >/dev/null 2>&1; then + # One jq pass: require a top-level object, emit captured_at plus the six + # numeric primary/secondary fields as an ASCII-unit-separated record. String + # fields carrying any control character (including the \x1f delimiter) become + # empty so untrusted content cannot shift the fixed-order read; every emitted + # field is validated numerically in Bash below before any use. + q_captured="" + q_p_pct="" + q_p_win="" + q_p_reset="" + q_s_pct="" + q_s_win="" + q_s_reset="" + codex_parsed=0 + if IFS=$'\x1f' read -r q_captured q_p_pct q_p_win q_p_reset q_s_pct q_s_win q_s_reset \ + < <(jq -er ' + def num_string: + if type == "string" then (if test("[[:cntrl:]]") then "" else . end) + elif type == "number" then tostring + else "" end; + if type != "object" then error("quota_state must be an object") + else + [ (if (.captured_at? | type) == "number" and + (.captured_at | floor) == .captured_at + then (.captured_at | tostring) else "" end), + (.primary_used_pct? | num_string), + (.primary_window_min? | num_string), + (.primary_reset_s? | num_string), + (.secondary_used_pct? | num_string), + (.secondary_window_min? | num_string), + (.secondary_reset_s? | num_string) ] + | join("\u001f") + end + ' "$quota_state_file" 2>/dev/null); then + codex_parsed=1 + fi + + # Primary window is mandatory: captured_at + used-percent (0 allowed) + a + # positive window length + reset-after (0 allowed) must all validate. + if [[ "$codex_parsed" -eq 1 ]] && \ + is_canonical_nonneg_decimal "$q_captured" && \ + is_canonical_nonneg_decimal "$q_p_pct" && \ + is_canonical_positive_decimal "$q_p_win" && \ + is_canonical_nonneg_decimal "$q_p_reset"; then + now_epoch=$(date +%s 2>/dev/null) || now_epoch="" + primary_reset_epoch=$((q_captured + q_p_reset)) + # Staleness: an expired primary window means the cached percent is stale, so + # drop the ENTIRE segment (no display beats a wrong display). + if [[ "$now_epoch" =~ ^[0-9]+$ && "$primary_reset_epoch" -gt "$now_epoch" ]]; then + cbody="" + pwl=$(window_label_for "$q_p_win") + pcolor=$(rl_color_for "$q_p_pct") + pcd=$(fmt_reset "$primary_reset_epoch") + cbody+="${pcolor}${pwl}:${q_p_pct}%${C_RESET}" + [[ -n "$pcd" ]] && cbody+="${C_GRAY}${pcd}${C_RESET}" + # Secondary renders only when its window is > 0 and its percent + # validates; live data shows an all-zero secondary, which is omitted. + if is_canonical_positive_decimal "$q_s_win" && \ + is_canonical_nonneg_decimal "$q_s_pct"; then + swl=$(window_label_for "$q_s_win") + scolor=$(rl_color_for "$q_s_pct") + cbody+=" ${scolor}${swl}:${q_s_pct}%${C_RESET}" + if is_canonical_nonneg_decimal "$q_s_reset"; then + secondary_reset_epoch=$((q_captured + q_s_reset)) + scd=$(fmt_reset "$secondary_reset_epoch") + [[ -n "$scd" ]] && cbody+="${C_GRAY}${scd}${C_RESET}" + fi + fi + rl_seg=" ${C_GRAY}|${C_RESET} ${C_GRAY}Plan usage:${C_RESET} ${cbody}" + fi + fi + fi +fi + # Build output: Model (effort) | Dir | Branch | Context [| Plan usage] output="${C_ACCENT}${model_disp}${C_GRAY} | 📁${dir}" [[ -n "$branch" ]] && output+=" | 🔀${branch}" diff --git a/scripts/provider_shim/anthropic_openai_shim.py b/scripts/provider_shim/anthropic_openai_shim.py index d2ac248d..ab31363d 100644 --- a/scripts/provider_shim/anthropic_openai_shim.py +++ b/scripts/provider_shim/anthropic_openai_shim.py @@ -63,6 +63,19 @@ # * GET /health endpoint for the manager's idempotency/status checks. # # Changelog: +# v1.3.1 (2026-07-20): Statusline Plan-usage telemetry (additive, no behavior change). +# On every chatgpt-lane 2xx (same guard as the D1 quota_snapshot line), the shim now +# also caches the latest quota snapshot to /quota_state.json — an atomic, +# 0600, absolutely-fail-open write next to shim.log. The file is a single JSON object +# {captured_at:, }; the READER (context-bar.sh) computes the absolute reset instant as +# captured_at + primary_reset_s. context-bar.sh renders it as the "Plan usage:" +# segment on shim-lane sessions (window labels derived from window-minutes; stale- +# window drop rule; zero secondary omitted). INSTALL-SHARED: under the shared-/daaf +# multi-install assumption this state file lives on the shared mount and is read by +# any install's statusline (auth under $HOME stays per-install). Pure additive +# telemetry: no new env vars, no off-switch; a write failure is swallowed and never +# touches the response path. SHIM_VERSION -> 1.3.1. # v1.3.0 (2026-07-20): ChatGPT-lane auth refresh DELEGATED to the codex CLI # (Tier 3 A1; minor bump — behavior + config-surface change). The shim is now a # pure READER of $CODEX_HOME/auth.json; codex is the SINGLE WRITER. This deletes @@ -794,6 +807,7 @@ import random import asyncio import logging +import tempfile import threading import urllib.parse from collections import OrderedDict @@ -803,7 +817,7 @@ import uvicorn # --- Config --- -SHIM_VERSION = "1.3.0" +SHIM_VERSION = "1.3.1" SHIM_SERVICE_ID = "daaf-anthropic-openai-shim" SHIM_PORT = int(os.environ.get("SHIM_PORT", "4141")) @@ -1251,6 +1265,56 @@ def _normalize_http_version(value): return normalized if normalized in {"HTTP/1.0", "HTTP/1.1", "HTTP/2"} else "unknown" +# v1.3.1: quota-state cache for the statusline Plan-usage segment. The file lives in +# the shim's own logs/ directory — the SAME directory start_shim.sh writes shim.log to +# (SCRIPT_DIR/logs); it is derived here from __file__ because the shim itself logs only +# to stderr and holds no log-directory constant of its own. Under the shared-/daaf +# multi-install assumption this file is INSTALL-SHARED across containers on the mount: +# any install's context-bar.sh reads it to render "Plan usage:" (auth under $HOME stays +# per-install). The shim records only captured_at (write-time epoch); the reader does the +# clock math (absolute reset = captured_at + primary_reset_s). +_QUOTA_STATE_DIR = os.path.join( + os.path.dirname(os.path.abspath(__file__)), "logs" +) +_QUOTA_STATE_PATH = os.path.join(_QUOTA_STATE_DIR, "quota_state.json") + + +def _write_quota_state(snapshot): + # Absolutely fail-open telemetry: a quota-state write must NEVER affect, delay, or + # raise on the response path. Every failure mode (unwritable dir, replace error, + # serialization surprise) is swallowed, leaving at most one debug line and no stale + # temp sibling. Pure additive — there is deliberately no env var and no off-switch. + tmp_path = None + try: + payload = {"captured_at": int(time.time())} + # snapshot carries the 11 raw header-string values (or "-" when the header was + # absent), exactly as emitted on the quota_snapshot line. + payload.update(snapshot) + data = json.dumps(payload).encode("utf-8") + # Atomic publish via a uniquely-named sibling temp file + os.replace(): a reader + # sees either the old file or the fully-written new one, never a partial write. + # mkstemp creates the temp with mode 0600 already (matching the shim.log + # permission discipline) and a unique name, so concurrent installs sharing this + # directory cannot clobber each other's in-flight temp file. + fd, tmp_path = tempfile.mkstemp( + prefix="quota_state.", suffix=".tmp", dir=_QUOTA_STATE_DIR + ) + with os.fdopen(fd, "wb") as handle: + handle.write(data) + os.replace(tmp_path, _QUOTA_STATE_PATH) + tmp_path = None + except Exception: + try: + log.debug("quota_state write skipped (fail-open)") + except Exception: + pass + if tmp_path is not None: + try: + os.unlink(tmp_path) + except Exception: + pass + + def _record_upstream_headers(response): state = _request_state() if state is None or response is None: @@ -1279,6 +1343,10 @@ def _record_upstream_headers(response): value = response.headers.get(header_name) snapshot[field] = value if value is not None else "-" _lifecycle_event("quota_snapshot", **snapshot) + # v1.3.1: also cache the snapshot to an install-shared JSON state file so the + # statusline (context-bar.sh) can render a "Plan usage:" segment on shim-lane + # sessions. Fail-open; never affects the response path. + _write_quota_state(snapshot) def _record_upstream_first_event(): diff --git a/tests/bash/context_bar.bats b/tests/bash/context_bar.bats index cf1a96f1..02ed8b54 100755 --- a/tests/bash/context_bar.bats +++ b/tests/bash/context_bar.bats @@ -21,6 +21,7 @@ setup() { CTX_CACHE="${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-ctx-window-${FAKE_SESSION}" MODEL_CACHE="${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-model-${FAKE_SESSION}" OR_CACHE="${DAAF_CONTEXT_BAR_CACHE_DIR}/claude-or-models-${FAKE_SESSION}" + QUOTA_STATE_FILE="${SCRATCH_DIR}/quota_state.json" mkdir -p "$MOCK_BIN" "$DAAF_CONTEXT_BAR_CACHE_DIR" cat > "${MOCK_BIN}/curl" <<'MOCK_CURL' #!/usr/bin/env bash @@ -33,7 +34,16 @@ MOCK_CURL unset DAAF_PROVIDER_SHIM unset SHIM_BACKEND_MODE export CONTEXT_BAR_SH FAKE_SESSION SCRATCH_DIR MOCK_BIN - export DAAF_CONTEXT_BAR_CACHE_DIR CTX_CACHE MODEL_CACHE OR_CACHE + export DAAF_CONTEXT_BAR_CACHE_DIR CTX_CACHE MODEL_CACHE OR_CACHE QUOTA_STATE_FILE +} + +# Write a quota_state.json fixture. Args: captured_at, primary_used_pct, +# primary_window_min, primary_reset_s, secondary_used_pct, secondary_window_min, +# secondary_reset_s. The four non-numeric snapshot fields are fixed constants the +# reader ignores. Mirrors the shim's _write_quota_state output shape. +_write_quota_state() { + printf '{"captured_at":%s,"plan_type":"pro","active_limit":"premium","primary_used_pct":"%s","primary_window_min":"%s","primary_reset_s":"%s","secondary_used_pct":"%s","secondary_window_min":"%s","secondary_reset_s":"%s","credits_has":"False","credits_balance":"0","credits_unlimited":"False"}' \ + "$1" "$2" "$3" "$4" "$5" "$6" "$7" > "$QUOTA_STATE_FILE" } teardown() { @@ -501,3 +511,98 @@ JSON refute_output --partial "of 370k tokens" done } + +# ========================================================================= +# Codex (ChatGPT-subscription) Plan-usage segment (v1.3.1) +# ------------------------------------------------------------------------- +# On shim-lane sessions (DAAF_PROVIDER_SHIM=openai AND SHIM_BACKEND_MODE=chatgpt) +# with no native rate limits in the payload, context-bar.sh reads the shim's +# quota_state.json (DAAF_QUOTA_STATE_FILE test seam) and renders "Plan usage:". +# The reader computes the absolute reset instant as captured_at + primary_reset_s. +# ========================================================================= + +@test "codex Plan-usage renders 7d label pct and countdown from a fresh state file" { + local now + now="$(date +%s)" + # primary_reset_s ~4.65 days -> a future reset -> a "(4d..h)" countdown. + _write_quota_state "$now" 73 10080 402168 0 0 0 + run env DAAF_PROVIDER_SHIM=openai SHIM_BACKEND_MODE=chatgpt \ + DAAF_QUOTA_STATE_FILE="$QUOTA_STATE_FILE" \ + bash "$CONTEXT_BAR_SH" < <(_payload "gpt-5.6-sol") + assert_success + assert_output --partial "Plan usage:" + assert_output --partial "7d:73%" + assert_output --partial "(4d" +} + +@test "codex Plan-usage omits an all-zero secondary window" { + local now + now="$(date +%s)" + _write_quota_state "$now" 73 10080 402168 0 0 0 + run env DAAF_PROVIDER_SHIM=openai SHIM_BACKEND_MODE=chatgpt \ + DAAF_QUOTA_STATE_FILE="$QUOTA_STATE_FILE" \ + bash "$CONTEXT_BAR_SH" < <(_payload "gpt-5.6-sol") + assert_success + assert_output --partial "7d:73%" + refute_output --partial "5h:" +} + +@test "codex Plan-usage renders a positive secondary window when present" { + local now + now="$(date +%s)" + # secondary: 300 min -> "5h", 5%, reset 600s in the future. + _write_quota_state "$now" 73 10080 402168 5 300 600 + run env DAAF_PROVIDER_SHIM=openai SHIM_BACKEND_MODE=chatgpt \ + DAAF_QUOTA_STATE_FILE="$QUOTA_STATE_FILE" \ + bash "$CONTEXT_BAR_SH" < <(_payload "gpt-5.6-sol") + assert_success + assert_output --partial "7d:73%" + assert_output --partial "5h:5%" +} + +@test "codex Plan-usage drops the whole segment when the primary window is stale" { + local past + past="$(( $(date +%s) - 100000 ))" + # captured_at + 50s is well in the past -> stale -> segment dropped entirely. + _write_quota_state "$past" 73 10080 50 0 0 0 + run env DAAF_PROVIDER_SHIM=openai SHIM_BACKEND_MODE=chatgpt \ + DAAF_QUOTA_STATE_FILE="$QUOTA_STATE_FILE" \ + bash "$CONTEXT_BAR_SH" < <(_payload "gpt-5.6-sol") + assert_success + refute_output --partial "Plan usage:" + # Rest of the bar is intact (chatgpt lane caps gpt-5.6-sol at 370k). + assert_output --partial "of 370k tokens" +} + +@test "codex Plan-usage is absent on malformed state JSON and the bar stays intact" { + printf '{"captured_at":' > "$QUOTA_STATE_FILE" + run env DAAF_PROVIDER_SHIM=openai SHIM_BACKEND_MODE=chatgpt \ + DAAF_QUOTA_STATE_FILE="$QUOTA_STATE_FILE" \ + bash "$CONTEXT_BAR_SH" < <(_payload "gpt-5.6-sol") + assert_success + refute_output --partial "Plan usage:" + assert_output --partial "of 370k tokens" +} + +@test "codex Plan-usage is gated off on a native session even with a state file present" { + local now + now="$(date +%s)" + _write_quota_state "$now" 73 10080 402168 0 0 0 + # No shim lane env (setup unsets DAAF_PROVIDER_SHIM/SHIM_BACKEND_MODE). + run env DAAF_QUOTA_STATE_FILE="$QUOTA_STATE_FILE" \ + bash "$CONTEXT_BAR_SH" < <(_payload "gpt-5.6-sol") + assert_success + refute_output --partial "Plan usage:" + assert_output --partial "of 1050k tokens" +} + +@test "codex Plan-usage renders a zero primary percent" { + local now + now="$(date +%s)" + _write_quota_state "$now" 0 10080 402168 0 0 0 + run env DAAF_PROVIDER_SHIM=openai SHIM_BACKEND_MODE=chatgpt \ + DAAF_QUOTA_STATE_FILE="$QUOTA_STATE_FILE" \ + bash "$CONTEXT_BAR_SH" < <(_payload "gpt-5.6-sol") + assert_success + assert_output --partial "7d:0%" +} diff --git a/tests/provider_shim/test_historical_regressions.py b/tests/provider_shim/test_historical_regressions.py index 6245789c..bf11a4c8 100644 --- a/tests/provider_shim/test_historical_regressions.py +++ b/tests/provider_shim/test_historical_regressions.py @@ -350,7 +350,7 @@ def test_complete_stable_health_contract_both_lanes_and_overrides(self) -> None: self.assertEqual(set(health), _HEALTH_KEYS) self.assertEqual(health["service"], "daaf-anthropic-openai-shim") self.assertEqual(health["status"], "ok") - self.assertEqual(health["version"], "1.3.0") + self.assertEqual(health["version"], "1.3.1") self.assertEqual(health["backend_mode"], mode) self.assertEqual(health["sanitize_tools"], sanitize) self.assertEqual(health["reasoning_effort"], effort) diff --git a/tests/provider_shim/test_reasoning_formatting.py b/tests/provider_shim/test_reasoning_formatting.py index 348b0ca4..ea208f95 100644 --- a/tests/provider_shim/test_reasoning_formatting.py +++ b/tests/provider_shim/test_reasoning_formatting.py @@ -76,7 +76,7 @@ def test_openai_streaming_legacy_bytes_unchanged(self) -> None: ) self.assertEqual(shim.health.get("status"), "ok") self.assertEqual(shim.health.get("backend_mode"), "openai") - self.assertEqual(shim.health.get("version"), "1.3.0") + self.assertEqual(shim.health.get("version"), "1.3.1") def test_openai_nonstreaming_legacy_bytes_unchanged(self) -> None: scenario = central_multipart_scenario() diff --git a/tests/provider_shim/test_stream_hardening.py b/tests/provider_shim/test_stream_hardening.py index 9121f708..86a5eb4b 100644 --- a/tests/provider_shim/test_stream_hardening.py +++ b/tests/provider_shim/test_stream_hardening.py @@ -1637,7 +1637,7 @@ def test_v1211_request_id_headers_and_lifecycle_correlation(self) -> None: startup_records = [ line for line in shim.captured_stderr().splitlines() - if "req_id=- phase=process shim v1.3.0 starting" in line + if "req_id=- phase=process shim v1.3.1 starting" in line ] self.assertEqual(len(startup_records), 1, startup_records) self.assertIn( diff --git a/tests/provider_shim/test_v131_quota_state.py b/tests/provider_shim/test_v131_quota_state.py new file mode 100644 index 00000000..e747a374 --- /dev/null +++ b/tests/provider_shim/test_v131_quota_state.py @@ -0,0 +1,213 @@ +"""Deterministic contracts for the v1.3.1 quota-state cache write. + +The v1.3.1 shim, on every chatgpt-lane 2xx, additionally caches the quota snapshot +to ``/quota_state.json`` (an atomic, 0600, absolutely-fail-open write) so +the statusline can render a "Plan usage:" segment on shim-lane sessions. These tests +exercise the write helper and its guard IN-PROCESS: the production shim module is +imported via importlib (the same seam the harness uses for its ownership probe) so the +module-level state-file path and ``SHIM_BACKEND_MODE`` can be monkeypatched to a tmp +directory — no subprocess, and the real ``scripts/provider_shim/logs/`` directory is +never touched. + +1. chatgpt-lane 2xx writes the state file: correct JSON keys, ``captured_at`` within + the test wall-clock bounds, header values passed through, absent headers -> "-". +2. The guard holds: the openai lane and a chatgpt non-2xx write no file. +3. Overwrite semantics: a second 2xx replaces the content and leaves no ``.tmp`` + sibling behind. +4. Write failure is swallowed: an unwritable log dir or an ``os.replace`` that raises + leaves the response path unaffected (no exception escapes) and no stale ``.tmp``. +""" + +from __future__ import annotations + +import importlib.util +import json +import os +import tempfile +import time +import unittest +import uuid +from pathlib import Path + +import httpx + +from ._loopback_harness import PRODUCTION_SHIM + + +def _load_shim_module(): + """Import a fresh instance of the production shim in-process (no subprocess).""" + module_name = f"provider_shim_quota_state_probe_{uuid.uuid4().hex}" + spec = importlib.util.spec_from_file_location(module_name, PRODUCTION_SHIM) + if spec is None or spec.loader is None: + raise RuntimeError("could not load production shim for quota-state probe") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +# Synthetic subscription-quota header surface modeled on the live chatgpt-lane +# capture. Secondary-window headers are intentionally omitted so the +# "absent header -> field renders as -" contract is exercised end to end. +QUOTA_HEADERS = { + "x-codex-plan-type": "pro", + "x-codex-active-limit": "premium", + "x-codex-primary-used-percent": "73", + "x-codex-primary-window-minutes": "10080", + "x-codex-primary-reset-after-seconds": "402168", + "x-codex-credits-has-credits": "False", + "x-codex-credits-balance": "0", + "x-codex-credits-unlimited": "False", +} + +_ALL_SNAPSHOT_FIELDS = ( + "plan_type", + "active_limit", + "primary_used_pct", + "primary_window_min", + "primary_reset_s", + "secondary_used_pct", + "secondary_window_min", + "secondary_reset_s", + "credits_has", + "credits_balance", + "credits_unlimited", +) + + +class _FakeResponse: + """Minimal stand-in for the httpx response _record_upstream_headers reads.""" + + def __init__(self, status_code, headers=None, http_version="HTTP/1.1"): + self.status_code = status_code + self.headers = httpx.Headers(headers or {}) + self.http_version = http_version + + +class ProviderShimV131QuotaStateTests(unittest.TestCase): + maxDiff = 12000 + + def setUp(self) -> None: + self.module = _load_shim_module() + # Redirect the install-shared state file into a per-test tmp directory so the + # real scripts/provider_shim/logs/ directory is never written during tests. + self._tmp = tempfile.TemporaryDirectory() + self.addCleanup(self._tmp.cleanup) + self.state_dir = Path(self._tmp.name) + self.state_path = self.state_dir / "quota_state.json" + self.module._QUOTA_STATE_DIR = str(self.state_dir) + self.module._QUOTA_STATE_PATH = str(self.state_path) + + def _drive_record(self, mode, response) -> None: + # Exercise the real call site (_record_upstream_headers) with a live request + # state, exactly as the request path would, under the chosen backend lane. + self.module.SHIM_BACKEND_MODE = mode + state = self.module._RequestLifecycleState() + token = self.module._REQUEST_STATE.set(state) + try: + self.module._record_upstream_headers(response) + finally: + self.module._REQUEST_STATE.reset(token) + + def _tmp_siblings(self): + return sorted(p.name for p in self.state_dir.glob("quota_state.*.tmp")) + + def test_chatgpt_2xx_writes_state_file_with_expected_fields(self) -> None: + before = int(time.time()) + self._drive_record("chatgpt", _FakeResponse(200, QUOTA_HEADERS)) + after = int(time.time()) + + self.assertTrue(self.state_path.exists(), "state file was not written") + payload = json.loads(self.state_path.read_text(encoding="utf-8")) + + # captured_at is an int recorded at write time, within the test's wall clock. + self.assertIsInstance(payload["captured_at"], int) + self.assertGreaterEqual(payload["captured_at"], before) + self.assertLessEqual(payload["captured_at"], after) + + # Every one of the 11 snapshot fields is present. + for field in _ALL_SNAPSHOT_FIELDS: + self.assertIn(field, payload) + + # Present headers pass through as their raw string values. + self.assertEqual(payload["plan_type"], "pro") + self.assertEqual(payload["active_limit"], "premium") + self.assertEqual(payload["primary_used_pct"], "73") + self.assertEqual(payload["primary_window_min"], "10080") + self.assertEqual(payload["primary_reset_s"], "402168") + self.assertEqual(payload["credits_has"], "False") + self.assertEqual(payload["credits_balance"], "0") + self.assertEqual(payload["credits_unlimited"], "False") + + # Absent secondary-window headers render as "-". + self.assertEqual(payload["secondary_used_pct"], "-") + self.assertEqual(payload["secondary_window_min"], "-") + self.assertEqual(payload["secondary_reset_s"], "-") + + # No temp sibling is left behind after a successful atomic publish. + self.assertEqual(self._tmp_siblings(), []) + + def test_openai_lane_writes_no_state_file(self) -> None: + self._drive_record("openai", _FakeResponse(200, QUOTA_HEADERS)) + self.assertFalse(self.state_path.exists()) + self.assertEqual(self._tmp_siblings(), []) + + def test_chatgpt_non_2xx_writes_no_state_file(self) -> None: + self._drive_record("chatgpt", _FakeResponse(429, QUOTA_HEADERS)) + self.assertFalse(self.state_path.exists()) + self.assertEqual(self._tmp_siblings(), []) + + def test_second_2xx_overwrites_and_leaves_no_temp_sibling(self) -> None: + self._drive_record("chatgpt", _FakeResponse(200, QUOTA_HEADERS)) + first = json.loads(self.state_path.read_text(encoding="utf-8")) + self.assertEqual(first["primary_used_pct"], "73") + + second_headers = dict(QUOTA_HEADERS) + second_headers["x-codex-primary-used-percent"] = "88" + second_headers["x-codex-secondary-used-percent"] = "5" + second_headers["x-codex-secondary-window-minutes"] = "300" + second_headers["x-codex-secondary-reset-after-seconds"] = "600" + self._drive_record("chatgpt", _FakeResponse(200, second_headers)) + + replaced = json.loads(self.state_path.read_text(encoding="utf-8")) + self.assertEqual(replaced["primary_used_pct"], "88") + self.assertEqual(replaced["secondary_used_pct"], "5") + self.assertEqual(replaced["secondary_window_min"], "300") + self.assertGreaterEqual(replaced["captured_at"], first["captured_at"]) + + # Overwrite still leaves no stale temp sibling. + self.assertEqual(self._tmp_siblings(), []) + + def test_write_failure_on_unwritable_dir_is_swallowed(self) -> None: + # Point the state dir at a path that is a FILE, not a directory, so mkstemp + # raises NotADirectoryError. The response path must be unaffected: no + # exception escapes and no file is created. + not_a_dir = self.state_dir / "regular_file" + not_a_dir.write_text("x", encoding="utf-8") + self.module._QUOTA_STATE_DIR = str(not_a_dir / "logs") + self.module._QUOTA_STATE_PATH = str(not_a_dir / "logs" / "quota_state.json") + + # Must not raise. + self._drive_record("chatgpt", _FakeResponse(200, QUOTA_HEADERS)) + self.assertFalse((not_a_dir / "logs" / "quota_state.json").exists()) + + def test_os_replace_raising_is_swallowed_and_leaves_no_temp(self) -> None: + # Directly exercise the helper with os.replace monkeypatched to raise: the + # write must be swallowed and the uniquely-named temp sibling cleaned up. + original_replace = os.replace + + def _raising_replace(src, dst): # noqa: ANN001 + raise RuntimeError("synthetic replace failure") + + os.replace = _raising_replace + try: + # Must not raise. + self.module._write_quota_state({field: "-" for field in _ALL_SNAPSHOT_FIELDS}) + finally: + os.replace = original_replace + + self.assertFalse(self.state_path.exists()) + self.assertEqual(self._tmp_siblings(), []) + + +if __name__ == "__main__": + unittest.main() From dba855c411ff962f79fd41ce509641e274dc98d5 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 22:48:16 +0000 Subject: [PATCH 17/66] test+docs(shim): v1.3.1 review fix-it (0600 assertion, hostile-string bats case, FAQ statusline scope, comment sweep) Co-Authored-By: Claude Fable 5 --- .claude/scripts/context-bar.sh | 5 ++++ tests/bash/context_bar.bats | 27 ++++++++++++++++++-- tests/provider_shim/test_v131_quota_state.py | 5 ++++ user_reference/07_faq_technical.md | 2 +- 4 files changed, 36 insertions(+), 3 deletions(-) diff --git a/.claude/scripts/context-bar.sh b/.claude/scripts/context-bar.sh index 7955f124..eb699f86 100755 --- a/.claude/scripts/context-bar.sh +++ b/.claude/scripts/context-bar.sh @@ -50,6 +50,7 @@ is_canonical_nonneg_decimal() { [[ "$value" =~ ^(0|[1-9][0-9]*)$ ]] || return 1 [[ ${#value} -lt ${#max_value} ]] && return 0 [[ ${#value} -gt ${#max_value} ]] && return 1 + # Equal-length canonical decimals are intentionally compared lexicographically before arithmetic. # shellcheck disable=SC2071 [[ "$value" == "$max_value" || "$value" < "$max_value" ]] } @@ -585,6 +586,10 @@ if [[ -z "$rl_seg" && \ is_canonical_positive_decimal "$q_p_win" && \ is_canonical_nonneg_decimal "$q_p_reset"; then now_epoch=$(date +%s 2>/dev/null) || now_epoch="" + # Both operands are already bounded to int64-representable canonical decimals + # (is_canonical_nonneg_decimal above), but if their sum were ever to overflow + # bash's 64-bit arithmetic, it wraps negative -- which safely fails the + # `-gt now_epoch` staleness check below rather than corrupting the display. primary_reset_epoch=$((q_captured + q_p_reset)) # Staleness: an expired primary window means the cached percent is stale, so # drop the ENTIRE segment (no display beats a wrong display). diff --git a/tests/bash/context_bar.bats b/tests/bash/context_bar.bats index 02ed8b54..21e8fcf5 100755 --- a/tests/bash/context_bar.bats +++ b/tests/bash/context_bar.bats @@ -39,8 +39,10 @@ MOCK_CURL # Write a quota_state.json fixture. Args: captured_at, primary_used_pct, # primary_window_min, primary_reset_s, secondary_used_pct, secondary_window_min, -# secondary_reset_s. The four non-numeric snapshot fields are fixed constants the -# reader ignores. Mirrors the shim's _write_quota_state output shape. +# secondary_reset_s. The five snapshot fields the reader never consults are fixed +# constants here (plan_type, active_limit, credits_has, credits_balance, +# credits_unlimited -- credits_balance is numeric-looking but still reader-ignored). +# Mirrors the shim's _write_quota_state output shape. _write_quota_state() { printf '{"captured_at":%s,"plan_type":"pro","active_limit":"premium","primary_used_pct":"%s","primary_window_min":"%s","primary_reset_s":"%s","secondary_used_pct":"%s","secondary_window_min":"%s","secondary_reset_s":"%s","credits_has":"False","credits_balance":"0","credits_unlimited":"False"}' \ "$1" "$2" "$3" "$4" "$5" "$6" "$7" > "$QUOTA_STATE_FILE" @@ -606,3 +608,24 @@ JSON assert_success assert_output --partial "7d:0%" } + +@test "codex Plan-usage segment is dropped when primary_used_pct carries a hostile control-char escape" { + local now + now="$(date +%s)" + # Valid JSON (a properly-escaped unicode sequence, not a raw control byte) + # whose parsed value embeds an ESC/ANSI sequence in primary_used_pct. + # Bypasses _write_quota_state (which only accepts plain values) to write + # the JSON literal directly; the double backslash below makes printf emit + # a literal backslash-u-0-0-1-b sequence in the file, mirroring how the + # earlier newline/NUL bats fixtures embed JSON unicode escapes rather + # than raw bytes. + printf '{"captured_at":%s,"plan_type":"pro","active_limit":"premium","primary_used_pct":"73\\u001b[31m","primary_window_min":"10080","primary_reset_s":"402168","secondary_used_pct":"0","secondary_window_min":"0","secondary_reset_s":"0","credits_has":"False","credits_balance":"0","credits_unlimited":"False"}' \ + "$now" > "$QUOTA_STATE_FILE" + run env DAAF_PROVIDER_SHIM=openai SHIM_BACKEND_MODE=chatgpt \ + DAAF_QUOTA_STATE_FILE="$QUOTA_STATE_FILE" \ + bash "$CONTEXT_BAR_SH" < <(_payload "gpt-5.6-sol") + assert_success + refute_output --partial "Plan usage:" + # Rest of the bar is intact (chatgpt lane caps gpt-5.6-sol at 370k). + assert_output --partial "of 370k tokens" +} diff --git a/tests/provider_shim/test_v131_quota_state.py b/tests/provider_shim/test_v131_quota_state.py index e747a374..2f322fec 100644 --- a/tests/provider_shim/test_v131_quota_state.py +++ b/tests/provider_shim/test_v131_quota_state.py @@ -23,6 +23,7 @@ import importlib.util import json import os +import stat import tempfile import time import unittest @@ -143,6 +144,10 @@ def test_chatgpt_2xx_writes_state_file_with_expected_fields(self) -> None: self.assertEqual(payload["secondary_window_min"], "-") self.assertEqual(payload["secondary_reset_s"], "-") + # Locks the telemetry-hygiene contract (0600) against a future refactor + # to the write call that silently loses the mkstemp-inherited mode. + self.assertEqual(stat.S_IMODE(os.stat(self.state_path).st_mode), 0o600) + # No temp sibling is left behind after a successful atomic publish. self.assertEqual(self._tmp_siblings(), []) diff --git a/user_reference/07_faq_technical.md b/user_reference/07_faq_technical.md index 7580fb68..434746c9 100644 --- a/user_reference/07_faq_technical.md +++ b/user_reference/07_faq_technical.md @@ -926,7 +926,7 @@ You can also read the `.qmd` source directly in the browser-based VS Code editor Two at-a-glance displays that Claude Code renders for you, both customized by DAAF. -**The status bar** runs along the bottom of your session. Reading across, it shows the active model (with its reasoning-effort level), the current working directory, the Git branch you're on, a context-usage meter for the session, and -- if you're on a Claude subscription -- your rate-limit windows (how much of your rolling 5-hour and 7-day usage allowance you've used up). That rate-limit readout is specific to Claude subscription plans for now; on API-key or other-provider sessions those segments simply don't appear. +**The status bar** runs along the bottom of your session. Reading across, it shows the active model (with its reasoning-effort level), the current working directory, the Git branch you're on, a context-usage meter for the session, and -- if you're on a Claude subscription -- your rate-limit windows (how much of your rolling 5-hour and 7-day usage allowance you've used up). That readout also appears on ChatGPT-subscription (provider-shim) sessions, where it shows your Codex plan windows instead (e.g. the 7-day window with its used-percent and reset countdown, sourced from the shim's cached quota snapshot). On API-key and other provider sessions, that segment simply doesn't appear. **The agent panel** shows up whenever DAAF has dispatched specialists to work in the background. It gives you one row per running specialist, and each row reports that specialist's type (what kind of agent it is), its model, its current status, its token count, and its own context meter -- so you can watch several agents make progress at once. From c2c17fe58f1ea3b16f72a201866c1d3b4669c242 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Mon, 20 Jul 2026 23:54:30 +0000 Subject: [PATCH 18/66] ci(scripts): add provider-shim-tests job and tests/provider_shim path filters (Tier 4) New provider-shim-tests job runs the full offline 184-test shim suite (loopback harness + fake-codex stub; httpx/uvicorn pinned to container versions) on ubuntu-latest with a 15-minute cap. tests/provider_shim/** added to push and pull_request path filters; header job list corrected to eleven jobs. Co-Authored-By: Claude Fable 5 --- .github/workflows/ci-scripts.yml | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci-scripts.yml b/.github/workflows/ci-scripts.yml index 186c181d..7bd7ccc7 100644 --- a/.github/workflows/ci-scripts.yml +++ b/.github/workflows/ci-scripts.yml @@ -3,7 +3,7 @@ # Validates all .sh and .ps1 scripts across the repository. # Companion workflow: ci-integration.yml runs Docker integration tests weekly. # -# Nine jobs: +# Eleven jobs: # 1. lint-bash - ShellCheck on all .sh files (ubuntu) # 2. lint-powershell - PSScriptAnalyzer on all .ps1 files (ubuntu/pwsh 7) # 3. hygiene-checks - Executable bits, LF endings, script pair parity @@ -12,6 +12,7 @@ # 6. pester-tests - Pester tests for .ps1 scripts (ubuntu + windows) # 7. daaf-conventions - Custom DAAF-specific lint checks (ubuntu) # 7b. deployment-smoke - Harness self-tests (provider-free, stdlib only) +# 7c. provider-shim-tests - Provider-shim unit suite (offline loopback harness) # 8. smoke-tests - Cross-platform dry-run smoke tests (ubuntu + macos + windows) # 9. bats-bash32 - Host-script BATS suite under real Bash 3.2 (ubuntu container) # @@ -32,6 +33,7 @@ on: - 'scripts/**.py' - 'scripts/**.R' - 'tests/python/**' + - 'tests/provider_shim/**' - 'tests/bash/**.py' - 'tests/fixtures/**' pull_request: @@ -46,6 +48,7 @@ on: - 'scripts/**.py' - 'scripts/**.R' - 'tests/python/**' + - 'tests/provider_shim/**' - 'tests/bash/**.py' - 'tests/fixtures/**' @@ -281,6 +284,29 @@ jobs: run: python3 -m unittest discover -s tests/python -p 'test_deploy_smoke.py' -v + # --------------------------------------------------------------------------- + # Job 7c: Provider-shim unit suite (offline: loopback harness + fake-codex + # stub; no network, no credentials, no real codex binary). The only + # third-party imports across the shim and its tests are httpx and uvicorn, + # pinned here to the versions validated in the DAAF container image. + # --------------------------------------------------------------------------- + provider-shim-tests: + name: Provider Shim Unit Tests + runs-on: ubuntu-latest + # The suite normally completes in ~3 minutes; the cap guards runner minutes + # against a hung asyncio/socket test rather than the 6-hour GitHub default. + timeout-minutes: 15 + steps: + - name: Checkout repository + uses: actions/checkout@v5 + + - name: Install shim runtime dependencies + run: python3 -m pip install httpx==0.28.1 uvicorn==0.51.0 + + - name: Run provider-shim unit tests + run: python3 -m unittest discover -s tests/provider_shim -t tests -p 'test_*.py' -v + + # --------------------------------------------------------------------------- # Job 8: Cross-platform dry-run smoke tests (no Docker needed) # --------------------------------------------------------------------------- From 24fba3c7bde842226f9439b03b509b8949cda63e Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Tue, 21 Jul 2026 00:03:42 +0000 Subject: [PATCH 19/66] test(shim): resolve harness DAAF_ROOT from repo location for CI portability The loopback harness hardcoded /daaf as the repo root; every test spawns the production shim through that path, which exists in the container but not on a CI runner checkout. Resolve from __file__ instead (identical in-container). Review finding from the Tier 4 CI wiring (c2c17fe). Co-Authored-By: Claude Fable 5 --- tests/provider_shim/_loopback_harness.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/provider_shim/_loopback_harness.py b/tests/provider_shim/_loopback_harness.py index aec21285..e7dac1ba 100644 --- a/tests/provider_shim/_loopback_harness.py +++ b/tests/provider_shim/_loopback_harness.py @@ -35,7 +35,11 @@ from unittest import mock -DAAF_ROOT = Path("/daaf") +# Repo root resolved from this file's location (tests/provider_shim/ -> repo +# root), not hardcoded to /daaf: in the DAAF container the two are identical, +# but on a CI runner the checkout lands wherever the workspace is (e.g. +# $GITHUB_WORKSPACE), and the harness must find the production shim there. +DAAF_ROOT = Path(__file__).resolve().parents[2] PRODUCTION_SHIM = DAAF_ROOT / "scripts/provider_shim/anthropic_openai_shim.py" # v1.3.0: the fake-codex stub the shim's delegated refresh spawns instead of the # real codex CLI. Injected via SHIM_CODEX_BIN. Behavior is driven by FAKE_CODEX_MODE. From 6527a972da26ae928dc0e648aaba3afc3f613f66 Mon Sep 17 00:00:00 2001 From: DAAF Container Date: Tue, 21 Jul 2026 00:51:07 +0000 Subject: [PATCH 20/66] fix(statusline): harden context-accounting renderers (escape, ID validation, classifier, cache races) Ports confirmed findings from an external hardening audit across the two statusline renderers and the context-reporter hook (applied to the installed hook from host; delivered via proposed copy since the hooks dir is edit-protected). - Escape-safe rendering: ANSI-C color literals plus literal %s render (drop %b) so untrusted model/branch/task text cannot inject terminal control bytes; strip C0/C1 from untrusted fields. - Identifier allowlist before path construction; closed-set GPT flagship classifier for physical-window mapping; full-transcript usage/model recovery (drop tail -50); numerator overflow bound before multiply; suppressed redirection-open diagnostics. - context-bar.sh also: atomic ctx-window cache publish, OpenRouter catalog validation with private temp, plan-usage percentage clamp. context-reporter.sh also: parent-model isolation, future-timestamp gate guard, and a DAAF_CONTEXT_REPORTER_CACHE_DIR test-isolation seam (default /tmp, inert in production). Tests: context_bar.bats +14 cases (53/53), subagent_bar.bats +9 (54/54); reporter suite 69/69. Three-angle review passed. Co-Authored-By: Claude Fable 5 --- .claude/hooks/context-reporter.sh | 190 ++++++++++++++++++----- .claude/scripts/context-bar.sh | 149 +++++++++++++----- .claude/scripts/subagent-bar.sh | 206 ++++++++++++++++++------- tests/bash/context_bar.bats | 244 ++++++++++++++++++++++++++++++ tests/bash/subagent_bar.bats | 192 +++++++++++++++++++++++ 5 files changed, 848 insertions(+), 133 deletions(-) diff --git a/.claude/hooks/context-reporter.sh b/.claude/hooks/context-reporter.sh index 98e5a27e..f74ed6dd 100755 --- a/.claude/hooks/context-reporter.sh +++ b/.claude/hooks/context-reporter.sh @@ -24,8 +24,10 @@ # # Performance: # Uses transcript byte-size + mtime signatures to reuse model caches without -# rescanning unchanged files. Changed transcripts are resolved from `tail -50`, -# which also bounds usage parsing to the end of the JSONL. +# rescanning unchanged files. When a transcript has changed, the model and +# usage are recovered from a full scan of the JSONL (last non-synthetic model, +# last positive usage sum) so a long run of trailing zero-token shim +# placeholders cannot hide the real values. # # Subagent support: # settings.json PreToolUse hooks also fire for tool calls made BY subagents. @@ -46,6 +48,17 @@ # orchestrator's utilization into the subagent's context, causing subagents # to falsely throttle or refuse work at HIGH/CRITICAL. # +# Parent-model isolation: +# The parent/session model is derived ONLY from a genuinely-parent source — +# the parent model cache (/tmp/claude-model-) refreshed from the +# parent's MAIN transcript. It is NEVER inferred by re-parsing a path that is +# itself a subagent transcript. When Claude Code passes the subagent's own +# transcript directly (basename == agent-.jsonl) there is no parent +# transcript to scan, so the parent model comes from the cache alone (or is +# left empty). A measured subagent is then mapped by ITS OWN model/window and +# never assumed to share the session model (see Convention 10 in the +# hardening spec). +# # Threshold tier (see calculate()): # The severity thresholds are keyed on the model identity of the agent being # measured — main-session measurements use the session model; subagent @@ -86,6 +99,17 @@ is_canonical_positive_decimal() { [[ "$value" == "$max_value" || "$value" < "$max_value" ]] } +# Bounded identifier allowlist (Convention 2). Leading alnum, then +# alnum/dot/underscore/hyphen, max 128 chars. Rejects slashes, traversal (..), +# control chars, leading dash, empty, and >128. Every session_id / agent_id must +# pass this BEFORE it is interpolated into any transcript or /tmp cache path, so +# an attacker-shaped id can neither escape the /tmp namespace nor the subagent +# transcript subtree. A present-but-invalid id is rejected exactly like an +# absent one (key-presence contract). +is_safe_id() { + [[ "${1:-}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ ]] +} + # Return a signature that changes when a transcript is appended or rewritten. # GNU stat supplies byte size plus nanosecond-resolution mtime in one probe. transcript_signature() { @@ -98,9 +122,13 @@ transcript_signature() { # paths consumed by audit-log.sh and statusline components. The neighboring # .transcript-signature sidecar is the commit marker: unchanged signatures reuse # a nonempty bare cache without parsing; changed signatures refresh from the last -# nonempty, non-synthetic model. Both files are replaced through same-directory -# temporary siblings, with the signature moved last so interrupted writes force -# a safe refresh on the next invocation. +# nonempty, non-synthetic model via a FULL scan of the JSONL (Convention 4 — +# a long tail of zero-token/synthetic placeholders cannot hide the real model). +# Both files are replaced through same-directory temporary siblings, with the +# signature moved last so interrupted writes force a safe refresh on the next +# invocation. When called with an empty transcript (e.g. the parent transcript +# is unavailable in the direct-subagent branch) it returns the cached value +# without scanning or writing. resolve_model_cache() { local transcript="${1:-}" local cache="${2:-}" @@ -119,16 +147,20 @@ resolve_model_cache() { fi if [[ -n "$transcript" && -f "$transcript" ]]; then - model=$(tail -n 50 "$transcript" 2>/dev/null | jq -rs ' + model=$(jq -rs ' [.[] | (.message.model // empty) | select(. != "" and . != "")] | last // empty - ' 2>/dev/null) || model="" + ' < "$transcript" 2>/dev/null) || model="" fi [[ -z "$model" ]] && model="$cached_model" if [[ -n "$model" && -n "$signature" ]]; then - if printf '%s' "$model" > "$model_tmp" 2>/dev/null && - printf '%s' "$signature" > "$signature_tmp" 2>/dev/null && + # Convention 6: wrap the redirection so an open() failure on the temp + # target cannot leak "No such file"/"Permission denied" to the display + # stream (redirections apply left-to-right, so a bare `> f 2>/dev/null` + # would still print the open() diagnostic before 2> takes effect). + if { printf '%s' "$model" > "$model_tmp"; } 2>/dev/null && + { printf '%s' "$signature" > "$signature_tmp"; } 2>/dev/null && mv "$model_tmp" "$cache" 2>/dev/null && mv "$signature_tmp" "$signature_cache" 2>/dev/null; then : @@ -146,47 +178,84 @@ SESSION_ID=$(echo "$INPUT" | jq -r '.session_id // "default"' 2>/dev/null) || SE TRANSCRIPT_PATH=$(echo "$INPUT" | jq -r '.transcript_path // empty' 2>/dev/null) || TRANSCRIPT_PATH="" AGENT_ID=$(echo "$INPUT" | jq -r '.agent_id // empty' 2>/dev/null) || AGENT_ID="" +# Convention 2: allowlist ids BEFORE any path/cache interpolation. Fail open +# (emit nothing, exit 0) on a rejected id — do not construct or read/write any +# path built from it. SESSION_ID is always used (every /tmp cache is keyed on +# it); AGENT_ID is only checked when present (its absence marks the main +# session). A present-but-invalid id is rejected like an absent one. +is_safe_id "$SESSION_ID" || exit 0 +if [[ -n "$AGENT_ID" ]]; then + is_safe_id "$AGENT_ID" || exit 0 +fi + +# Production caches live in /tmp. The override is a deterministic-test seam so +# Bats can exercise the gate/model/window cache reads and writes inside project +# scratch without touching live session state. The default is /tmp so +# production behavior is byte-identical; the name follows the per-script +# DAAF_