Report an aggregate token-volume-weighted compression percentage

README.md

@@ -128,7 +128,9 @@ headroom_enabled: true
 headroom_install_method: docker # only if your runner lacks a usable Python; pip is the default
 ```
 
-**Savings are visible on the run itself, not just reachable by a caller reading an output.** A `Verify Headroom actually proxied this run` step reads the proxy's own `/stats` after the run and writes a `### Headroom context compression` section straight to the job's Step Summary — the tab everyone already looks at — reporting the request count, tokens saved, and average compression percentage (or a clear warning if nothing was proxied despite `headroom_enabled: true`). The same numbers are also exposed as the `headroom_tokens_saved` and `headroom_compression_percent` outputs, for a caller that wants to gate on or record them programmatically, alongside the existing `headroom_proxied` boolean.
+**Savings are visible on the run itself, not just reachable by a caller reading an output.** A `Verify Headroom actually proxied this run` step reads the proxy's own `/stats` after the run and writes a `### Headroom context compression` section straight to the job's Step Summary — the tab everyone already looks at — reporting the request count, tokens saved, and two distinct compression figures (or a clear warning if nothing was proxied despite `headroom_enabled: true`). The same numbers are also exposed as the `headroom_tokens_saved`, `headroom_compression_percent`, and `headroom_total_percent_saved` outputs, for a caller that wants to gate on or record them programmatically, alongside the existing `headroom_proxied` boolean.
+
+**Two different compression percentages are reported, because they answer different questions.** `headroom_compression_percent` is Headroom's own `avg_compression_pct` — the unweighted average of each individual request's own compression ratio, so one huge, highly-compressible request among many small, barely-compressible ones drags the average down even though most of the actual token volume was saved. `headroom_total_percent_saved` is the aggregate instead, computed as `total_tokens_removed / total_tokens_before * 100` across every proxied request this run — the share of all tokens actually sent that Headroom removed, weighted by real token volume rather than by request count. A synthetic local test with a deliberate mix of one large, highly-compressible request and several small, moderately-compressible ones confirmed the two figures genuinely diverge (the aggregate landed well above the average, exactly as the weighting predicts), so both are shown side by side rather than picking one as "the" figure.
 
 **`headroom_show_savings` (on by default) also posts the same numbers directly on the pull request or issue**, as a short standalone comment — the Step Summary above is only visible to someone who opens the run in the Actions tab, which most people looking at a PR never do. This is deliberately a separate comment, not an addition to the existing progress-comment mechanism above: that comment is deleted outright on a successful run, precisely when there would be real savings to show. It's sticky, not one-per-run: a repeated push to the same pull request deletes this bot's own prior Headroom comment before posting a fresh one, the same delete-then-recreate pattern the progress comment uses for its own stale entries, so a long-lived PR never accumulates a near-duplicate comment per push. The comment is skipped when nothing was proxied (a zero-savings comment on every run would be noise) and in `generic` mode, which posts nothing to GitHub by design regardless of this setting. Set `headroom_show_savings: false` to keep the numbers in the Step Summary only.
 

action.yml

@@ -401,8 +401,11 @@ outputs:
     description: "Total tokens Headroom's compression removed this run (its own /stats summary.compression.total_tokens_removed), or '0' when headroom_enabled is false or nothing was proxied. Also written to the job's Step Summary as a human-readable line, so this isn't only reachable by a caller reading the output -- it's visible on the run itself."
     value: ${{ steps.headroom-verify.outputs.headroom_tokens_saved }}
   headroom_compression_percent:
-    description: "Headroom's own average compression percentage this run (summary.compression.avg_compression_pct), or '0' when headroom_enabled is false or nothing was proxied."
+    description: "Headroom's own average compression percentage this run (summary.compression.avg_compression_pct) -- the unweighted average of each individual request's own compression ratio, or '0' when headroom_enabled is false or nothing was proxied. See headroom_total_percent_saved for the volume-weighted aggregate figure instead."
     value: ${{ steps.headroom-verify.outputs.headroom_compression_percent }}
+  headroom_total_percent_saved:
+    description: "The aggregate share of this run's total token volume that Headroom removed (summary.compression.total_tokens_removed / total_tokens_before * 100), or '0' when headroom_enabled is false or nothing was proxied. Unlike headroom_compression_percent (an unweighted average of each request's own ratio), this is weighted by actual token volume, so it answers 'what fraction of everything sent was saved' rather than 'what was the typical per-request ratio' -- the two can diverge meaningfully when request sizes vary."
+    value: ${{ steps.headroom-verify.outputs.headroom_total_percent_saved }}
 
 runs:
   using: "composite"
@@ -1700,32 +1703,37 @@ runs:
         set -euo pipefail
         # requests.total counts every request Headroom actually forwarded to https://api.anthropic.com, incrementing even on a response the real API itself rejects (confirmed: a deliberately invalid key still increments it, since the proxy did genuinely forward the request) -- so it distinguishes "the proxy handled a real Claude Code call" from "nothing ever reached it," which is exactly what this check needs.
         #
-        # summary.compression.total_tokens_removed/avg_compression_pct are NOT documented anywhere in Headroom's own public docs (only /health's differently-shaped stats and /metrics' Prometheus counters are) -- these were confirmed empirically, by inspecting a live /stats response from the exact version this action pins (headroom_version for pip, headroom_image's digest for docker -- both pin the identical underlying release). Given a mismatch would fail silently (the ?? 0 fallback reports a plausible-looking "0 tokens saved" forever, not an error), the risk is bounded specifically because both pins are exact: a PyPI version number is immutable once published, just as a digest is, so the response shape cannot change under this action's feet on its own the way a floating :latest tag or unpinned version range could. It only changes when a maintainer deliberately bumps one of the pins (action.yml's own defaults, tracked via docker/headroom.Dockerfile's Dependabot entry for the digest) -- which is exactly the moment to re-verify these field paths against the new version's real /stats response, the same way this action always re-verifies a maintainer's own claims before trusting them (see verify_prior_findings). Confirmed against the currently-pinned version twice before merge -- a synthetic local test and a real CI run against a genuine multi-hundred-line diff -- both producing plausible non-zero values, never the "always 0" failure mode this note exists to flag.
+        # summary.compression.total_tokens_removed/avg_compression_pct/total_tokens_before are NOT documented anywhere in Headroom's own public docs (only /health's differently-shaped stats and /metrics' Prometheus counters are) -- these were confirmed empirically, by inspecting a live /stats response from the exact version this action pins (headroom_version for pip, headroom_image's digest for docker -- both pin the identical underlying release). Given a mismatch would fail silently (the ?? 0 fallback reports a plausible-looking "0 tokens saved" forever, not an error), the risk is bounded specifically because both pins are exact: a PyPI version number is immutable once published, just as a digest is, so the response shape cannot change under this action's feet on its own the way a floating :latest tag or unpinned version range could. It only changes when a maintainer deliberately bumps one of the pins (action.yml's own defaults, tracked via docker/headroom.Dockerfile's Dependabot entry for the digest) -- which is exactly the moment to re-verify these field paths against the new version's real /stats response, the same way this action always re-verifies a maintainer's own claims before trusting them (see verify_prior_findings). Confirmed against the currently-pinned version twice before merge -- a synthetic local test and a real CI run against a genuine multi-hundred-line diff -- both producing plausible non-zero values, never the "always 0" failure mode this note exists to flag.
         STATS="$(curl -fsS "http://127.0.0.1:${HEADROOM_PORT}/stats" 2>/dev/null || echo '{}')"
-        read -r PROXIED_COUNT TOKENS_SAVED COMPRESSION_PCT <<<"$(node -e '
+        # avg_compression_pct is the unweighted average of each individual request's own compression percentage -- one huge, highly-compressible request among many tiny, barely-compressible ones drags that average down even though most of the actual token volume was saved. total_tokens_removed / total_tokens_before is the aggregate instead: what fraction of everything actually sent across every proxied request this run was removed, weighted by real token volume rather than by request count. Both numbers are genuine and can differ meaningfully; showing both rather than picking one avoids a misleading single figure either way.
+        read -r PROXIED_COUNT TOKENS_SAVED COMPRESSION_PCT TOTAL_PERCENT_SAVED <<<"$(node -e '
           let d="";
           process.stdin.on("data",c=>d+=c);
           process.stdin.on("end",()=>{
             try {
               const j = JSON.parse(d);
               const c = j.summary?.compression ?? {};
-              console.log([j.requests?.total ?? 0, c.total_tokens_removed ?? 0, c.avg_compression_pct ?? 0].join(" "));
-            } catch { console.log("0 0 0"); }
+              const before = c.total_tokens_before ?? 0;
+              const removed = c.total_tokens_removed ?? 0;
+              const totalPct = before > 0 ? (removed / before * 100).toFixed(1) : 0;
+              console.log([j.requests?.total ?? 0, removed, c.avg_compression_pct ?? 0, totalPct].join(" "));
+            } catch { console.log("0 0 0 0"); }
           });
-        ' <<< "$STATS" 2>/dev/null || echo "0 0 0")"
+        ' <<< "$STATS" 2>/dev/null || echo "0 0 0 0")"
 
         {
           echo "requests_proxied=${PROXIED_COUNT}"
           echo "headroom_tokens_saved=${TOKENS_SAVED}"
           echo "headroom_compression_percent=${COMPRESSION_PCT}"
+          echo "headroom_total_percent_saved=${TOTAL_PERCENT_SAVED}"
         } >> "$GITHUB_OUTPUT"
 
         if [ "${PROXIED_COUNT:-0}" -gt 0 ] 2>/dev/null; then
           echo "headroom_proxied=true" >> "$GITHUB_OUTPUT"
-          echo "Headroom proxied ${PROXIED_COUNT} request(s) this run, saving ${TOKENS_SAVED} tokens (${COMPRESSION_PCT}% average compression)."
+          echo "Headroom proxied ${PROXIED_COUNT} request(s) this run, saving ${TOKENS_SAVED} tokens (${TOTAL_PERCENT_SAVED}% of all tokens sent; ${COMPRESSION_PCT}% average per-request compression)."
           {
             echo "### Headroom context compression"
-            echo "Proxied **${PROXIED_COUNT}** request(s) this run — saved **${TOKENS_SAVED} tokens** (**${COMPRESSION_PCT}%** average compression)."
+            echo "Proxied **${PROXIED_COUNT}** request(s) this run — saved **${TOKENS_SAVED} tokens**, **${TOTAL_PERCENT_SAVED}%** of all tokens sent this run (${COMPRESSION_PCT}% average per-request compression)."
           } >> "$GITHUB_STEP_SUMMARY"
         else
           echo "headroom_proxied=false" >> "$GITHUB_OUTPUT"
@@ -1760,6 +1768,7 @@ runs:
         REQUESTS_PROXIED: ${{ steps.headroom-verify.outputs.requests_proxied }}
         TOKENS_SAVED: ${{ steps.headroom-verify.outputs.headroom_tokens_saved }}
         COMPRESSION_PERCENT: ${{ steps.headroom-verify.outputs.headroom_compression_percent }}
+        TOTAL_PERCENT_SAVED: ${{ steps.headroom-verify.outputs.headroom_total_percent_saved }}
       run: |
         set -euo pipefail
 
@@ -1780,7 +1789,7 @@ runs:
           gh api -X DELETE "repos/${REPOSITORY}/issues/comments/${id}" >/dev/null 2>&1 || true
         done
 
-        BODY=$(printf '🗜️ **Headroom** compressed this run'"'"'s context: proxied %s request(s), saving %s tokens (%s%% average compression).\n\n%s' "$REQUESTS_PROXIED" "$TOKENS_SAVED" "$COMPRESSION_PERCENT" "$MARKER")
+        BODY=$(printf '🗜️ **Headroom** compressed this run'"'"'s context: proxied %s request(s), saving %s tokens, %s%% of all tokens sent this run (%s%% average per-request compression).\n\n%s' "$REQUESTS_PROXIED" "$TOKENS_SAVED" "$TOTAL_PERCENT_SAVED" "$COMPRESSION_PERCENT" "$MARKER")
         for attempt in 1 2 3; do
           if gh api "repos/${REPOSITORY}/issues/${ENTITY_NUMBER}/comments" -f body="$BODY" >/dev/null; then
             echo "Posted Headroom savings comment."

examples/direct/claude-review-with-headroom.yml

@@ -41,4 +41,4 @@ jobs:
           anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
           claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
           headroom_enabled: true
-          # headroom_image and headroom_port both default sensibly -- see action.yml. Every other input is optional with a sensible default too -- see action.yml for the full list. No separate reporting step needed: the action itself writes a "Headroom context compression" section (requests proxied, tokens saved, compression %) straight to this job's Step Summary, and exposes the same numbers as the headroom_proxied/headroom_tokens_saved/headroom_compression_percent outputs if you want to consume them programmatically instead.
+          # headroom_image and headroom_port both default sensibly -- see action.yml. Every other input is optional with a sensible default too -- see action.yml for the full list. No separate reporting step needed: the action itself writes a "Headroom context compression" section (requests proxied, tokens saved, both compression percentages) straight to this job's Step Summary, and exposes the same numbers as the headroom_proxied/headroom_tokens_saved/headroom_compression_percent/headroom_total_percent_saved outputs if you want to consume them programmatically instead.