feat: golden-file tests, ten examples, case studies, PR comment mode - #189
Merged
Merged
Conversation
added 2 commits
September 20, 2026 14:01
The three strategy items that were still outstanding (1.5, 3.6, 3.7). Golden-file tests cover the terminal summary, the --json payload and the SARIF report. Each renders from a fixed findings set - no scanner, no network, no clock - and compares against a stored file, so an unintended change fails CI instead of reaching a user. Verified by reintroducing the fix-command ordering regression from #187: two tests fail, and pass again when it is reverted. Re-record with DOCKSEC_UPDATE_GOLDEN=1 and read the diff. Ten examples with documented expected findings, every count produced by running the scan. Eight Dockerfiles across Node, Python, Java, Go and BuildKit secret mounts, plus the existing two compose stacks, each insecure file paired with a hardened one. Two carry a lesson beyond the delta: the distroless Go image reports a HEALTHCHECK finding that is correct to keep, and the BuildKit example passes a build secret without tripping the secret rule that the ENV-based examples do. Three case studies against official images - node:18 (2,200 findings, 9 worth acting on today, one at the 100th EPSS percentile), python:3.12-slim (44 findings, none fixable) and nginx:1.31.6-alpine (clean, and what that does not prove). Official images keep the numbers reproducible and name no third party unfavourably. PR comment mode ships as two workflows. Stage one runs in the untrusted pull-request context with contents: read and no secrets, and emits an artifact. Stage two runs on workflow_run in the base repository, reads only that artifact, never checks out the PR's code, and posts the comment. Everything the renderer prints is escaped, because a fork controls the text of its own findings; the PR number is validated as an integer before it reaches an API path. Injection cases are covered by tests: pipes, HTML, Markdown links, newline-injected rows and backticks.
A pull request touching many compose services rendered a 156KB comment. GitHub rejects an issue comment over 65536 characters, so the API call would fail and the job would end with nothing posted - the per-file row cap bounds each table but not the total. Whole per-file blocks are now kept while they fit within a 60000 character budget, then the count of dropped files is stated. Truncating mid-table would leave unbalanced <details> tags, so blocks are never split. A realistic result is unaffected: the 8-file comment from this branch renders byte-identical.
github-actions
Bot
added
documentation
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Files
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
added 2 commits
September 20, 2026 14:05
CodeQL flagged py/unused-global-variable: the test module imported _md but only exercised it through rendered output. It is the single choke point every untrusted value passes through, so test it directly rather than drop the import.
The README told new users to pin 2026.8.19 - two releases behind, and before the fixes that make the container usable from the command line. The tag does not exist until the release is cut, which is the same ordering the previous bumps used.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The three strategy items that were still outstanding: 1.5, 3.6 and 3.7. With these, Stages 0-3 are genuinely complete.
1.5 Golden-file tests
The terminal summary, the
--jsonpayload and the SARIF report are now compared against stored files. Each renders from a fixed findings set — no scanner, no network, no clock — so a diff means the output changed, not that the world did.Verified they have teeth by reintroducing the fix-command ordering regression from #187: two tests fail, and pass again when reverted.
Re-record deliberately with
DOCKSEC_UPDATE_GOLDEN=1 pytest tests/test_golden_output.pyand review the diff.This closes a real gap. The 1.2-1.4 output surfaces changed in #187 without golden coverage, which is exactly the drift this item exists to catch.
3.6 Ten examples + three case studies
Eight Dockerfiles (Node, Python, Java, Go, BuildKit secrets) plus the two compose stacks. Every count in
examples/README.mdwas produced by running the scan.Two are instructive beyond the insecure/hardened delta:
ENV-based examples do.Case studies use official images, so the numbers are reproducible and no third party is named unfavourably:
node:18python:3.12-slimnginx:1.31.6-alpine3.7 PR comment mode
Two-stage, per the threat model in the strategy doc:
pr-scan.ymlruns in the untrusted PR context withcontents: read, no secrets, no commenting. Emits an artifact.pr-comment.ymlruns onworkflow_runin the base repo. Reads only that artifact, never checks out the PR's code, and posts.Because a fork controls the text of its own findings, everything the renderer prints is escaped — pipes, HTML, Markdown links, newline-injected rows, backticks — and the PR number is validated as an integer before reaching an API path. All covered by tests.
Found and fixed while testing: a PR touching many services rendered a 156KB comment, which GitHub's 65536-character limit would reject — the job would fail with nothing posted. The body is now budgeted to 60000 characters, dropping whole blocks (never splitting a table) and stating how many. A realistic 8-file comment renders byte-identical.
Verification
ruff check .clean