Repository navigation
Count clone and fetch decisions hourly instead of writing a row per fetch - #712
Merged
Merged
Conversation
coopernetes
force-pushed
the
feature/fetch-audit-records
branch
from
October 1, 2026 02:08
bfc9683 to
a21f227
Compare
…etch Every HTTP clone wrote a fetch record synchronously, so a busy read gateway paid a database write per clone and a refused request cost the caller nothing and fogwall a row. SSH fetches wrote nothing, and the records could not be queried. Fetch decisions are now counted in memory and written as hourly totals per repository, transport, proxy mode, result, refusal reason and matching rule, on an interval and pruned after a retention window. Both HTTP modes and SSH are counted, including refusals by the credential filter and the serve-fetch switch. Distinct keys between writes are capped. The totals are served at /api/fetches and open from the Repos page, and a fogwall.fetch.decisions metric carries them without the repository. Allowed fetches now log at DEBUG. fetch_records is no longer written and is left in place for operators to drop. Closes #652 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
coopernetes
force-pushed
the
feature/fetch-audit-records
branch
from
October 1, 2026 02:43
a21f227 to
31c1a5a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every HTTP clone wrote a fetch record synchronously, so a busy read gateway paid one database write per clone, and a refused request cost the caller nothing and fogwall a row. SSH fetches wrote nothing, and the records could not be queried.
Fetch decisions are now counted in memory (
FetchActivityRecorder) and written as hourly totals to a newfetch_activitytable/collection everyserver.fetch-activity.flush-interval-seconds(60), pruned afterretention-days(30). A row is keyed by hour, repository, transport, proxy mode, result, refusal reason and matching rule; no caller identity is kept. Distinct keys between writes are capped bymax-keys(10000); past it, new keys are counted without their repository.FetchDecision) by the URL rules, the fogwall credential filter and the serve-fetch switch, and counted byFetchActivityFilterafter the chain runs, so a later refusal is never counted as allowed. SSH counts inSshGitUploadCommand.GET /api/fetchesserves the hourly rows; the Repos page fetch counts open them.fogwall.fetch.decisionsOTel counter (provider, mode, result, refusal).AuditLogFilter,ServerRepositoryResolver, SSH upload); refusals stay at INFO.fetch_recordsis no longer written and is left in place for operators to drop; documented in the database and access-rules docs.Tests:
FetchStoreContractruns against the in-memory, JDBC (H2) and Mongo stores; MySQL and MariaDB smoke tests cover the V21 variant, including that an increment leavesbucket_startalone.FetchActivityRecorderTestcovers flush, hour buckets, the key cap, store and prune failures, hourly pruning, telemetry, concurrent recording during a flush, and thatstop()writes what is still in memory.SshGitUploadCommandTestcovers each SSH refusal being counted with its reason (serve-fetch off, no allow rule, deny rule, no agent forwarding) before any upstream contact.JettyConfigurationBuilderTestcovers startup refusing aserver.fetch-activitysetting below 1.AuditLogFilterTestcovers the log level: DEBUG for a fetch nothing refused, INFO for refusals and pushes.FetchActivityFilterTest,UrlRuleFilterTest,FogwallCredentialFilterTest,DisabledFetchUploadPackFactoryTestcover the decisions and their precedence.UrlRuleE2ETest(allowed and denied clones, both HTTP modes) andSshE2ETest(allowed and denied SSH clones) assert the counted rows.Not covered by a test: the JDBC retry when two instances insert the same hour's row at once. An HTTP protocol v2 clone counts twice (ref listing and fetch are separate requests); telling them apart would need the request body.
Closes #652
🤖 Generated with Claude Code