Skip to content

Count clone and fetch decisions hourly instead of writing a row per fetch - #712

Merged
coopernetes merged 1 commit into
mainfrom
feature/fetch-audit-records
Oct 1, 2026
Merged

coopernetes merged 1 commit into
mainfrom
feature/fetch-audit-records

Conversation

@coopernetes

@coopernetes coopernetes commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Every HTTP clone wrote a fetch record synchronously, so a busy read gateway paid one database write per clone, and a refused request cost the caller nothing and fogwall a row. SSH fetches wrote nothing, and the records could not be queried.

Fetch decisions are now counted in memory (FetchActivityRecorder) and written as hourly totals to a new fetch_activity table/collection every server.fetch-activity.flush-interval-seconds (60), pruned after retention-days (30). A row is keyed by hour, repository, transport, proxy mode, result, refusal reason and matching rule; no caller identity is kept. Distinct keys between writes are capped by max-keys (10000); past it, new keys are counted without their repository.

  • Decisions are carried on the request (FetchDecision) by the URL rules, the fogwall credential filter and the serve-fetch switch, and counted by FetchActivityFilter after the chain runs, so a later refusal is never counted as allowed. SSH counts in SshGitUploadCommand.
  • GET /api/fetches serves the hourly rows; the Repos page fetch counts open them.
  • fogwall.fetch.decisions OTel counter (provider, mode, result, refusal).
  • Allowed fetches now log at DEBUG (AuditLogFilter, ServerRepositoryResolver, SSH upload); refusals stay at INFO.
  • fetch_records is no longer written and is left in place for operators to drop; documented in the database and access-rules docs.

Tests:

  • FetchStoreContract runs against the in-memory, JDBC (H2) and Mongo stores; MySQL and MariaDB smoke tests cover the V21 variant, including that an increment leaves bucket_start alone.
  • FetchActivityRecorderTest covers flush, hour buckets, the key cap, store and prune failures, hourly pruning, telemetry, concurrent recording during a flush, and that stop() writes what is still in memory.
  • SshGitUploadCommandTest covers each SSH refusal being counted with its reason (serve-fetch off, no allow rule, deny rule, no agent forwarding) before any upstream contact.
  • JettyConfigurationBuilderTest covers startup refusing a server.fetch-activity setting below 1.
  • AuditLogFilterTest covers the log level: DEBUG for a fetch nothing refused, INFO for refusals and pushes.
  • FetchActivityFilterTest, UrlRuleFilterTest, FogwallCredentialFilterTest, DisabledFetchUploadPackFactoryTest cover the decisions and their precedence.
  • Wiring: UrlRuleE2ETest (allowed and denied clones, both HTTP modes) and SshE2ETest (allowed and denied SSH clones) assert the counted rows.

Not covered by a test: the JDBC retry when two instances insert the same hour's row at once. An HTTP protocol v2 clone counts twice (ref listing and fetch are separate requests); telling them apart would need the request body.

Closes #652

🤖 Generated with Claude Code

@coopernetes
coopernetes force-pushed the feature/fetch-audit-records branch from bfc9683 to a21f227 Compare October 1, 2026 02:08
…etch

Every HTTP clone wrote a fetch record synchronously, so a busy read gateway
paid a database write per clone and a refused request cost the caller nothing
and fogwall a row. SSH fetches wrote nothing, and the records could not be
queried.

Fetch decisions are now counted in memory and written as hourly totals per
repository, transport, proxy mode, result, refusal reason and matching rule,
on an interval and pruned after a retention window. Both HTTP modes and SSH
are counted, including refusals by the credential filter and the serve-fetch
switch. Distinct keys between writes are capped. The totals are served at
/api/fetches and open from the Repos page, and a fogwall.fetch.decisions
metric carries them without the repository. Allowed fetches now log at DEBUG.

fetch_records is no longer written and is left in place for operators to drop.

Closes #652

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@coopernetes
coopernetes force-pushed the feature/fetch-audit-records branch from a21f227 to 31c1a5a Compare October 1, 2026 02:43
@coopernetes
coopernetes merged commit 7309cc0 into main Oct 1, 2026
25 checks passed
@coopernetes
coopernetes deleted the feature/fetch-audit-records branch October 1, 2026 02:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fetch audit records: fix the identity fields, record SSH fetches, make them queryable

1 participant