feat: multi-destination flag log routing - #483
Open
vahidlazio wants to merge 13 commits into
Open
Conversation
Add account-level log destination configuration so the Cloudflare resolver can route flag logs to either the existing edge endpoint or a new Cloudflare ingest worker that stores logs as NDJSON in R2. - Add LogDestination enum to ResolverState proto (tag 11) - Parse log_destination in Rust ResolverState::from_proto - Route logs in CF worker queue consumer based on state destination - Add new confidence-flag-log-ingest worker crate (R2 storage with account-level folder partitioning) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…e naming
- Proto: change to `repeated LogDestination log_destinations` so logs
can be sent to multiple destinations simultaneously
- Rust lib: parse as Vec<LogDestination>, default to [Edge] when empty
- CF worker: iterate over all destinations and send to each
- Ingest worker: flat file naming {account}_{timestamp}_{uuid}.ndjson
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Keep proto enum name in sync with admin repo. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tate Read log_destinations from SetResolverStateRequest (ClientResolverState) instead of from the inner ResolverState. Matches admin proto change. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…verStateRequest Proves that log_destinations at tag 3 breaks WASM parsing (InvalidProtocolBufferException) while tag 4 is safely skipped. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update Cloudflare destination URL to epx-flags-logs worker - Encode flag logs as protobuf IngestFlagLogsRequest for Cloudflare destination instead of JSON (matching ingestor's expected format) - Add encode_message helper to confidence-resolver for cross-prost-version protobuf encoding - Add confidence-flag-log-ingest to Dockerfile build stages Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
vahidlazio
force-pushed
the
feat/multi-destination-flag-logs
branch
from
August 6, 2026 16:16
3e42378 to
acef531
Compare
When log_destinations includes Cloudflare, the ingestor handles forwarding to BQ. Sending to Edge as well would create duplicate writes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
vahidlazio
marked this pull request as ready for review
August 6, 2026 20:13
When both Cloudflare and Edge destinations are configured, Cloudflare is always primary regardless of order in the state, with Edge as the retry fallback on failure (HTTP 4xx/5xx or network error). Single destination (Cloudflare-only or Edge-only) sends to that destination only. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The workspace lists confidence-flag-log-ingest as a member, so every stage that runs cargo against the workspace needs the crate present. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The crate denies clippy::expect_used; encoding into a Vec cannot fail, so use prost's infallible encode_to_vec instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LogDestinationenum (EDGE/CLOUDFLARE) toResolverStateproto (field tag 11, backwards compatible — unset defaults to Edge)log_destinationin RustResolverState::from_protoand expose as typed enumconfidence-flag-log-ingestworker crate — receivesWriteFlagLogsRequestvia HTTP, writes NDJSON to R2 with account-level folder partitioning (flag-logs/{account_id}/date=.../hour=.../)Deployment order
log_destination=CLOUDFLAREon test account via flags-admin (separate PR)Flags-admin changes (separate PR)
log_destinationto admin's copy ofresolver.protoResolverStatefor CDN pushTest plan
confidence_resolver,confidence-cloudflare-resolver,confidence-flag-log-ingest)WriteFlagLogsRequest, verify NDJSON in R2log_destination=CLOUDFLARE, deploy CF resolver, verify logs routed to ingest worker🤖 Generated with Claude Code