Skip to content

feat(rust): upgrade workspace to posthog-rs 0.18.0 and on_error metric#67600

Merged
eli-r-ph merged 3 commits into
masterfrom
eli.r/posthog-rs-0.17-sdk-observability
Jul 10, 2026
Merged

feat(rust): upgrade workspace to posthog-rs 0.18.0 and on_error metric#67600
eli-r-ph merged 3 commits into
masterfrom
eli.r/posthog-rs-0.17-sdk-observability

Conversation

@eli-r-ph

@eli-r-ph eli-r-ph commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Problem

When a posthog-rs capture, /flags, or local-evaluation request fails terminally (retries exhausted or a permanent reject), our Rust services only emit a log line. There is no metric, so we cannot alert on the internal telemetry pipeline silently dropping events during an outage.

The workspace was also stuck on a split posthog-rs: everything ran 0.13.3, and batch-import-worker could not move forward because 0.14.0 made batch capture fire-and-forget, dropping the synchronous delivery Result the import commit path relies on to gate Kafka offset rollback (its at-least-once guarantee).

posthog-rs 0.18.0 unblocks both at once. 0.17.0 added the on_error observability hook, and 0.18.0 (PostHog/posthog-rs#175) restores an awaited, Result-returning batch capture (capture_batch_immediate). So we can unify the whole workspace on one version and get the failure signal.

Changes

  • Upgrade the workspace posthog-rs to 0.18.0 and drop batch-import-worker's separate 0.13.3 pin, so the entire Rust workspace now resolves a single posthog-rs version (verified: Cargo.lock collapses two entries to one).
  • Register on_error in common_posthog::init, emitting posthog_sdk_delivery_failures_total labeled service / surface (capture, flags, local_evaluation) / reason (bounded failure class). The reason split mirrors the worker's existing failure_reason so quota (expected billing enforcement) stays separable from actionable transport/server failures. The service label keeps the shared metric faceted per service.
  • cymbal: capture has been non-blocking since 0.14, so the old spawning_capture (spawn a task, await, log on error) collapses to a direct fire_capture. The on_error hook now owns the failure signal it used to log.
  • batch-import-worker: capture_batch -> capture_batch_immediate. This is compile-forced, not cosmetic: in 0.18.0 capture_batch is fire-and-forget (non-async, returns nothing). capture_batch_immediate reuses the identical v1 Ok/Err response classification, so the worker's offset gating and success/failure metrics are unchanged.

Companion Grafana dashboard PR: PostHog/grafana-dashboards#231.

How did you test this code?

Automated tests I (actually Claude, in Cursor) ran under flox, all passing:

  • cargo build + cargo clippy --all-targets + cargo fmt --check for batch-import-worker, common-posthog, cymbal — clean (only a pre-existing celes future-incompat note, unrelated).
  • cargo test -p batch-import-worker — the 21 emit::capture tests are the parity gate and pass verbatim against capture_batch_immediate: test_commit_write_fails_immediately_on_400, _fails_on_402_billing_limit, _retries_on_500_then_succeeds, _exhausts_retries_on_persistent_500, quota_failure_is_separable_from_actionable_failures, bad_request_failure_labels_events_and_requests, success_counts_events_total_and_requests_per_subbatch. These assert the exact offset-rollback and metric outcomes per HTTP status, so a green run proves behavior is identical to the 0.13.3 pin.
  • cargo test -p common-posthogsdk_error_reason_maps_every_variant (the alerting contract: quota stays separable) and on_error_hook::init_hook_emits_delivery_failure_metric_on_terminal_reject (drives a real client through init against a mock server returning 400, asserts the metric fired exactly once with the right labels).

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

No user-facing docs affected (internal service telemetry only).

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Directed by @eli-r-ph; implemented by Claude (Cursor). Key decisions:

  • Gated the whole upgrade on 0.18.0 actually publishing — verified it live on crates.io and confirmed the downloaded .crate exposes both capture_batch_immediate and the on_error hook before touching the workspace.
  • Proved the bump is safe rather than assuming it: the public-API delta 0.17.0 -> 0.18.0 is purely additive (CaptureSummary, capture_immediate, capture_batch_immediate, a new secret_key builder), nothing this PR relies on was removed or changed.
  • Proved worker parity from source: 0.13.3 capture_batch and 0.18.0 capture_batch_immediate drive the identical v1 after_response classification (is_retryable_status and Error::from_http_response are byte-identical across the two tags), so Err (roll back the offset) and Ok (advance) fire on exactly the same responses. The worker ignored the old Ok(()) payload and ignores the new Ok(CaptureSummary) the same way.
  • Kept the worker on the synchronous Result and deliberately did NOT route it through on_error: the hook fires post-return on a background thread and cannot gate a Kafka offset. on_error is for the fire-and-forget crates (cymbal, embedding-worker) only.
  • No repo skills applied (no migrations, DRF, or frontend touched).

@eli-r-ph eli-r-ph self-assigned this Jul 2, 2026
@greptile-apps

greptile-apps Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Reviews (1): Last reviewed commit: "feat(observability): surface posthog-rs ..." | Re-trigger Greptile

@eli-r-ph eli-r-ph changed the title feat(observability): surface posthog-rs SDK delivery failures via on_error feat(rust): upgrade posthog-rs and extend observability in Rust workspace dependents Jul 2, 2026
@eli-r-ph eli-r-ph changed the title feat(rust): upgrade posthog-rs and extend observability in Rust workspace dependents feat(rust): upgrade workspace to posthog-rs 0.18.0 and on_error metric Jul 9, 2026
eli-r-ph added 3 commits July 9, 2026 17:49
…error

Register posthog-rs's on_error hook in common_posthog::init so terminal SDK
delivery failures (exhausted retries / permanent rejects) across capture,
flags, and local-evaluation become a metric instead of only a log line.

- common_posthog::init wires on_error -> posthog_sdk_delivery_failures_total,
  labeled service/surface/reason; the service label keeps the shared metric
  faceted per service regardless of scrape topology
- registering the hook also silences the SDK's default per-reject WARN, so
  this metric now owns that signal
- document why batch-import-worker stays pinned at posthog-rs 0.13.3 (0.14.0
  made capture fire-and-forget, dropping the synchronous delivery Result its
  at-least-once offset gating relies on)
Registering an on_error hook suppresses posthog-rs's own default per-drop
warn!, so a metric-only hook went dark on logs (and cymbal's old per-capture
error! was already gone with the fire-and-forget migration. Re-emit a single
warn! alongside the metric so the hook is a superset of the SDK default rather
than a regression. warn (not error) matches the SDK's severity for dropped
telemetry and fires only on terminal failure, so it stays bounded.
EOF
)
posthog-rs 0.18.0 restores an awaited, Result-returning batch capture
(capture_batch_immediate), so the worker no longer needs its 0.13.3 pin.

- workspace posthog-rs 0.17.3 -> 0.18.0; batch-import-worker drops its
  separate 0.13.3 override and inherits the workspace pin, so the whole
  Rust workspace now resolves a single posthog-rs version
- emit/capture.rs: capture_batch -> capture_batch_immediate. In 0.18.0
  capture_batch is fire-and-forget (non-async, returns nothing), so this
  is compile-forced; capture_batch_immediate reuses the identical v1
  Ok/Err response classification, so offset gating and failure metrics
  are unchanged (all 21 emit::capture tests pass verbatim)
@eli-r-ph eli-r-ph force-pushed the eli.r/posthog-rs-0.17-sdk-observability branch from 3e9ac67 to 6c8750c Compare July 10, 2026 01:20
@trunk-io

trunk-io Bot commented Jul 10, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@eli-r-ph eli-r-ph requested review from a team and cat-ph July 10, 2026 03:05
@eli-r-ph eli-r-ph marked this pull request as ready for review July 10, 2026 03:05
@greptile-apps

greptile-apps Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Reviews (2): Last reviewed commit: "feat(rust): migrate batch-import-worker ..." | Re-trigger Greptile

@cat-ph cat-ph left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@eli-r-ph eli-r-ph merged commit 5bbebce into master Jul 10, 2026
306 of 310 checks passed
@eli-r-ph eli-r-ph deleted the eli.r/posthog-rs-0.17-sdk-observability branch July 10, 2026 18:19
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 10, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-10 18:55 UTC Run
prod-us ✅ Deployed 2026-07-10 19:06 UTC Run
prod-eu ✅ Deployed 2026-07-10 19:07 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants