Skip to content

refactor(alerts): unify on the v2 alert system - #293

Open
Makisuo wants to merge 3 commits into
mainfrom
agent/unify-alert-system
Open

refactor(alerts): unify on the v2 alert system#293
Makisuo wants to merge 3 commits into
mainfrom
agent/unify-alert-system

Conversation

@Makisuo

@Makisuo Makisuo commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • remove the v1 alerts HTTP surface and make v2 the only control plane, including canonical delivery history
  • route custom metric alerts through the shared query-builder/query-engine path and remove direct metric schema, storage, and runtime paths
  • remove legacy Slack/Hazel webhook destinations and UI compatibility fallbacks
  • add migration 0026_windy_bromley to convert metric rules and delete retired destination data

Why

Alerts had parallel HTTP, evaluation, storage, provider, and UI paths. This hard cutover leaves one Postgres-backed alert service and one shared query-engine evaluation path.

User and operator impact

  • Existing direct metric rules are converted to builder_query drafts.
  • Legacy slack and hazel destinations, plus their delivery history, are deleted.
  • Rules left without destinations are disabled.
  • Existing integrations must use slack-bot or hazel-oauth.
  • /api/alerts is removed; clients must use /v2/alerts/*.

Deployment warning

This is a destructive, forward-only hard cutover. Production rollout should:

  1. take a restorable database backup
  2. deploy the application code
  3. immediately apply 0026_windy_bromley through the direct PlanetScale connection with runtime grants
  4. smoke-test /v2/alerts/{rules,destinations,incidents,deliveries} and the scheduler

Reverting after the migration requires a database restore or a forward fix.

Validation

  • bun typecheck — 33 packages passed
  • API alert suite — 110 tests passed
  • focused web alert suites, including final chart-prefill checks — passed
  • domain contracts — 64 tests passed
  • query engine — 70 tests passed
  • database migrations — 3 tests passed
  • Alchemy provider — 17 tests passed
  • React Doctor — no changed-file diagnostics
  • git diff --check — clean

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown

Ingest Rust Test + Benchmark Results

Commit: 6b81e80799065cce1c39e1962d159b0c2776c8d2

Load Benchmark — tinybird mode, median of 3 run(s) vs main

Metric main (median) PR (median) Delta
Requests/sec 13782.30 11320.34 -17.9% worse
Rows/sec 137823.04 113203.38 -17.9% worse
p50 latency 4.31 ms 5.42 ms +25.9% worse
p95 latency 8.46 ms 10.70 ms +26.5% worse
p99 latency 10.99 ms 14.65 ms +33.3% worse
Export catch-up 0.026 s 0.026 s +0.2% worse
Max RSS 15.63 MiB 16.95 MiB +8.5% worse
Failures 0 0 same

Same code path on both sides (same LOAD_TEST_INGEST_MODE), so the delta column is meaningful. Numbers come from ubuntu-latest, which is noisy — treat single-digit-percent deltas as noise.

PR load benchmark JSON (per-iteration)
[
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 11,
    "duration_seconds": 0.186789461,
    "export_catchup_seconds": 0.026361865,
    "request_rps": 10707.24220356308,
    "row_rps": 107072.4220356308,
    "p50_ms": 5.729,
    "p95_ms": 11.183,
    "p99_ms": 15.605,
    "max_rss_mb": 17.80078125,
    "max_cpu_percent": 16.6,
    "avg_cpu_percent": 16.6
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 11,
    "duration_seconds": 0.175117086,
    "export_catchup_seconds": 0.026213256,
    "request_rps": 11420.930108441846,
    "row_rps": 114209.30108441845,
    "p50_ms": 5.214,
    "p95_ms": 10.21,
    "p99_ms": 14.651,
    "max_rss_mb": 15.375,
    "max_cpu_percent": 0.0,
    "avg_cpu_percent": 0.0
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 11,
    "duration_seconds": 0.176673177,
    "export_catchup_seconds": 0.026469277,
    "request_rps": 11320.337551862784,
    "row_rps": 113203.37551862783,
    "p50_ms": 5.423,
    "p95_ms": 10.702,
    "p99_ms": 12.173,
    "max_rss_mb": 16.953125,
    "max_cpu_percent": 0.0,
    "avg_cpu_percent": 0.0
  }
]
main load benchmark JSON (per-iteration)
[
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 10,
    "duration_seconds": 0.145113613,
    "export_catchup_seconds": 0.02630124,
    "request_rps": 13782.304489930933,
    "row_rps": 137823.04489930935,
    "p50_ms": 4.309,
    "p95_ms": 8.463,
    "p99_ms": 10.31,
    "max_rss_mb": 15.62890625,
    "max_cpu_percent": 16.6,
    "avg_cpu_percent": 16.6
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 9,
    "duration_seconds": 0.137557727,
    "export_catchup_seconds": 0.026003161,
    "request_rps": 14539.350450302221,
    "row_rps": 145393.5045030222,
    "p50_ms": 4.189,
    "p95_ms": 8.067,
    "p99_ms": 11.403,
    "max_rss_mb": 16.48046875,
    "max_cpu_percent": 0.0,
    "avg_cpu_percent": 0.0
  },
  {
    "ingest_mode": "tinybird",
    "requests": 2000,
    "successes": 2000,
    "failures": 0,
    "rows_sent": 20000,
    "rows_exported": 20000,
    "imports": 10,
    "duration_seconds": 0.156519869,
    "export_catchup_seconds": 0.02633294,
    "request_rps": 12777.930449200669,
    "row_rps": 127779.30449200669,
    "p50_ms": 4.68,
    "p95_ms": 9.109,
    "p99_ms": 10.993,
    "max_rss_mb": 15.375,
    "max_cpu_percent": 0.0,
    "avg_cpu_percent": 0.0
  }
]

WAL-acked microbench (cargo bench --bench ingest_bench)

   Compiling maple-ingest v0.1.0 (/home/runner/_work/maple/maple/apps/ingest)
    Finished `bench` profile [optimized] target(s) in 15.19s
     Running benches/ingest_bench.rs (target/release/deps/ingest_bench-f6ec3881b8bd23cb)
Gnuplot not found, using plotters backend
test ingest_accept/logs_10_rows_wal_ack ... bench:       88739 ns/iter (+/- 5641)
test ingest_accept/traces_10_spans_wal_ack ... bench:      123350 ns/iter (+/- 8493)

cargo test


test result: ok. 66 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.74s

     Running unittests src/bin/load_test.rs (target/debug/deps/load_test-3a24fd5dee8f57f0)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/main.rs (target/debug/deps/maple_ingest-b89fa52019d5d48f)

running 60 tests
test autumn::tests::flat_hardcap_depleted_blocks ... ok
test autumn::tests::allowed_only_no_balance_field ... ok
test autumn::tests::flat_hardcap_with_remaining_allows ... ok
test autumn::tests::flat_overage_allows ... ok
test autumn::tests::flat_sub_one_gb_remaining_still_allows ... ok
test autumn::tests::nested_balance_object_depleted_blocks ... ok
test autumn::tests::flat_unlimited_allows ... ok
test autumn::tests::nested_balance_object_with_remaining_allows ... ok
test autumn::tests::nested_overage_allows ... ok
test autumn::tests::null_balance_no_subscription_blocks ... ok
test autumn::tests::unrecognized_shape_returns_none ... ok
test tests::a_browser_sessions_cap_blocks_replay_without_touching_other_signals ... ok
test tests::api_error_from_pipeline_maps_variants_to_status ... ok
test tests::caller_fault_rejections_do_not_mark_the_span_error ... ok
test tests::clickhouse_destination_is_terminal_in_dual_mode ... ok
test tests::api_error_kind_maps_status_to_stable_label ... ok
test tests::clickhouse_destination_uses_native_pipeline_even_in_forward_mode ... ok
test tests::clickhouse_target_resolver_decrypts_current_schema_password ... ok
test tests::clickhouse_target_resolver_rejects_password_over_http ... ok
test tests::clickhouse_target_resolver_requires_current_schema ... ok
test tests::cloudflare_ndjson_payload_parses_multiple_records ... ok
test tests::cloudflare_log_record_maps_body_severity_and_attributes ... ok
test tests::cloudflare_timestamps_support_rfc3339_unix_and_unix_nano ... ok
test tests::cloudflare_validation_payload_is_detected ... ok
test tests::data_loss_labels_match_the_kinds_the_gateway_emits ... ok
test tests::data_loss_rejections_mark_the_span_error_even_at_4xx ... ok
test tests::decrypt_aes256_gcm_matches_node_crypto_fixture ... ok
test tests::empty_export_request_is_accepted ... ok
test tests::enrichment_applies_to_a_payload_with_no_resource ... ok
test tests::enrichment_overwrites_tenant_fields ... ok
test tests::extract_ingest_key_returns_sentinel_literal_unchanged ... ok
test tests::hash_is_deterministic ... ok
test tests::lenient_log_payload_is_accepted ... ok
test tests::malformed_payload_is_rejected_with_the_parser_reason ... ok
test autumn::tests::fails_open_on_transport_error ... ok
test autumn::tests::fail_open_decisions_are_not_cached ... ok
test tests::pipeline_error_fault_split_matches_http_status ... ok
test tests::numeric_nanos_no_longer_silently_empty_a_metric ... ok
test tests::rejection_reason_is_bounded ... ok
test tests::replay_budget_disabled_when_limit_is_zero ... ok
test tests::replay_budget_scopes_totals_per_session_and_org ... ok
test tests::replay_budget_truncates_session_at_its_ceiling ... ok
test tests::resolve_connector_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_carries_spend_pause_state_from_the_key_lookup ... ok
test tests::resolve_ingest_key_keeps_stale_schema_on_managed_native_path ... ok
test tests::replay_metadata_is_rejected_and_unmetered_when_sessions_are_capped ... ok
test tests::resolve_ingest_key_returns_none_when_hash_missing ... ok
test tests::resolve_ingest_key_returns_self_managed_false_when_no_settings_row ... ok
test tests::resolve_ingest_key_returns_self_managed_true_when_active_settings_row ... ok
test tests::resolve_ingest_key_refreshes_routing_before_auth_cache_expires ... ok
test tests::sentinel_token_matches_only_exact_literal ... ok
test tests::server_faults_stay_error ... ok
test tests::native_request_emits_a_span_per_pipeline_stage ... ok
test tests::spec_compliant_payloads_still_decode ... ok
test tests::spend_blocks_signal_pauses_the_whole_org_or_just_the_capped_signal ... ok
test tests::tinybird_destination_keeps_forward_mode_on_forward_path ... ok
test tests::resolve_ingest_key_serves_last_known_routing_when_refresh_fails ... ok
test tests::forward_mode_switches_ready_org_to_clickhouse_without_forwarding_again ... ok
test autumn::tests::a_failed_flush_retries_under_the_same_idempotency_key ... ok
test autumn::tests::usage_after_a_successful_flush_gets_a_fresh_key ... ok

test result: ok. 60 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.03s

   Doc-tests maple_ingest

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown

🍁 Maple PR preview

Warning

The latest preview deployment did not complete. The Alchemy deploy outcome was failure; any earlier preview should be treated as stale.

Commit 5e9ca12 · View workflow run

@Makisuo
Makisuo marked this pull request as ready for review July 31, 2026 11:00
@pullfrog

pullfrog Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Your Pullfrog Router balance is exhausted.

You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

Pullfrog  | ⚠️ this action is pinned to a commit SHA, which freezes the cleanup step — switch to @v0 or keep the SHA fresh with Dependabot | Rerun failed job ➔View workflow run | via Pullfrog | Using Claude Opus𝕏

@pullfrog

pullfrog Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Your Pullfrog Router balance is exhausted.

You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

Pullfrog  | ⚠️ this action is pinned to a commit SHA, which freezes the cleanup step — switch to @v0 or keep the SHA fresh with Dependabot | Rerun failed job ➔View workflow run | via Pullfrog | Using Claude Opus𝕏

@pullfrog

pullfrog Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Your Pullfrog Router balance is exhausted.

You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

Pullfrog  | ⚠️ this action is pinned to a commit SHA, which freezes the cleanup step — switch to @v0 or keep the SHA fresh with Dependabot | Rerun failed job ➔View workflow run | via Pullfrog | Using Claude Opus𝕏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant