perf(ci): cache everything, unserialize typecheck, fan out the archive probes - #286
Merged
Conversation
…robes CI was 7-22 min per PR with zero caching anywhere. The turbo step reported 'Cached: 0 cached, 59 total' on every single run. - turbo.json: drop `^typecheck` from typecheck.dependsOn. No tsconfig here uses composite/references and every typecheck is `tsc --noEmit`, so the edge was ordering-only and serialized a topological wave across ~26 packages. - Cache .turbo/cache and the bun module store (new .github/actions/bun-install composite). Blacksmith routes actions/cache to its own colocated backend, so no vendor-specific action is needed. - Cache Playwright browsers, keyed on the resolved version. install-deps stays unconditional — apt paths aren't covered by ~/.cache/ms-playwright, so the combined --with-deps form would skip them on a cache hit and break webkit. - TypeScript job to 8 vCPU with --concurrency=10; the job is now a wide fan rather than a chain, and 15-way concurrency on 4 cores was thrash + OOM risk. - local-archive-native: 7 sequential probes on ubuntu-latest -> a 7-way matrix on Blacksmith, timeout 90 -> 20. Each leg rebuilds the bundle rather than sharing one via artifact: dist/ is ~403MB and the build only takes ~14s. - Typecheck tsconfig.alchemy.json, which CI never covered. - timeout-minutes on every job.
…ry job - deploy-prd/deploy-stg triggered on `push: main` with their own concurrency groups, racing ci.yml — a red CI still shipped to production. Both now trigger on workflow_run of CI and require conclusion == 'success'. Everything pins workflow_run.head_sha, since github.sha on that event is the branch head rather than the commit CI ran against; workflow_dispatch keeps working as the manual escape hatch. - ingest-rust-tests: the base-branch comparison build ran in a $RUNNER_TEMP worktree, outside Swatinem/rust-cache's `workspaces: apps/ingest` scope, so it cold-compiled every dependency on every PR. Point it at the head's target dir, which the cache does cover. Head binaries are already copied out beforehand. - cargo test/build now pass --locked; Cargo.lock is committed but was never enforced (verified in sync via `cargo metadata --locked`). - timeout-minutes on all remaining jobs — 14 of 17 workflows ran on the 6h default, including all three deploys.
Every job ran on every PR, so a CSS-only change still paid ~7 min of native archive probes and a full ClickHouse E2E matrix. A `changes` job (dorny/paths-filter) gates the native CLI probes, service-map perf, ClickHouse E2E, slack-agent and the TypeScript job. Knip stays unfiltered because it is a whole-repo dead-code check and a filter would make it unsound. The slack filter deliberately includes apps/api — the approval-list drift canary fails when a mutating MCP tool lands on the API side without being mirrored. All filter outputs are forced true on push to main: paths-filter compares against the previous commit on push, which is the wrong base for a merge commit. Because a skipped job never reports a status, individual jobs are not safe to use as required checks once filtered. Added a 'CI passed' aggregator that always runs and fails only on failure/cancelled — skipped is intentionally not a failure. main currently has no branch protection or rulesets, so nothing needs changing today; if protection is ever added, require exactly 'CI passed'. Also added .github/actionlint.yaml declaring the Blacksmith runner labels, so actionlint reports real findings instead of an unknown-label per runs-on.
Contributor
|
Your Pullfrog Router balance is exhausted. You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer. Top up balance → · Enable auto-reload →
|
Ingest Rust Test + Benchmark ResultsCommit: Load Benchmark —
|
| Metric | main (median) | PR (median) | Delta |
|---|---|---|---|
| Requests/sec | 7478.83 | 11493.86 | +53.7% better |
| Rows/sec | 74788.33 | 114938.56 | +53.7% better |
| p50 latency | 8.12 ms | 5.30 ms | -34.8% better |
| p95 latency | 16.20 ms | 10.46 ms | -35.4% better |
| p99 latency | 21.33 ms | 15.14 ms | -29.0% better |
| Export catch-up | 0.026 s | 0.026 s | -0.3% better |
| Max RSS | 15.29 MiB | 16.24 MiB | +6.2% worse |
| Failures | 0 | 0 | same |
Same code path on both sides (same LOAD_TEST_INGEST_MODE), so the delta column is meaningful. Numbers come from ubuntu-latest, which is noisy — treat single-digit-percent deltas as noise.
PR load benchmark JSON (per-iteration)
[
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 13,
"duration_seconds": 0.220816483,
"export_catchup_seconds": 0.02612211,
"request_rps": 9057.294875944564,
"row_rps": 90572.94875944564,
"p50_ms": 6.764,
"p95_ms": 13.713,
"p99_ms": 19.491,
"max_rss_mb": 20.28125,
"max_cpu_percent": 20.0,
"avg_cpu_percent": 20.0
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 11,
"duration_seconds": 0.174006011,
"export_catchup_seconds": 0.025639989,
"request_rps": 11493.855807084734,
"row_rps": 114938.55807084734,
"p50_ms": 5.3,
"p95_ms": 10.458,
"p99_ms": 15.14,
"max_rss_mb": 16.23828125,
"max_cpu_percent": 16.6,
"avg_cpu_percent": 16.6
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 11,
"duration_seconds": 0.160514972,
"export_catchup_seconds": 0.026285547,
"request_rps": 12459.896887375715,
"row_rps": 124598.96887375714,
"p50_ms": 4.913,
"p95_ms": 9.65,
"p99_ms": 10.726,
"max_rss_mb": 14.953125,
"max_cpu_percent": 16.6,
"avg_cpu_percent": 16.6
}
]main load benchmark JSON (per-iteration)
[
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 13,
"duration_seconds": 0.268456025,
"export_catchup_seconds": 0.026317507,
"request_rps": 7450.009736231475,
"row_rps": 74500.09736231474,
"p50_ms": 8.249,
"p95_ms": 16.338,
"p99_ms": 23.855,
"max_rss_mb": 15.29296875,
"max_cpu_percent": 16.6,
"avg_cpu_percent": 16.6
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 13,
"duration_seconds": 0.259783152,
"export_catchup_seconds": 0.026190088,
"request_rps": 7698.7286689015145,
"row_rps": 76987.28668901515,
"p50_ms": 7.838,
"p95_ms": 15.597,
"p99_ms": 21.329,
"max_rss_mb": 14.87890625,
"max_cpu_percent": 40.0,
"avg_cpu_percent": 40.0
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 13,
"duration_seconds": 0.267421419,
"export_catchup_seconds": 0.025926635,
"request_rps": 7478.832501445967,
"row_rps": 74788.32501445967,
"p50_ms": 8.123,
"p95_ms": 16.197,
"p99_ms": 18.733,
"max_rss_mb": 15.58203125,
"max_cpu_percent": 33.3,
"avg_cpu_percent": 33.3
}
]WAL-acked microbench (cargo bench --bench ingest_bench)
Compiling maple-ingest v0.1.0 (/home/runner/_work/maple/maple/apps/ingest)
Finished `bench` profile [optimized] target(s) in 17.19s
Running benches/ingest_bench.rs (target/release/deps/ingest_bench-f6ec3881b8bd23cb)
Gnuplot not found, using plotters backend
test ingest_accept/logs_10_rows_wal_ack ... bench: 118853 ns/iter (+/- 23773)
test ingest_accept/traces_10_spans_wal_ack ... bench: 162639 ns/iter (+/- 40786)
cargo test
test result: ok. 66 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.74s
Running unittests src/bin/load_test.rs (target/debug/deps/load_test-3a24fd5dee8f57f0)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running unittests src/main.rs (target/debug/deps/maple_ingest-b89fa52019d5d48f)
running 60 tests
test autumn::tests::allowed_only_no_balance_field ... ok
test autumn::tests::flat_hardcap_depleted_blocks ... ok
test autumn::tests::flat_hardcap_with_remaining_allows ... ok
test autumn::tests::flat_sub_one_gb_remaining_still_allows ... ok
test autumn::tests::flat_overage_allows ... ok
test autumn::tests::flat_unlimited_allows ... ok
test autumn::tests::nested_balance_object_with_remaining_allows ... ok
test autumn::tests::nested_balance_object_depleted_blocks ... ok
test autumn::tests::nested_overage_allows ... ok
test autumn::tests::null_balance_no_subscription_blocks ... ok
test autumn::tests::unrecognized_shape_returns_none ... ok
test tests::api_error_from_pipeline_maps_variants_to_status ... ok
test tests::a_browser_sessions_cap_blocks_replay_without_touching_other_signals ... ok
test tests::caller_fault_rejections_do_not_mark_the_span_error ... ok
test tests::api_error_kind_maps_status_to_stable_label ... ok
test tests::clickhouse_destination_is_terminal_in_dual_mode ... ok
test tests::clickhouse_destination_uses_native_pipeline_even_in_forward_mode ... ok
test tests::clickhouse_target_resolver_rejects_password_over_http ... ok
test tests::clickhouse_target_resolver_decrypts_current_schema_password ... ok
test tests::clickhouse_target_resolver_requires_current_schema ... ok
test tests::cloudflare_ndjson_payload_parses_multiple_records ... ok
test tests::cloudflare_timestamps_support_rfc3339_unix_and_unix_nano ... ok
test tests::cloudflare_log_record_maps_body_severity_and_attributes ... ok
test tests::cloudflare_validation_payload_is_detected ... ok
test tests::data_loss_labels_match_the_kinds_the_gateway_emits ... ok
test tests::data_loss_rejections_mark_the_span_error_even_at_4xx ... ok
test tests::empty_export_request_is_accepted ... ok
test tests::enrichment_applies_to_a_payload_with_no_resource ... ok
test tests::enrichment_overwrites_tenant_fields ... ok
test tests::extract_ingest_key_returns_sentinel_literal_unchanged ... ok
test tests::decrypt_aes256_gcm_matches_node_crypto_fixture ... ok
test tests::hash_is_deterministic ... ok
test tests::lenient_log_payload_is_accepted ... ok
test tests::malformed_payload_is_rejected_with_the_parser_reason ... ok
test autumn::tests::fail_open_decisions_are_not_cached ... ok
test tests::numeric_nanos_no_longer_silently_empty_a_metric ... ok
test tests::pipeline_error_fault_split_matches_http_status ... ok
test tests::rejection_reason_is_bounded ... ok
test tests::replay_budget_disabled_when_limit_is_zero ... ok
test tests::replay_budget_scopes_totals_per_session_and_org ... ok
test tests::replay_budget_truncates_session_at_its_ceiling ... ok
test autumn::tests::fails_open_on_transport_error ... ok
test tests::resolve_connector_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_carries_spend_pause_state_from_the_key_lookup ... ok
test tests::resolve_ingest_key_keeps_stale_schema_on_managed_native_path ... ok
test tests::replay_metadata_is_rejected_and_unmetered_when_sessions_are_capped ... ok
test tests::native_request_emits_a_span_per_pipeline_stage ... ok
test tests::resolve_ingest_key_returns_none_when_hash_missing ... ok
test tests::resolve_ingest_key_returns_self_managed_true_when_active_settings_row ... ok
test tests::resolve_ingest_key_returns_self_managed_false_when_no_settings_row ... ok
test tests::sentinel_token_matches_only_exact_literal ... ok
test tests::server_faults_stay_error ... ok
test tests::spec_compliant_payloads_still_decode ... ok
test tests::spend_blocks_signal_pauses_the_whole_org_or_just_the_capped_signal ... ok
test tests::tinybird_destination_keeps_forward_mode_on_forward_path ... ok
test tests::resolve_ingest_key_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_serves_last_known_routing_when_refresh_fails ... ok
test tests::forward_mode_switches_ready_org_to_clickhouse_without_forwarding_again ... ok
test autumn::tests::usage_after_a_successful_flush_gets_a_fresh_key ... ok
test autumn::tests::a_failed_flush_retries_under_the_same_idempotency_key ... ok
test result: ok. 60 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.03s
Doc-tests maple_ingest
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
🍁 Maple PR previewNote Preview resources were removed when this pull request closed. Final commit |
Caught by the new tsconfig.alchemy.json check in CI, which had never run before.
CreateAlertingWorkerOptions declared `chatFlue: { workerName: string }`, but
createChatFlueWorker returns a not-yet-applied worker resource whose workerName
is `Output<string, never>` — resolved at apply time by the service binding. The
root alchemy.run.ts passed exactly that value, so the config only typechecked by
never being typechecked.
Contributor
|
Your Pullfrog Router balance is exhausted. You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer. Top up balance → · Enable auto-reload →
|
…stall ~/.bun/install/cache is ~739MB for this repo. Restoring and re-saving it costs more than bun spends fetching packages: knip install 34s cold -> 57s cached local-checkpoint-native 44s cold -> 113s cached Keep the composite action as the single place install behaviour lives, with the measurement recorded so this does not get re-added on intuition. The turbo cache is unaffected and is where the actual win is (59/59 hits, 6m01s -> 2.2s).
Contributor
|
Your Pullfrog Router balance is exhausted. You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer. Top up balance → · Enable auto-reload →
|
Collaborator
Author
Measured resultsVerified across 4 runs on this branch. Numbers are the full suite — this PR touches
Cold-cache runs land at ~6m, so even a first-run-on-a-branch beats the old typical. The critical path is now Two things did not go as planned, and are corrected hereThe bun store cache was a regression and has been removed. The composite action stays as the single place install behaviour lives, with the measurement recorded in it so this doesn't get re-added on intuition. The Playwright cache is a wash, not a win. Browser install now takes 0s, but the 12s cache restore plus the irreducible 24s apt Cache portability confirmed
Flake soakThe archive matrix is the riskiest change — crash/GC probes simulate process kills against a One real bug this surfacedThe new |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
CI was 7–22 min per PR. The turbo step reported this on every single run:
There was no
actions/cacheanywhere in.github/workflows/exceptSwatinem/rust-cache. So.turbo/cachedied with the runner,bun installran cold in ~13 job executions per push, and Playwright re-downloaded three browser engines every run.Measured baseline (
gh run view --json jobs):ubuntu-latestWhat changed
Caching —
.turbo/cache, the bun module store (new.github/actions/bun-installcomposite), and Playwright browsers. Blacksmith routesactions/cacheto its own colocated backend, so no vendor-specific action is needed.install-depsstays unconditional: apt paths aren't covered by~/.cache/ms-playwright, so the combined--with-depsform would skip them on a cache hit and break webkit.Turbo graph — dropped
^typecheckfromtypecheck.dependsOn. No tsconfig here usescomposite/referencesand every typecheck istsc --noEmit, which emits nothing downstream — so the edge was ordering-only and serialized a topological wave across ~26 packages.^buildis the real edge and stays. TypeScript job moved to 8 vCPU with--concurrency=10; 15-way concurrency on 4 cores was thrash plus OOM risk once the DAG flattens.Archive probes — 7 sequential probes on
ubuntu-latest→ a 7-way matrix on Blacksmith, timeout 90 → 20 min. Each leg rebuilds the bundle rather than sharing one via artifact:dist/is ~403 MB andbuild-local-binary.shonly takes ~14s.Path filtering —
dorny/paths-filtergates the CLI probes, perf, ClickHouse E2E and slack-agent. Knip stays unfiltered (whole-repo dead-code check). ACI passedaggregator collapses the fan-in, since a skipped job never reports a status.Correctness fixes found along the way
deploy-prd/deploy-stgfired onpush: mainwith their own concurrency groups, racingci.yml— a red CI still shipped to production. Both now gate onworkflow_runof CI withconclusion == 'success', pinned tohead_sha.ci.ymlnever typecheckedtsconfig.alchemy.json(everyapps/*/alchemy.run.ts) — only the roottypecheckscript chains it, and CI calls turbo directly.$RUNNER_TEMPworktree, outsiderust-cache'sworkspaces: apps/ingestscope, so it cold-compiled every dependency on every PR.timeout-minutes, including all three deploys.Verifying
The second run on this branch is the one that matters — the first only populates the cache. Watch for
Cached: 0 cachedflipping to a high hit count, and confirmWeb bundle budgetstill passes against a restoredapps/web/dist.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.