🤖 fix: self-heal corrupt analytics rows that wallpaper the dashboard - #3869
🤖 fix: self-heal corrupt analytics rows that wallpaper the dashboard#3869ibetitsmike wants to merge 7 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b41af269f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b41af269f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c13eb1a19
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 32a0868c43
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 65c58aef78
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
A rare native-layer corruption can materialize a phantom events row whose varchar columns concatenate every non-null value from an entire insert batch (observed: a 17KB model string spanning ~670 events). That single row renders as one giant legend entry in Spend over time, covering the whole Analytics dashboard. Add deleteCorruptAnalyticsRows: a length-threshold sweep over events and delegation_rollups, run at worker init and after each write cycle (ingest, syncCheck incremental/full_rebuild, rebuildAll). Donor rows are written correctly, so deleting the phantom loses no real data. Validated against the affected production DB: exactly 1 row deleted, 58,478 kept.
Validated against a copy of a production analytics.db: aggregate stats are identical across versions (only sub-ulp FP summation noise that also varies run-to-run within one version), 1.4.4 can reopen and write a DB that 1.5.5 wrote and checkpointed (downgrade-safe), and uncheckpointed WALs replay cleanly in both upgrade and downgrade directions.
- Raise sweep thresholds above legal maxima: migrated legacy workspace
IDs (${projectBasename}-${workspaceBasename}) can reach 511 chars, so
cap workspace/agent identifiers at 1024 and model strings at 512;
document the derivation. Add a retention test for a max-length legacy ID.
- Make sweepCorruptRows best-effort: catch and log failures so a sweep
error cannot reject worker init (caching a permanent workerError) or
fail an otherwise-successful ingest.
Codex round 2: custom-provider model IDs have no schema max length (ProviderModelEntrySchema), so a model-length cap could delete real spend. Since the corruption concatenates every varchar column at once and workspace identifiers appear on every row, identifier evidence alone detects the phantom row; drop the model clause from both tables and pin retention of a 2KB custom model ID.
Codex round 3: a corrupted batch small enough to keep concatenated identifiers under the length caps would evade the sweep while its 3KB model string still breaks the dashboard. Add structural evidence that is batch-size independent: a concatenation of two or more workspace IDs can never equal a real workspace ID, and every legitimate row's workspace has an ingest_watermarks entry by the time sweeps run (crash-window orphans are safe to delete because the missing watermark forces a full re-ingest of that workspace on the next syncCheck). Rollups join on parent only; children may be legitimately removed. Verified on the production DB copy: the join evidence alone identifies exactly the one phantom row among 58,479 events and 14,671 watermarks.
65c58ae to
6ca4948
Compare
|
@codex review Rebased onto main (Mux -> Shux rename included). Diff vs main is unchanged from the previously approved change set (duckdb bump + analytics corrupt-row sweep); new commits are a prettier fix for README.md (pre-existing format breakage on main tip) and a flake offline-cache hash refresh for the merged lockfile. |
1 participant
Summary
Adds a self-healing sweep that deletes a rare class of corrupt analytics rows which wallpaper the Analytics dashboard, and upgrades
@duckdb/node-apifrom 1.4.4-r.1 to 1.5.5-r.4 after verifying the upgrade preserves stats and is downgrade-safe.Background
A production
analytics.dbcontained exactly one corrupteventsrow whose every VARCHAR column was the concatenation of that column's non-null values across an entire insert batch (672 rows spanning 3 workspaces): a 17,229-charmodelstring, a 6,720-charworkspace_id,thinking_levelofhighhigh,agent_idofexecexplore. The fragment arithmetic is exact (e.g.project_path= 672 x 17 bytes of/home/coder/coder). SinceSpendChartcreates one legend entry and bar series per distinct model, that single row rendered as a giant legend entry covering the whole dashboard.The corruption happened below our JS layer (the write path is a bound-parameter
INSERTper row; numerics were copied from one donor row while varchars concatenated across the batch). All 672 donor rows were also written correctly, so the phantom row is pure junk and deleting it loses no data.Implementation
deleteCorruptAnalyticsRows()(etl.ts): length-thresholdDELETEovereventsanddelegation_rollups. Thresholds are generous vs. legitimate maxima (workspace IDs are 10-char hex; the longest real model string is 41 chars).ingest,syncCheckincremental/full-rebuild,rebuildAll), logging only when rows were deleted, so any recurrence self-heals within one sync interval.@duckdb/node-api1.4.4-r.1 -> 1.5.5-r.4: the corruption mechanism lives in the native bindings, so pick up a year of upstream fixes.Validation
make static-check-fullgreen.Risks
DELETE ... WHERE LENGTH(...)scans on every write cycle. The tables are small (tens of thousands of rows) and DuckDB scans them in-memory, so cost is negligible; thresholds are far above any legitimate value, so false-positive deletion would require values 6x longer than today's longest observed model string.Generated with
mux• Model:anthropic:claude-fable-5• Thinking:xhigh