Repository navigation
fix(config): disable plans left with no accounts when their account is deleted - #1065
Conversation
…s deleted DeleteCloudAccount let the plan_accounts cascade remove an enabled plan's only target, leaving it enabled with zero accounts (the unscoped shape migration 000060 removed). Inside the existing transaction it now locks the account row first, so a writer attaching the account to a plan is waited for, then disables every enabled plan whose only plan_accounts row is this account, before the DELETE. A refused delete rolls the disable back with it. Closes LeanerCloud#264 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reached
This review includes 4 billable files and costs up to $1.00. Or wait 59 minutes for your next included review. View limit details
Comment |
cristim
left a comment
There was a problem hiding this comment.
Independent review of 077fdc2: draft: blocks ready
Findings
-
BLOCKING. internal/config/store_postgres.go:3933-3942 and 3974-3978 (lock +
disableSoleAccountPlansTx): two accounts of the same plan deleted concurrently leave the plan enabled with zero accounts. The lock is per account row (FOR UPDATE ... WHERE id = $1), so deleting A and B takes different locks and nothing serializes them. Under READ COMMITTED each transaction'sNOT EXISTS (other account)still sees the other account's uncommittedplan_accountsrow, so neither disables the plan; each DELETE then cascades its own row away and both commit. Scenario: plan P enabled with accounts {A, B}, admin (or two admins, or a bulk cleanup) deletes A and B in parallel. Reproduced on real Postgres (testcontainers, postgres:16): a scratch test seeding {A,B} plan and callingDeleteCloudAccount(A)andDeleteCloudAccount(B)from two goroutines released together left the plan enabled with 0 accounts in 39 of 40 iterations. That is exactly the state #264 is meant to remove, and it is the common outcome, not a rare window. The PR's concurrency test covers only an account being attached concurrently. Suggested fix: lock the candidate plan rows before deciding, in a fixed order, e.g.SELECT id FROM purchase_plans WHERE id IN (SELECT plan_id FROM plan_accounts WHERE account_id = $1) ORDER BY id FOR UPDATEbefore the disable, so the second transaction waits for the first to commit and then re-evaluates (READ COMMITTED re-reads plan_accounts after the wait). Alternatively re-check "enabled plans with zero accounts" after the DELETE in the same transaction and disable those. Add a test with two concurrent deletes (loop or a lock-wait gate like the existing one). -
NON-BLOCKING, same class: a
SetPlanAccountsthat removes B from {A, B} concurrently with the delete of A is not serialized either (SetPlanAccounts locks the plan FOR SHARE, but the delete's UPDATE only touches plans that match its WHERE, so it sees B before that commit). The plan-row lock in finding 1 would cover this too. The PR body already documents a related deadlock window; the plan-lock ordering fix should be checked against it. -
NON-BLOCKING: the PR body lists no tests for managed/ramp plans.
disableSoleAccountPlansTxsetsenabled = falsedirectly and does not go through the #631 legacy-writer guards or touchdates_managed; I found no reason it must (it does not write dates or tranches), but a one-line test that a managed plan is disabled with its schedule columns unchanged would pin that.
What I attacked
- Tenant scoping: SQL is keyed on the account UUID and
plan_accounts; no cross-account or cross-plan rows are touched. Plans on other accounts and shared plans stay enabled with unchangedupdated_at(PR test passes; I also ran it). - Idempotency and refused delete: already disabled plans are not rewritten; a refused delete rolls the disable back (
TestDeleteCloudAccount_RefusedDeleteKeepsPlanEnabledpasses). - Concurrent attach (the PR's case): lock-first ordering holds; test passes.
- Concurrent delete of two accounts of one plan: FAILS, finding 1.
- Mutation probe (1 of a handful, core logic): dropped the
NOT EXISTS (other account)clause;TestDeleteCloudAccount_DisablesPlansLeftWithoutAccountsfails. Reverted. The race above survives the real code, not a mutant. - Tests at this SHA:
go test ./internal/config/849 pass; integrationTestDeleteCloudAccount_*plusTestAccountDeleteWaitsForCommandAccountShare20 pass. - Scheduler and "disabled": the PR cites
notifications.goskipping!plan.EnabledandvalidatePurchaseSchedulingrefusing a disabled plan; I did not find a contradiction, but disabling does not cancel already-created pending executions (the PR notes this as out of scope). - History/audit: the disable is a direct UPDATE with no history row;
updated_atis left to whatever trigger exists (not re-verified). The deletion is only logged at info.
Gate status (origin/main Review gate)
- Independent adversarial review of the full diff naming the SHA: this comment; verdict blocks ready.
- Actionable findings resolved: no, finding 1 is open. CodeRabbit skipped (draft); no threads.
- CI on 077fdc2: all listed checks pass (Unit, Integration, E2E, Lint, pre-commit, security); fork run approval not an issue as checks ran.
- Local verification on macOS: real Postgres integration tests run and pass, but they do not exercise finding 1.
- Verdict, SHA and evidence on the PR: this comment.
- Labels: match the closing issue per the PR body list (priority/p2, severity/medium, triaged, urgency/this-quarter, impact/few, effort/s, type/bug).
- Size: 47 additions in production code, 165 in tests, under the cap. The branch is 2 commits behind main; re-check after rebase.
Blocks ready
- Finding 1 (two-account concurrent delete leaves plan enabled with zero accounts) plus a regression test for it.
|
@cristim This merged at 077fdc2 with the review's BLOCKING finding 1 still open. Two concurrent deletes of a plan's accounts can still leave the plan enabled with zero accounts. Was that deliberate? I've tracked it as #1072, and the fix is written: plan rows are locked in id order before the account row, which also serializes |
Closes #264
Root cause
plan_accounts.account_idisON DELETE CASCADE(000011), andDeleteCloudAccountnever looked at plans. When an account was the only target of an enabled plan, the delete removed that plan's lastplan_accountsrow and left the plan enabled. That is the unscoped "universal plan" shape that migration 000060 deleted. The notification sweep then kept creating executions for it with no account to buy in. #1020 lets more accounts be deleted (accounts whose executions are all terminal), so this state now comes up more often.Fix
All in
internal/config/store_postgres.go, insideDeleteCloudAccount's existing transaction. No migration.disableSoleAccountPlansTx: runs just before theDELETE. It setsenabled = falseon every enabled plan whose onlyplan_accountsrow is the account being deleted, and returns their IDs. After the commit, the IDs are logged at info level. Plans that keep another account are not touched. Plans that are already disabled are not rewritten, so theirupdated_at(the plan edit token, 000102) does not move. A plan with no accounts at all is not touched either;createPlanwrites the plan and its accounts in separate calls, so a plan can briefly exist with none.notifications.go:!plan.Enabled), andvalidatePurchaseSchedulingrefuses a disabled plan.SELECT 1 FROM cloud_accounts WHERE id = $1 FOR UPDATE). A writer that is attaching this account to a plan (SetPlanAccountsinside a transaction that has not committed yet) holds a KEY SHARE lock on the account row through theplan_accountsFK check. Without the lock, the plan query would run before that writer commits. TheDELETEwould then wait for the writer and cascade its new row away, which leaves that plan enabled with zero accounts. With the lock, the delete waits for the writer first, so the plan query sees the writer's committed rows.TestAccountDeleteWaitsForCommandAccountShare(the editor-footprint race test) now waits on the new lock statement instead of onDELETE FROM cloud_accounts, because that is now where the delete queues behind the footprint's accountFOR SHARE. Its assertions are unchanged.Ramp-step interaction (from #1020's notes): this does not touch
purchase_executions. The detach from #1020 and the ramp-step unit CTE instore_postgres_ramp_step.gowork as before. The plan is now disabled as well, so no new step rows are created for it.Verification
Unit tests use pgxmock and integration tests use real Postgres through testcontainers. No cloud calls, no deploy. All tests ran with
AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null, because this machine's default AWS profile is invalid.New real-Postgres tests in
internal/config/store_postgres_delete_account_integration_test.go:TestDeleteCloudAccount_DisablesPlansLeftWithoutAccounts: covers a sole-account plan (disabled), a plan shared with another account (stays enabled,updated_atunchanged), an already disabled plan (updated_atunchanged), a plan on another account (untouched), and a plan with no accounts (untouched).TestDeleteCloudAccount_RefusedDeleteKeepsPlanEnabled: arunningexecution blocks the delete, and the plan stays enabled.TestDeleteCloudAccount_DisablesPlanThatGainedAccountConcurrently: an open transaction moves a plan from account B to account A. The delete of A is started, the test waits until the delete is blocked on a lock, then the writer commits. The plan must end up disabled.Fail before / pass after:
RefusedDeleteKeepsPlanEnabledpasses on main as well. It guards the rollback and is not a regression test.Mutation probes, 5/5 killed by the integration tests:
FOR UPDATEdropped)DisablesPlanThatGainedAccountConcurrently,TestAccountDeleteWaitsForCommandAccountShareNOT EXISTS (other account)droppedDisablesPlansLeftWithoutAccountsEXISTS (this account)droppedDisablesPlansLeftWithoutAccounts(no-account plan)p.enabledfilter droppedDisablesPlansLeftWithoutAccounts(updated_atof the already disabled plan)pgxmock: the existing
DeleteCloudAccounttests now expect the lock and the disable query. New tests:TestPGXMock_DeleteCloudAccount_LockFailsRollsBackandTestPGXMock_DeleteCloudAccount_DisablePlansFailsRollsBack.Other checks:
go build ./...,go vet ./internal/config/,go vet -tags integration ./internal/config/go test ./internal/config/ -count=1,go test ./internal/mocks/ -count=1go test -tags integration ./internal/config/ -count=1golangci-lintv2.10.1run --allow-parallel-runners ./internal/config/, with and without--build-tags integration: 0 issuesgocyclo -over 10 internal/config/store_postgres.go: silentgit secrets --scanon all tracked files, in a fresh temp repo with an isolated HOME: clean.scripts/test-git-secrets-allowlist.sh: 42 PASS, 0 FAIL.Not covered: no deployed-environment or real-cloud check.
Out of scope
executePurchaserefuse a plan-scoped execution whose plan has zero accounts. I left that out. Manyinternal/purchasetests (and theMockConfigStoredefault) drive plan-backed executions through the single-account path with zero plan accounts, and the issue fix(purchase): a plan crossing zero-to-N cloud accounts between attempts silently changes every idempotency token #97 pinned-single lineage path re-drives that shape on purpose. Changing it is a separate executor decision. Today a plan-backed execution with no recommendations is already refused (ErrPlanStepNoRecommendations), and recommendations that name the deleted account fail credential resolution (fix(server): exclude uninitialized replicas from readiness #646).PUT /api/plans/:idwithenabled: truedoes not check that the plan has accounts, so an admin can turn a disabled plan with no accounts back on.validatePurchaseSchedulingstill refuses to schedule it (409). A follow-up could add the same check to the update handler.SetPlanAccountsthat re-sends{A}for a plan whose only account is already A, at the same moment A is deleted, can deadlock with the delete (plan FOR SHARE then account, against account FOR UPDATE then plan). Postgres detects it and rolls back one side whole. Nothing is left half-done, and the request can be retried.Maintainer: please mirror the issue's triage labels on this PR:
priority/p2,severity/medium,triaged,urgency/this-quarter,impact/few,effort/s,type/bug.🤖 Generated with Claude Code