Skip to content

bug: Fix TOCTOU With Budget Leasing - #87

Open
markovejnovic wants to merge 15 commits into
mainfrom
bug/toctou-budget
Open

bug: Fix TOCTOU With Budget Leasing#87
markovejnovic wants to merge 15 commits into
mainfrom
bug/toctou-budget

Conversation

@markovejnovic

Copy link
Copy Markdown
Contributor

There was a bug wherein we acquired budget leases after trying to provision a VM. This has been inverted.

Reserve-before-boot is not yet implemented, so these are RED by design:
try_run/3 boots the VM and only then asks the budget for permission, so a
concurrent herd can put N VMs on a node budgeted for k.

Also records two deferred defects in TODO.txt: the unguarded :erpc.call in
Scheduler.run/4, and the uid leak between Users.claim/0 and Users.bind/2.

Claude-Session: https://claude.ai/code/session_01LCkuEmceEDTAeewU5Csoup
node_state.ex still described Budget.admit/2 as the authoritative
admission check; that function was deleted in favor of Budget.lease/2.
Also mark the budget-admission window in TODO.txt as fixed: capacity
is now leased before boot instead of after.
TODO.txt still quoted the pre-branch try_run/Scheduler.run shapes
(the deleted stop_fun parameter, Budget.Hard.reserve) as if current,
and several line citations into scheduler.ex/node.ex/hyper.ex had
drifted from the edits those files received in this branch. Corrects
the quoted code and citations without changing either entry's open
status or findings.
Hyper.Node.start_vm_or_release/3 did not handle the :ignore
DynamicSupervisor.start_child returns when Hyper.Node.FireVMM.init/1
declines a boot for lack of a budget lease. The unmatched case clause
raised, skipping uid/mutable-layer release and permanently leaking
the uid (merge blocker).

Also folds in the rest of the final review's fix wave: a budget
assertion in the E2E VM lifecycle test, guarding the two republish()
calls in Hard that were provably no-ops, a missing bad-duration
refusal row in the budget config test, and a TODO.txt entry tracking
the now-tractable Hard-restart reconciliation gap.
Closes four residuals from the re-review of the :ignore fix:

- Reword the :ignore comment (lib/hyper/node.ex and the new test's
  moduledoc) to name both causes FireVMM.init/1 can decline for
  (stale routing registration, not just a missing budget lease),
  since a reader debugging :not_admitted would otherwise never
  suspect routing.
- FakeMutable now reports its release to the test process instead of
  silently no-op'ing it, and the test asserts receipt. Verified by
  mutation: deleting Img.Mutable.release(mutable) from the :ignore
  arm now fails the test (confirmed, then reverted).
- Match the module's `_ = if ..., do: republish()` idiom in the
  leaser-DOWN path of hard.ex, which had drifted to a bare `if`.
- Fix a drifted line reference in TODO.txt (:365 -> :370, following
  the @doc false/@SPEC lines added ahead of start_vm_or_release/3).
@codecov

codecov Bot commented Aug 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.27273% with 14 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
lib/hyper/node/fire_vmm.ex 30.00% 7 Missing ⚠️
lib/hyper/node/budget/hard.ex 96.25% 3 Missing ⚠️
lib/hyper/node.ex 80.00% 2 Missing ⚠️
lib/hyper.ex 0.00% 1 Missing ⚠️
lib/hyper/cluster/scheduler.ex 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@github-actions

Copy link
Copy Markdown

Test Results

    3 files   -   1  206 suites   - 21   1m 14s ⏱️ - 2m 9s
  515 tests  - 160  510 ✅  - 164    0 💤 ± 0  5 ❌ +4 
1 022 runs   - 129  510 ✅  - 164  507 💤 +31  5 ❌ +4 

For more details on these failures, see this check.

Results for commit 7b88918. ± Comparison against base commit 3ac24e1.

This pull request removes 195 and adds 35 tests. Note that renamed tests count towards both.
Elixir.Hyper.Node.Budget.HardStatePropertiesTest ‑ property cut undoes bump for any spec
Elixir.Hyper.Node.Budget.HardStatePropertiesTest ‑ property the running total is the sum of bumped specs (hence order-independent)
Elixir.Hyper.Node.Budget.HardStatePropertiesTest ‑ property untrack of a ref that was never tracked yields nil and an unchanged state
Elixir.Hyper.Node.Budget.HardStatePropertiesTest ‑ property untrack returns exactly the spec track stored, leaving the rest unchanged
hyper-suidhelper::config_network ‑ absent_network_table_means_disabled
hyper-suidhelper::config_network ‑ partial_network_table_without_uplink_disables_not_bricks
hyper-suidhelper::config_network ‑ present_table_with_only_uplink_defaults_clone_pool
hyper-suidhelper::config_seam ‑ override_path_is_consulted_through_the_real_lexical_gate
hyper-suidhelper::config_uid_gid_range ‑ absent_range_yields_default
hyper-suidhelper::config_uid_gid_range ‑ min_exceeds_max_always_rejected
…
Elixir.Hyper.Cfg.BudgetTest ‑ test load/0 rejects [boot_lease_ttl: "forever"] with {:error, {:bad_value, :boot_lease_ttl, "forever"}}
Elixir.Hyper.Cfg.BudgetTest ‑ test the lease timings are overridable from the [budget] TOML table
Elixir.Hyper.Node.Budget.HardStatePropertiesTest ‑ property conservation laws allocation is the sum of live entries, whatever order they arrived in
Elixir.Hyper.Node.Budget.HardStatePropertiesTest ‑ property conservation laws claiming a lease does not change what is allocated
Elixir.Hyper.Node.Budget.HardStatePropertiesTest ‑ property inverse laws dropping a lease restores the allocation it took
Elixir.Hyper.Node.Budget.HardStatePropertiesTest ‑ property inverse laws releasing a claimed reservation and re-claiming it restores the allocation
Elixir.Hyper.Node.Budget.HardStatePropertiesTest ‑ property never under-reserve a stale token never drops the lease that replaced it
Elixir.Hyper.Node.Budget.HardStatePropertiesTest ‑ property never under-reserve dropping a claimed vm_id does not release its reservation
Elixir.Hyper.Node.Budget.HardStatePropertiesTest ‑ property never under-reserve expire removes only leases past their deadline, never a reservation
Elixir.Hyper.Node.Budget.HardStatePropertiesTest ‑ property never under-reserve leasing a vm_id that already holds capacity is refused, not overwritten
…

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant