Represent absent pages in Table and snapshots

[gefjon]

Description of Changes

Preparation for freeing empty pages.
Because the snapshot format depends on the density of page vectors and didn't previously reserve a sentinel,
to preserve rollback safety we have to do this preparation step before actually implementing freeing pages as part of the row delete operation.

In this PR, the Table/Pages switches to a Vec<Option<Box<Page>>>, with pages allowed to be absent.
However, until a later patch, outside of tests, no page entry will ever be None. The table code is still able to use and reason about None page entries, as they may arise if we
deploy said later patch, free a page, capture a snapshot, then roll back to this version.

In the snapshot format, absent pages are recorded in the pages vec as the all-zeroes hash. Page objects are not written or read in this case; the all-zeroes hash does not correspond to an actual object on disk.

When allocating a new page, we attempt to fill the lowest empty slot. We do this in log time by storing a BTreeSet of the empty slots, and popping the lowest value from it to use as the slot for the newly allocated page. I believe that for at least some access patterns, this should allow us to gradually converge on a dense array of pages in the case where rows are deleted at a higher rate than new inserts.

As a result of this PR, some operators have changed in such a way as to cause the table crate benchmarks to no longer compile. These benchmarks appear to have already bitrotted significantly. I removed benchmarks sufficient to get cargo check --tests --benches passing, but did not attempt to repair cargo test --benches, as that appears to have been broken prior to this change.

Per Joshua's review, this PR also adds a new metric, spacetime_replay_snapshot_num_absent_pages.
Pages which aren't read from files due to having the all-zeroes hash are counted towards that metric and not towards the existing spacetime_replay_snapshot_num_objects_read.

API and ABI breaking changes

Changes the snapshot format on-disk to recognize a new special sentinel, the all-zeroes hash. When the all-zeroes hash appears in a table's vector of page hashes, it means that no page exists in that slot, and so no object file is read.

As of this PR, it is (or at least, should be) impossible to reach the added codepaths or representation without using a snapshot created by a newer version of SpacetimeDB or through manual editing.

Expected complexity level and risk

3: if it is mistakenly possible to put None in a Pages within this PR, then the release that introduces this commit may not be rollback-safe. Also, this touches the datastore, which contains unsafe code, though no new unsafe code is introduced, nor is any of it modified in any ways that would affect safety invariants.

Testing

• A couple new automated tests in the table crate.
• Manually tested rollback safety by using Free pages which become empty during a delete #5771 to construct a snapshot with an absent page, then used this PR to replay it. Described in more detail in comment below.

[gefjon]

Manual testing of rollback safety:

$ git status
On branch phoebe/free-empty-pages/free-during-delete
# That's the branch from the next PR, #5771.

$ nix build
-- SNIP --

$ result/bin/spacetime server clear
WARNING: This command is UNSTABLE and subject to breaking changes.

Database path: /home/phoebe/.local/share/spacetime/data
Are you sure you want to delete all data from the local database? [y/N]y
Deleted database: /home/phoebe/.local/share/spacetime/data

$ result/bin/spacetime start > server-logs.txt 2>&1 &

# lib.rs for this module is attached.
$ result/bin/spacetime publish --server local --module-path ../../page-alloc-thrashing/spacetimedb/ page-alloc-thrashing
-- SNIP --

$ result/bin/spacetime logs -s local page-alloc-thrashing -n 10
2026-08-21T15:41:13.504599Z  INFO: run_loop src/lib.rs:82: Running loop iteration 169880
2026-08-21T15:41:13.506351Z  INFO: run_loop src/lib.rs:82: Running loop iteration 169881
2026-08-21T15:41:13.508498Z  INFO: run_loop src/lib.rs:82: Running loop iteration 169882
2026-08-21T15:41:13.509593Z  INFO: run_loop src/lib.rs:82: Running loop iteration 169883
2026-08-21T15:41:13.510622Z  INFO: run_loop src/lib.rs:82: Running loop iteration 169884
2026-08-21T15:41:13.511707Z  INFO: run_loop src/lib.rs:82: Running loop iteration 169885
2026-08-21T15:41:13.512798Z  INFO: run_loop src/lib.rs:82: Running loop iteration 169886
2026-08-21T15:41:13.513887Z  INFO: run_loop src/lib.rs:82: Running loop iteration 169887
2026-08-21T15:41:13.514911Z  INFO: run_loop src/lib.rs:82: Running loop iteration 169888
2026-08-21T15:41:13.516017Z  INFO: run_loop src/lib.rs:82: Running loop iteration 169889

# Wait patiently...
# On my machine, I do about 1024 iterations/TXes in 1.5 seconds.
# I imagine this is mostly attributable to the overhead of the scheduler,
# and could be dramatically improved by having a client shove requests in via a WebSocket.

$ ls ~/.local/share/spacetime/data/replicas/1/snapshots/
00000000000000000000.snapshot_dir  00000000000001000000.snapshot_dir

$ fg
result/bin/spacetime start > server-logs.txt 2>&1
^C

$ hexdump -x ~/.local/share/spacetime/data/replicas/1/snapshots/00000000000001000000.snapshot_dir/00000000000001000000.snapshot_bsatn 
-- SNIP --
# Observe some blocks of zeroes in there.

$ git checkout phoebe/free-empty-pages/snapshot-format-sentinel
Switched to branch 'phoebe/free-empty-pages/snapshot-format-sentinel'

$ nix build
-- SNIP --

$ result/bin/spacetime start > server-logs-reload.txt 2>&1 &
-- SNIP --

$ result/bin/spacetime logs -s local page-alloc-thrashing -n 10 -f
2026-08-21T16:22:23.556253Z  INFO: run_loop src/lib.rs:83: Running loop iteration 929792
2026-08-21T16:22:25.225354Z  INFO: run_loop src/lib.rs:83: Running loop iteration 930816
2026-08-21T16:22:26.881702Z  INFO: run_loop src/lib.rs:83: Running loop iteration 931840
2026-08-21T16:22:28.542179Z  INFO: run_loop src/lib.rs:83: Running loop iteration 932864
2026-08-21T16:22:30.163726Z  INFO: run_loop src/lib.rs:83: Running loop iteration 933888
2026-08-21T16:22:31.774820Z  INFO: run_loop src/lib.rs:83: Running loop iteration 934912
2026-08-21T16:22:33.417616Z  INFO: run_loop src/lib.rs:83: Running loop iteration 935936
2026-08-21T16:22:35.056625Z  INFO: run_loop src/lib.rs:83: Running loop iteration 936960
2026-08-21T16:22:36.699834Z  INFO: run_loop src/lib.rs:83: Running loop iteration 937984
2026-08-21T16:22:38.362725Z  INFO: run_loop src/lib.rs:83: Running loop iteration 939008
2026-08-21T16:28:37.357073Z  INFO: run_loop src/lib.rs:83: Running loop iteration 940032
2026-08-21T16:28:39.163325Z  INFO: run_loop src/lib.rs:83: Running loop iteration 941056
2026-08-21T16:28:40.961482Z  INFO: run_loop src/lib.rs:83: Running loop iteration 942080
2026-08-21T16:28:42.762467Z  INFO: run_loop src/lib.rs:83: Running loop iteration 943104
^C

# It replayed successfully and is now printing more logs, showing that it's doing more transactions.

page_alloc_thrashing_lib.txt

[joshua-spacetime]

LGTM pending the update to the "number of files in snapshot" metric.

[gefjon]

Ah, nuts, I made both of those changes on the wrong branch. Cherry-picking.