Repository navigation
esplora: serve the BlindBit oracle v1 API for silent-payment wallets - #1169
Conversation
With --sp-tweaks, the Esplora listener serves the v1 HTTP API that spdk's BlindBit backend calls (Dana): info, block-height, block-hash, tweaks and tweak-index with dustLimit, utxos, both BIP158 filters, spent-index, and forward-tx. Tweaks and utxos read the thin tweak index; tweaks is cut-through. dustLimit keeps a tx whose largest P2TR output is at least the limit, as BlindBit does; it is not --sp-tweaks-dust. The filters and the spent index come from one index-window read of the block and the parents it spends, so nothing new is stored. Where spdk and BlindBit differ, spdk wins: spent-index is always an object, forward-tx answers the txid as a JSON string, and info.network is a Core chain name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…asks Each answer read confirmed[h] once for the hash and again for the data. A reorg between the two reads could key a filter or salt the spent index with one block's hash over another block's contents. The header fk from the first read now travels with the request: the index window is read only if confirmed[h] still names it, and a thin-index read is checked after. A moved or disconnected height is 503 chain view moved. The filter path no longer reads witnesses. It took txids by asking for the tweak window, which also reads seqsigwit of every P2TR-output tx. Txids now come from one page-grouped read of only the parents whose spent output is P2TR. All /blindbit/v1 routes share one limit of 32 requests in flight. A spdk scan keeps about 200 filter asks open; each reads a block and its parents. axum layers each route separately, so this is one global semaphore rather than a per-route limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![rearden-grok[bot]](https://avatars.githubusercontent.com/u/730881?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verdict
REQUEST_CHANGES. I would not merge c7798301 as-is. The BlindBit v1 shapes spdk parses match the upstream client and the v1 oracle on the cases checked below, and /blindbit/v1 stays unmounted unless --sp-tweaks was on when the listener bound. Two problems are enough to hold it: filter and spent-index builds are an uncached parent-window read on the public listener, and GET /info reports height 0 when the index has no served height.
High
Per-request filter and spent-index reads will not survive the client this API is for. block_sp_elements calls index_heights(h, h, Some(h)) and read_index_window. That loads the block's outputs and input edges, every parent txout the block spends, and, because the tweaks flag is set, seqsigwit for every transaction with a P2TR output. The spent hash only needs the parent txid; the witness bytes are unused. The comment that this is bounded by the block is wrong: parent txout bodies live in earlier blocks. One spending block can pull output records from many prior transactions, each up to the 64 MiB body-read cap.
spdk's v1 backend runs buffered(200) (CONCURRENT_FILTER_REQUESTS) and, per height, requests tweaks then /filter/new-utxos then /filter/spent. Those two filter routes each do a full window read. Esplora's default cap is 256 connections, and the request timeout is the 120s idle timeout. spawn_join uses spawn_blocking. When the timeout drops the handler, dropping the JoinHandle detaches that task, so the concurrency slot frees while the read keeps going. A Dana scan, or anyone else holding ~200 of these open, is a sustained multi-gigabyte disk and RAM load. Nothing is cached. served_block only rejects h > served tip, so /filter/* and /spent-index also run this for every height from 0 up to the index tip, including pre-Taproot blocks that thin_rows correctly 404s.
Medium
GET /info advertises genesis when the index cannot serve. served_height is None while the table is empty or still below sptweaks_origin (mainnet origin is Taproot activation, and backfill lags the tip). info then sends "height": 0. block-height returns 503 in that same state. spdk deserializes height as bitcoin::absolute::Height, and Height::from_consensus(0) is Height::ZERO, so the client accepts genesis as the indexed tip. A real indexed genesis (regtest, origin 0, next height 1) is indistinguishable from not ready.
Low
store_error turns every QueryError into 500 and e.to_string(). The rest of Esplora maps Stale, Rejected (including body read too large), and Pruned to 503 or 404. A parent span that trips the body cap, or a disconnect racing index_heights, becomes a 500 with the internal message.
Nits
- The journey covers the happy path (dust
>=boundary, cut-through vs full index, empty spent-index object, filter membership, flag off, one acceptedforward-tx). It does not cover an empty index, a height below origin, the index lagging the tip, a reorg, or a broadcast the mempool must reject. max_eligible: usize::MAXdoes not change single-height behavior:load_thin_tweaks_rangealways keeps the first height even when the default cap is set.
Checked, and sound
No Cargo.toml or Cargo.lock change. Routes are merged only when sptweaks_enabled() is already true at run_esplora; startup index mode runs before the listener binds. The off test gets 404 on /blindbit/v1/info and 200 on /blocks/tip/height.
Wire format, checked against setavenger/blindbit-oracle branch v1 (cfilter.go, spentutxos.go, api.go) and cygnet3/spdk backends/blindbit-v1:
dustLimitkeeps a row when the largest P2TR value is>=the limit. Cut-through (/tweaks) uses outputs still unspent at the tip;/tweak-indexuses every P2TR output. That matches BlindBit'sHighestValue >= dustLimitand its mutable cut-through index. Disconnect truncates the tweak table through the new tip, so a replaced height is not served until it is rewritten.- Spent hash is
sha256(internal txid || vout_le || internal spending-block hash)[..8], the same layout as spdkinput_hashes_mapand BlindBitSerialiseToOutpointplus the reversed block hash. Empty spent-index is an object, which is what spdk can parse. - Filters are BIP158 GCS P=19, M=784931, keyed by the internal block hash the same way
encode_basic_filterand btcsuiteDeriveKeydo,filter_type4. New-utxos elements are the x-only key of every P2TR output, eligible or not.GcsFilterWriterdedupes viaBTreeSet; N is inside the filter, so a repeated key still matches.is_p2trrequires a 34-byte script before the[2..]copy. networkisbitcoin::Network::to_core_arg()(main/test/signet/regtest), which is whatNetwork::from_core_argaccepts. The test only checks regtest.POST /forward-txis the same admit path asPOST /tx:accept_tx_asynctoaccept_tx_from(..., None),waiveis false, thenmark_local_origin. It does not skip standardness, fee, or RBF. The existing 1 MiBRequestBodyLimitLayercaps the body. The success body is a JSON txid string, which spdk parses asTxid. A policy failure is 400{"error"}, which fails that parse. This is not a relay bypass.
Build and tests
Detached worktree at c7798301d845be839836344ae1a6a04cd117ffc8, toolchain 1.95.0.
cargo fmt -p rbitcoin-esplora -- --check: exit 0cargo clippy -p rbitcoin-esplora --all-targets -- -D warnings: exit 0, Finished dev in 16.22s, no warningscargo test -p rbitcoin-esplora --test suite -- --quiet: exit 0, 36 passed, 0 failed, 0.85s
Those 36 are the whole esplora suite, including both new tests (blindbit_v1_routes_serve_the_thin_tweak_index, blindbit_v1_routes_need_the_tweak_index) via include! in tests/server.rs.
| Json(json!({ | ||
| // `spdk` parses this with `Network::from_core_arg`. | ||
| "network": st.network.to_core_arg(), | ||
| "height": served_height(&st.query).unwrap_or(0), |
There was a problem hiding this comment.
unwrap_or(0) makes "index not ready" look like genesis. served_height is None until next_height is past sptweaks_origin (mainnet: Taproot activation, and backfill lags). spdk deserializes InfoResponse.height as bitcoin::absolute::Height, and Height::from_consensus(0) succeeds (Height::ZERO), so a client treats 0 as the indexed tip. block_height already returns 503 in this state. Return that same unavailable status from /info, and add a test where tweaks are enabled, origin is above 0, and nothing has been indexed yet.
| #[allow(clippy::result_large_err)] // axum Response, as in handlers | ||
| fn served_block(query: &Query, raw: &str) -> Result<(u32, [u8; 32], u32), Response> { | ||
| let h = parse_height(raw)?; | ||
| if served_height(query).is_none_or(|top| h > top) { |
There was a problem hiding this comment.
This only rejects h > served tip. Heights below the tweak origin still pass, and header_at_height succeeds for them. thin_rows then 404s, but /filter/new-utxos, /filter/spent, and /spent-index go on to block_sp_elements. On a synced node that is every block from genesis through the index tip, not the Taproot-onward range COMPAT.md describes. Reject h < sptweaks_origin here with the same 404 as an unindexed height.
| h: u32, | ||
| block_hash: &[u8; 32], | ||
| ) -> Result<BlockSpElements, QueryError> { | ||
| let heights = query.index_heights(h, h, Some(h))?; |
There was a problem hiding this comment.
Some(h) sets IndexHeight.tweaks, so each call reads the block, every parent txout it spends, and seqsigwit of every P2TR-output tx. The filter and spent-index only need output scripts and parent txids; the witness bytes are unused. Parent txout bodies are not bounded by this block. spdk issues these two filter routes with buffered(200), and the listener default is 256 in-flight requests with a 120s timeout. spawn_join detaches the spawn_blocking task when that timeout fires, so the concurrency slot frees while the read continues. There is no cache. This needs a stored or memoized result, or a small semaphore that actually cancels the blocking read, before it is safe on the public listener. A test that only checks match_any on a 2-tx chain does not cover this cost.
There was a problem hiding this comment.
Verdict
REQUEST_CHANGES. Head 3fd53615a1e33da6fdeb1a72cd208cd6fa4c4e96 (merge base cc070629f1fc7856c811ae6e0bd0cd43dbcc32ff). The follow-up from c7798301 drops the witness reads and pins a filter answer to one header. The parent-txout read and the info height sentinel are still there.
Prior findings
HIGH — not fixed. block_sp_elements still calls index_heights + read_index_window for /filter/new-utxos, /filter/spent, and /spent-index. read_parents still loads every parent txout the block spends, uncached, per request. /filter/new-utxos never reads spent, so it pays for those parents anyway. served_block admits every h <= min(tip, index top). Origin is taproot activation, so pre-taproot heights take the full read. The "bounded by the block" comment is still in the source.
Witnesses are actually gone: index_heights(..., None) keeps tweaks false, so the window does not read seqsigwit, and txids are one page-grouped txids_get_many of only the P2TR parents (order preserved, duplicates included).
The 32-wide GlobalConcurrencyLimitLayer sits on the nested router, inside TimeoutLayer (default 120s). Handlers spawn_join into spawn_blocking. On timeout the permit drops with the handler and the JoinHandle detaches, so read_index_window continues and the next request starts another. The process blocking pool is available_parallelism (minimum 4); 32 of these reads fill it. The new test holds incomplete forward-tx bodies and probes /blocks/tip/height, which does not use spawn_join.
MEDIUM — not fixed. GET /info still sends "height": 0 when served_height is None (empty index, or next <= sptweaks_origin, including backfill that has not passed taproot). GET /block-height returns 503 no height indexed in that state. spdk treats info.height as a block height, so 0 is genesis. No test.
LOW — partial. Stale is 503, and the reorg path uses it. Rejected (body read too large from ReadJob in the window read) and Pruned still become 500 via e.to_string(). store_err on the rest of the listener maps those to 503 and 404. Keep the JSON {"error"} body and use the same arms.
Nits — partial. In-process reorg of a captured ServedBlock is tested (disconnect, and a replacement block) and returns chain view moved. Still missing: empty index, height below origin, tweak index lagging the tip, and a mempool-rejected forward-tx. The inflight test does not run a filter read.
Nit on the new pin: thin_rows loads by height, then still_confirmed. A height that moves away and back between those calls can return the other block's rows. utxos does this twice. The filter path captures header_fk in index_heights and reads that snapshot, which holds. The stay-moved case is what the test locks.
Checked, no separate finding
txids_get_many zips in request order. is_p2tr requires 34 bytes before the key copy. The GCS writer dedups through a BTreeSet. forward-tx uses the same mempool admit as POST /tx and returns the txid as a JSON string. No new crate, no schema change.
Built and tested
cargo fmt --all -- --check passed. cargo clippy -p rbitcoin-esplora --all-targets -- -D warnings passed. cargo test -p rbitcoin-esplora --test suite -- --quiet: 37 passed, 0 failed.
| if heights.first().map(|x| x.header_fk) != Some(block.header_fk) { | ||
| return Err(StoreError::Stale(CHAIN_MOVED)); | ||
| } | ||
| let window = query.read_index_window(&heights)?; |
There was a problem hiding this comment.
read_index_window is still the body of /filter/new-utxos, /filter/spent, and /spent-index. read_parents loads every parent txout the block spends, uncached, on every request. /filter/new-utxos only uses new_keys from this block's own outputs, so that route still pays for the parent read.
served_block (line 148) rejects only h > served_height, and served_height is min(tip, index top). Origin is taproot activation, so a pre-taproot height is served and takes this full read even though the block cannot contain a P2TR output or a P2TR spend. The comment above still calls that bounded by the block. The parent set is one txout-body read per spent create, anywhere in the chain, repeated per request.
Dropping seqsigwit via index_heights(..., None) and limiting txids_get_many to P2TR parents is the right cut. It leaves the parent-txout read in place. A cap has to be held until that read returns (see the layer on line 54); rejecting h < sptweaks_origin before index_heights removes the pre-taproot case.
| .route("/filter/new-utxos/{height}", get(filter_new_utxos)) | ||
| .route("/filter/spent/{height}", get(filter_spent)) | ||
| .route("/forward-tx", post(forward_tx)) | ||
| .layer(GlobalConcurrencyLimitLayer::new(BLINDBIT_MAX_INFLIGHT)), |
There was a problem hiding this comment.
This semaphore is inside TimeoutLayer in run_esplora (outer → inner: connection cap, body limit, timeout, then the router). The default timeout is idle_timeout, 120s. Filter and spent-index handlers call spawn_join, which is spawn_blocking. When the timeout drops the handler future, this permit is released and dropping the JoinHandle detaches the blocking task, so read_index_window runs to completion. The next request takes the freed permit and starts another read. Thirty-two of those already fill the process blocking pool (available_parallelism, minimum 4), which the other spawn_join routes share.
blindbit_v1_routes_share_one_inflight_limit holds incomplete forward-tx bodies (the permit is taken in the extractor, before spawn_blocking) and then requests /blocks/tip/height, which reads the tip on the async worker. That shows the HTTP semaphore is shared across the prefix. It does not show a timed-out filter read keeping its slot until the window read returns.
| Json(json!({ | ||
| // `spdk` parses this with `Network::from_core_arg`. | ||
| "network": st.network.to_core_arg(), | ||
| "height": served_height(&st.query).unwrap_or(0), |
There was a problem hiding this comment.
served_height is None while the tweak index is empty or next <= sptweaks_origin (backfill that has not passed taproot activation). This sends "height": 0. block_height returns 503 no height indexed for the same state. spdk parses info.height as a block height, so 0 is indistinguishable from a real genesis. Return 503 here as well (or omit the field) until served_height is Some. The suite never starts the listener in that state.
| fn store_error(e: QueryError) -> Response { | ||
| match e { | ||
| StoreError::Stale(m) => error(StatusCode::SERVICE_UNAVAILABLE, m), | ||
| e => error(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()), |
There was a problem hiding this comment.
Stale is 503, which is what the reorg path needs. This wildcard still turns every other QueryError into 500 with e.to_string(). read_index_window can return Rejected("body read too large") from ReadJob. The listener's store_err maps Rejected to 503 and Pruned to 404 (and NotFound to 404). Same status codes, with this JSON {"error"} body.
| tokio::time::sleep(Duration::from_millis(50)).await; | ||
| } | ||
| assert!(waited, "a full BlindBit limit makes the next ask wait"); | ||
| let (st, _) = tokio::time::timeout(quick, http_get(addr, "/blocks/tip/height")) |
There was a problem hiding this comment.
Nit: /blocks/tip/height reads query.tip_height() on the async worker and does not call spawn_join. It stays fast while BlindBit permits are held in the body extractor, and it would also stay fast if those permits were inside read_index_window on the blocking pool. The assertion does not show the rest of the listener is clear of the filter read.
Same gap as the previous nits: nothing covers an empty tweak index (info vs block-height), a height below sptweaks_origin, the index lagging the tip, or a consensus-valid forward-tx the mempool rejects.
Follow-up: BlindBit filter routes vs Electrum tweak serving, and a fix path without a store changeTwo follow-ups to the review above: why the filter and spent routes are heavier than the existing Electrum silent-payment tweak serving, and what a fix looks like given that a store or schema change is off the table. Comparison with Electrum tweak servingThe High finding stands for the three filter and spent routes, and it is worse than Electrum tweak serving.
spdk keeps about 200 HTTP calls open and asks for both filters per height, so one rescan is hundreds of full window reads. Cake and kiss-bdk sit on one Electrum socket; the server will not start the next wave until it has written the last one, and pre-taproot costs nothing.
Fix path without a store or schema changeConstraint from the maintainer: no store or schema change, so precomputing filters, keys, or spent hashes at index time is out. Within that: The filter routes can be served without a schema change. The current spdk's scan (
Recommendation: (a) fixable without a store change. Stop calling |
…ed under thin_rows checked confirmed[h] after the read, which an A -> B -> A move between the plan and the check would pass. The plan reads confirmed[h] and the tweak row together under the sp_tweaks lock, so return that fk from the loader and compare it with the pinned block. An unindexed height whose block left the chain is now a moved view (503), not 404.
There was a problem hiding this comment.
Verdict
REQUEST_CHANGES. Head b4369771acfc1f56366ad6939b037a5a63d945be (merge base cc070629f1fc7856c811ae6e0bd0cd43dbcc32ff). The two commits since 3fd53615 pin thin tweak rows to the header they were planned under, and the new COMPAT sentences match the code below sptweaks_origin once a height is served. The parent-txout read and the info height sentinel are still in the handler.
Prior findings
HIGH — still open. block_sp_elements still calls index_heights and read_index_window for /filter/new-utxos, /filter/spent, and /spent-index. read_parents still loads every parent txout the block spends, uncached, on every request. /filter/new-utxos builds its key list from that same window and leaves spent unused. served_block admits every h <= min(tip, index top). After the index passes the origin, pre-taproot heights take the full read. The comment "Bounded by the block, not the chain" is still on the function. Parent txout bodies live in earlier blocks, each read capped at 64 MiB. COMPAT now says block-hash, filter/*, and spent-index answer those heights. That matches the code and leaves the read in place.
The 32-wide GlobalConcurrencyLimitLayer sits on the nested router, inside TimeoutLayer (120s idle) and inside client-disconnect cancellation. Handlers spawn_join into spawn_blocking. Dropping the JoinHandle detaches the task, so the permit returns while read_index_window keeps running. The node runtime caps that pool at available_parallelism (minimum 4) in blocking_pool_size. The inflight test still holds incomplete forward-tx bodies and probes /info plus tweak-index/0. It never starts a filter read.
MEDIUM — still open. GET /info sends "height": 0 when served_height returns None: the tweak table is empty, or sptweaks_next_height <= sptweaks_origin (backfill that has not passed taproot). GET /block-height returns 503 no height indexed in that same state. spdk reads info.height as a block height, so 0 is genesis. A real indexed genesis (regtest, origin 0, next height 1) is the same JSON. The new COMPAT /info row says height is the served height. No test.
LOW — still open. Stale is 503, and the reorg path uses it. Rejected (body read too large from ReadJob in the window read) and Pruned still become 500 via e.to_string(). store_err on the rest of the listener maps those to 503 and 404.
Nits — partly addressed. The new assertion locks thin_rows at 503 after disconnect_tip while the height is unindexed. That is the empty-plan arm plus still_confirmed. Still missing: an empty index, a height below origin, the tweak index lagging the tip, and a mempool-rejected forward-tx. The inflight test does not run a filter read. The steady-state replacement block was already 503 before this pin, because still_confirmed failed after the load.
What the new commits do
f572fd57 adds header_fk to HeightPlan. thin_tweak_height_plans reads confirmed[h] and get_eligible_range in one sp_tweaks critical section. thin_rows returns those rows only when that fk is the caller's ServedBlock. The txout join runs after the lock, keyed by header_txs of that fk. Disconnect leaves Class A fks and header_txs in place, so an A → B → A move cannot attach the other block's rows to the old fk. The new test does not interleave that move. The structure does.
Below the origin, locate returns None for height < origin, the plan is empty, and thin_rows answers 404 when the served header is still there. That is what COMPAT says for tweaks, tweak-index, and utxos. block-hash, filter/*, and spent-index go through served_block only, so they answer any height up to the served height, including below the origin, and the filter routes do the window read. Until next > origin, served_height is None and every height route is 404. COMPAT's "up to the served height" matches that only once a served height exists.
On the matching-fk arm, thin_rows returns without still_confirmed. A reorg that commits during the txout join is a 200 of the planned block. A check on that arm is safe now that the rows are bound to fk, and it would answer 503. The comment that a later confirmed[h] check would accept A → B → A describes the old load-by-height gap. This load does not re-read the tweak table by height.
No Cargo.toml or schema change. load_thin_tweaks_range keeps its old return type.
Built and tested
Worktree at b4369771acfc1f56366ad6939b037a5a63d945be, toolchain 1.95.0.
cargo fmt --all -- --check: exit 0cargo clippy -p rbitcoin-esplora --all-targets -- -D warnings: exit 0cargo clippy -p rbitcoin-query --all-targets -- -D warnings: exit 0cargo test -p rbitcoin-esplora --test suite -- --quiet: exit 0, 37 passed, 0 failed, 0.96s
Those 37 are the esplora suite, including blindbit_v1_routes_serve_the_thin_tweak_index (with the new 503 assertion), blindbit_v1_routes_need_the_tweak_index, and blindbit_v1_routes_share_one_inflight_limit.
| if heights.first().map(|x| x.header_fk) != Some(block.header_fk) { | ||
| return Err(StoreError::Stale(CHAIN_MOVED)); | ||
| } | ||
| let window = query.read_index_window(&heights)?; |
There was a problem hiding this comment.
HIGH. This is still index_heights plus read_index_window for /filter/new-utxos, /filter/spent, and /spent-index. read_parents loads every parent txout the block spends, uncached, per request. /filter/new-utxos never uses spent and still pays for those parents. served_block allows every h <= min(tip, index top), so once the index has passed sptweaks_origin a pre-taproot height takes the same read. The "Bounded by the block, not the chain" comment above is wrong: parent txout bodies are earlier blocks, each capped at 64 MiB inside ReadJob. COMPAT now says these routes answer below the origin, which matches this function and leaves the read in place. Stop calling read_index_window here: new-utxos from this block's txout span, spent filter and spent index from input edges plus txids_get_many, and an empty answer below sptweaks_origin with no IO.
| .route("/filter/new-utxos/{height}", get(filter_new_utxos)) | ||
| .route("/filter/spent/{height}", get(filter_spent)) | ||
| .route("/forward-tx", post(forward_tx)) | ||
| .layer(GlobalConcurrencyLimitLayer::new(BLINDBIT_MAX_INFLIGHT)), |
There was a problem hiding this comment.
HIGH. This semaphore is inside TimeoutLayer (120s) and inside client-disconnect cancellation. The handlers spawn_join into spawn_blocking, and dropping that JoinHandle detaches the task, so the permit comes back while read_index_window is still on the blocking pool. The node runtime sizes that pool with blocking_pool_size (available_parallelism, minimum 4). Hold the permit until the read returns. blindbit_v1_routes_share_one_inflight_limit fills the limit with incomplete forward-tx bodies and never runs a filter read, so it does not lock this.
| Json(json!({ | ||
| // `spdk` parses this with `Network::from_core_arg`. | ||
| "network": st.network.to_core_arg(), | ||
| "height": served_height(&st.query).unwrap_or(0), |
There was a problem hiding this comment.
MEDIUM. served_height is None when the tweak index is empty or next <= sptweaks_origin, and this sends "height": 0. block_height returns 503 no height indexed in that state. spdk accepts 0 as genesis, and a real indexed genesis is the same JSON. COMPAT's /info row says height is the served height. Use the same 503 as block-height when there is no served height, and add a test for the empty index and for a next height still at the origin.
| fn store_error(e: QueryError) -> Response { | ||
| match e { | ||
| StoreError::Stale(m) => error(StatusCode::SERVICE_UNAVAILABLE, m), | ||
| e => error(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()), |
There was a problem hiding this comment.
LOW. Stale is 503. Rejected (body read too large from the window ReadJob) and Pruned still fall through to 500 and e.to_string(). store_err maps those to 503 and 404. Keep the {"error"} body and use the same arms.
| // Checking `confirmed[h]` again afterwards would pass an A -> B -> A move. | ||
| match query.load_thin_tweaks_range_with_header(Height(block.height), limits) { | ||
| Ok(mut batch) => match batch.pop() { | ||
| Some((_, fk, rows)) if fk == block.header_fk => Ok(rows), |
There was a problem hiding this comment.
Nit. This arm returns the planned block's rows with no still_confirmed after the txout join. A reorg that commits during that join is a 200 for a height that has moved. The rows are already bound to fk (eligible tweaks captured with confirmed[h] under the sp_tweaks lock, txouts joined from that header's first_id), so a still_confirmed here would 503 and would not bring back the old A → B → A mix. The comment above describes that old load-by-height gap.
| matches!(block_sp_elements(&q, &served), Err(rbitcoin_store::StoreError::Stale(m)) if m == CHAIN_MOVED), | ||
| "a disconnected height is a moved view, not a 500" | ||
| ); | ||
| let status = thin_rows(&q, &served, false).err().map(|r| r.status()); |
There was a problem hiding this comment.
Nit. This locks the empty-plan arm: after disconnect_tip, thin_rows is 503 rather than 404. It does not cover an empty index, a height below sptweaks_origin, the tweak index lagging the tip, or a mempool-rejected forward-tx. The replacement-block 503 further down was already true before the header pin, because still_confirmed failed on the other block. Nothing here interleaves A → B → A, and the inflight test still never reads a filter.
…ore a listener slot Per review on reardencode#1169: - /filter/* and /spent-index no longer call read_index_window. Keys come from one txout span of the block, spends from input edges plus the txid sidefile (Query::block_sp_outpoints). No parent txout is read, so every spent outpoint is hashed, a superset of BlindBit's P2TR-only set that spdk matches the same way. Below the tweak origin both are empty with no IO. - The BlindBit permit is a middleware outside the listener layers, so asks queued on it hold no listener slot. Handlers move it into the blocking read (forward-tx holds it across the body), so a timeout or disconnect no longer frees it while the read runs. Default limit is half the blocking pool; EsploraConfig::blindbit_max_inflight sets it. - /info is 503 with no served height, not height 0. - Store errors use the listener's statuses with the {"error"} body. - thin_rows checks confirmed[h] after the txout join on the matching arm. - Tests: empty index, below origin, index lagging the tip, a mempool-rejected forward-tx, and asks queued on BlindBit leaving a listener slot free.
|
Addressed in fdcf3d0:
|
There was a problem hiding this comment.
REQUEST_CHANGES.
- Fixed.
/filter/*and/spent-indexbuild fromQuery::block_sp_outpointsand return empty belowsptweaks_originwith no read. - Still open for
forward_tx. The other routes park the permit insideheld. - Fixed.
/infois 503 whenserved_heightisNone. - Fixed.
store_errormapsNotFoundandPrunedto 404, andStaleandRejectedto 503. - Fixed.
thin_rowscallsstill_confirmedon the matching header after the join, andblock_sp_elementscalls it after the read.
1 bug, 1 nit.
| Ok(t) => t, | ||
| Err(_) => return error(StatusCode::BAD_REQUEST, "invalid tx"), | ||
| }; | ||
| match mp.accept_tx_async(tx).await { |
There was a problem hiding this comment.
forward_tx awaits accept_tx_async, which spawn_blockings the admit and waits on the JoinHandle. Timeout and client disconnect drop the handler, so _permit drops with it and the detached admit keeps running with the semaphore already free. The next post takes that permit and starts another admit. Call synchronous accept_tx_from inside held. Awaiting accept_tx_async from held would sit one blocking thread on another. The inflight test never gets here; those posts stall in the body extractor.
There was a problem hiding this comment.
Addressed in 9d19e19: the synchronous admit runs inside held. New test blindbit_forward_tx_holds_its_permit_through_the_admit stalls the admit on the pool lock and fails without the fix.
| assert_eq!(queued.len(), 3, "a full BlindBit limit makes the next asks wait"); | ||
| let (st, _) = tokio::time::timeout( | ||
| Duration::from_millis(1000), | ||
| http_get(addr, "/blocks/tip/height"), |
There was a problem hiding this comment.
Nit. /blocks/tip/height is on the main router, whose ConcurrencyLimitLayer this test leaves empty. A 200 does not show that a BlindBit request waiting on the semaphore holds no listener slot.
The admit ran in accept_tx_async's own spawn_blocking, so a timeout or disconnect that dropped the handler freed the permit while the admit kept running. Run the synchronous admit inside held instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Closes #1119. With
--sp-tweaks, the Esplora listener serves the BlindBit oracle v1 HTTP API under/blindbit/v1/, the API spdk'sbackends/blindbit-v1(Dana wallet) calls. Code:crates/rbitcoin-esplora/src/blindbit.rs. No store, schema or flag change.tweaks,tweak-index,utxos) read the existing thin tweak index.dustLimituses BlindBit's edge (largest P2TR output>= dustLimit), not--sp-tweaks-dust.Query::block_sp_outpoints): onetxoutspan for the keys, input edges plus the txid sidefile for the spends. No parent tx is read. As a result the spent index hashes every non-coinbase input, not only P2TR prevouts; spdk only tests membership of its own outpoints, so this answers the same. Below the tweak index origin both are empty and read nothing.spent-indexis always an object,forward-txreturns the txid as a JSON string, andinfo.networkis the Core chain name.chain view moved.infoandblock-heightare 503 until the index passes its origin (notheight: 0).Interaction with #1180: whichever lands second needs a small rebase so both routers share the listener's
GlobalConcurrencyLimitLayer. Until then, taking the BlindBit permit before the listener slot can't be tested, because master's listener limit is per route.Evidence
tests/blindbit_journey.rscovers every route's values: dust edges, cut-through, spent hashes against an independent sha256, both filters decoded with rust-bitcoinBlockFilter::match_anyas spdk does, reorg pins, empty and lagging index, below-origin heights, a mempool-rejectedforward-tx, the mount gate, and the shared limit. The guards fail named assertions when removed.spdk (
c58d7339) scanned a regtest node end to end: filters matched, the spent output was detected, andforward-txbroadcast.Mainnet cost. These numbers come from a synced node on HDD storage, so expect other machines to be faster. Throughput is measured with
curl; CPU is the node's utime+stime. The node had no other load.spent-index, this head, from 789000tweak-index(stored), from 937000A full scan from Taproot activation (~263k blocks) now costs about 45 CPU-minutes, down from 5–24 CPU-hours.
Merge Danger
Door: two-way
Blast Radius: new routes under
/blindbit/v1/, mounted only with--sp-tweaks; existing routes unchanged.rbitcoin-queryaddsload_thin_tweaks_range_with_headerandblock_sp_outpoints;load_thin_tweaks_rangeis unchanged for Electrum.forward-txadmits through the same mempool call asPOST /tx.🤖 Generated with Claude Code