design: initial shielded outputs support

[r4mmer]

Design: wallet-service shielded outputs support

Initial design proposal for indexing, registering, and serving shielded outputs in the wallet-service. The work is split into three sibling RFCs:

Doc	Scope
0000-daemon-and-database.md	Schema, ingestion pipeline, reorg/void handling, push/WS event surfaces
0001-wallet-registration.md	Key registration (creation + upgrade), HD derivation, historical re-scan
0002-shielded-outputs-wallet-api.md	Read API surface — balances, history, UTXOs

Motivation

Hathor is introducing shielded outputs, a new output type that hides amount (and optionally token) using Pedersen commitments, Bulletproofs, and surjection proofs. The wallet-service is the canonical indexer for hosted Hathor wallets and must index shielded outputs the same way it indexes transparent outputs.

The challenge is that shielded outputs are physically very different from transparent outputs: rows are an order of magnitude larger (~770–930 bytes of cryptographic material), ownership is determined by a different mechanism (script-extracted spend_partial_address rather than a fullnode-pre-decoded address), and recovering the value requires CPU-bound cryptographic work. Bolting these into the existing transparent code path would balloon row sizes, branch every hot-path query on a new dimension, and couple the well-tested transparent flow to an experimental new flow.

Acceptance criteria

The design aims to satisfy the following goals:

• Transparent path stays stable. A bug in shielded handling cannot regress transparent balances, reads, reorg handling, or any existing endpoint.
• Shielded ingestion mirrors transparent ingestion. Every transparent capability — live ingest, reorg/void/unvoid, mempool, push/WS notifications — has a shielded counterpart with the same semantics.
• Synchronous and consistent. A vertex is either fully indexed (transparent + shielded) or not at all; no inconsistency window between the two views of the same tx.
• Read path stays cheap. No HD derivation on the API hot path; ownership matches are O(1) on an indexed key; unowned outputs cost essentially nothing.
• Existing wallets can opt in. A transparent-only wallet can attach shielded keys later and retroactively claim its historical receives via a bounded, idempotent, restartable catch-up scan.
• API contract stays additive. No /v2, no Accept-Version header. Old clients reading the same endpoints see a strict superset of today's response shape; existing fields keep their meaning.
• Wallet-service never holds spend authority. Only scan-side material (read access) is registered; spending stays a client responsibility.
• Schema is forward-compatible. Encryption-at-rest, nullifier-based consumption tracking, and shielded mint/melt support can all be added later without structural migrations.

Design overview

The decisions below are the ones most likely to attract pushback. Each is argued in the corresponding RFC's Rationale and alternatives section; flagged here so reviewers can jump straight to the tradeoff.

Separate tables, not a kind flag on tx_output

Six new tables (shielded_tx_output, shielded_address, shielded_address_balance, shielded_wallet_balance, shielded_address_tx_history, shielded_wallet_tx_history) hold all shielded state. Extending tx_output with nullable shielded columns and a kind discriminator was rejected, 99% of historical rows would carry columns they never use, every existing query would need a WHERE kind clause, and a bug in shielded ingestion could corrupt rows the transparent code path also reads. The only shared concept across the two paths is the vertex, not the storage layer.

Synchronous in-line rewind on the daemon

When a shielded output's spend_partial_address matches an owned row, the daemon synchronously calls rewindAmountShieldedOutput / rewindFullShieldedOutput (~1 ms per match) inside the same per-vertex DB transaction. Async post-processing was rejected because it would open an inconsistency window between transparent and shielded views and complicate reorg ordering. The match step itself is O(1) on an indexed key, so unowned outputs cost essentially nothing.

Per-index pre-derived scan privkeys

shielded_address.scan_privkey stores each child privkey ahead of time; the API never runs HD derivation on the read path. The wallet-service has two execution environments (daemon, Lambda API) and Lambda CPU per request is cost-sensitive, so HD derivation is moved to the gap-extension boundary which is rare.

Wallet-load endpoint reused for shielded upgrade

There is no separate upgrade endpoint. The existing wallet.load handler (POST wallet/init) is extended with four optional fields (scanXpriv, spendXpub, plus signatures over both) and reconciles its stored state against the request. A reconciliation matrix in the RFC enumerates all six (wallet exists?) × (keys stored?) × (keys submitted?) cases, including a 409 Conflict for the case where submitted keys differ from those already stored.

Storage split, but the API merges by default

Storage stays per-kind (separate tx_output / shielded_tx_output, separate balance/history tables), but GET wallet/balances returns one merged number per token in the existing balance.unlocked field, and GET wallet/utxos returns merged results unless filtered with a new optional kind=transparent|shielded query parameter. A wallet UI almost always wants one number per token; doing the merge once in the wallet-service is cheaper than multiplying it across the client ecosystem. If a future feature needs the per-kind split exposed, the API can grow a query parameter additively.

Backward-compatibility analysis

The RFC includes a per-endpoint table covering the four combinations of {old client, new client} × {wallet has shielded keys, wallet doesn't}. The conclusion: no fund loss in any scenario. The only observable old-client effects are in the rare downgrade scenario (user upgraded, then opened an old client), where they see a cosmetic balance discrepancy or stumble on a shielded UTXO at signing time — both recoverable.

Out of scope

• Encryption-at-rest for scan-side secrets — schema is shaped to absorb it later.
• Range-proof verification by the indexer — currently delegated to the fullnode.
• Wallet deletion / shielded-disable flow — no transparent equivalent exists today.

References

• Upstream parent RFC — Shielded outputs (feat/shielded-outputs)
• Upstream follow-on — Shielded mint/melt RFC
• Existing wallet-service repo — HathorNetwork/hathor-wallet-service