[UTXO-BUG] fix _evict_stale_data_input_txs() fetchall() OOM — apply_transaction() DoS

[Ivan-LB]

Bug

_evict_stale_data_input_txs() loads ALL mempool tx_data_json into memory at once via .fetchall(), called on every apply_transaction() commit:

# BEFORE (vulnerable)
all_mempool = conn.execute(
    "SELECT tx_id, tx_data_json FROM utxo_mempool"
).fetchall()                 # ← loads entire pool into RAM
for mp_row in all_mempool:
    ...

Memory worst-case:
MAX_POOL_SIZE (10,000) × MAX_TX_DATA_JSON_BYTES (262,144 bytes) = 2.56 GB per apply_transaction() call.

Distinct from prior report

The previously-reported mempool_get_block_candidates() fetchall was fixed with the MAX_TX_DATA_JSON_BYTES cap. _evict_stale_data_input_txs() is a separate, independent code path on the block-application critical path that was not addressed by that fix.

Attack vector:

1. Fill mempool with 10,000 transactions near the 256 KB per-tx limit (~2.56 GB total)
2. Mine any valid block → apply_transaction() triggers _evict_stale_data_input_txs()
3. .fetchall() loads 2.56 GB into Python RAM → OOM → node crash

Fix

# AFTER (fixed)
for mp_row in conn.execute(
    "SELECT tx_id, tx_data_json FROM utxo_mempool"
):                           # ← cursor iteration: one row at a time
    ...

SQLite cursor iteration streams one row per iteration — memory stays proportional to a single tx, not the entire pool.

PoC Test Results

node/test_utxo_evict_fetchall_oom_poc.py (new, 2 tests):

Mempool: 10 txs, 1003 KB total tx_data_json
Memory delta from apply_transaction: 1 KB   ← bounded, not proportional to pool
✅ Memory delta well below 50% of mempool data

Ran 2 tests in 0.042s
OK

Bounty Reference

Issue #2819 — Mempool DoS, Medium severity (50 RTC).

RTC Wallet: Ivan-LB

[github-actions]

Welcome to RustChain! Thanks for your first pull request.

Before we review, please make sure:

• Non-doc PRs have a BCOS-L1 or BCOS-L2 label
• Doc-only PRs are exempt from BCOS tier labels when they only touch docs/**, *.md, or common image/PDF files
• New code files include an SPDX license header
• You've tested your changes against the live node

Bounty tiers: Micro (1-10 RTC) | Standard (20-50) | Major (75-100) | Critical (100-150)

A maintainer will review your PR soon. Thanks for contributing!

[Scottcjn]

Hi @Ivan-LB — I see you self-closed PR #6510 and #6511, but both findings are legitimate and reproduce as described:

• [UTXO-BUG] fix compute_state_root() little-endian count_bytes — consensus divergence #6510: compute_state_root() little-endian count_bytes while every other encoding in the module is big-endian — this IS a real consensus-divergence bug per the module docstring ("All nodes with the same UTXO set produce the same root" — currently false). Your PoC test test_count_bytes_endian_divergence is exactly right.

• [UTXO-BUG] fix _evict_stale_data_input_txs() fetchall() OOM — apply_transaction() DoS #6511: _evict_stale_data_input_txs() fetchall() OOM under large mempools is a real DoS path.

These deserve bug-report bounty (#305) at the Medium tier (~15-25 RTC each) plus the fix-PR severity (#2819) at Medium-High depending on Codex audit.

Could you:

1. Reply with your RTC wallet (format: RTC + 40 hex chars) — or use a miner_id string like Ivan-LB if you'd rather
2. (Optional) Re-open the PRs or file fresh ones — your fix is correct and we'd merge them gladly

We're at ~761 holders ($0.10 reference rate) — your work is paid at current rates regardless of when you respond. See rustchain-bounties#12458 for the rate-reduction schedule going forward.

[Ivan-LB]

Hi @Scottcjn, thanks for confirming.
Updated wallet address:

RTC Wallet: RTC64aa3fc417e75224e1574acae906fea34d94d140

Also, since the original branches were deleted the PRs could not be reopened, so I filed fresh ones instead:

• Endianness fix → [UTXO-BUG] fix compute_state_root() little-endian count_bytes — consensus divergence #6525
• OOM fix → [UTXO-BUG] fix _evict_stale_data_input_txs() fetchall() OOM — apply_transaction() DoS #6526

Both include PoC tests that pass. Sorry for the confusion.