Skip to content

fix(decoders): report completion from terminal states instead of stalling - #124

Merged
MagicalTux merged 1 commit into
masterfrom
fix/decoder-terminal-state-stall
Aug 16, 2026
Merged

fix(decoders): report completion from terminal states instead of stalling#124
MagicalTux merged 1 commit into
masterfrom
fix/decoder-terminal-state-stall

Conversation

@MagicalTux

Copy link
Copy Markdown
Member

Follow-up to #123, which fixed this class of bug in deflate/deflate64/zlib. Reviewing all 53 raw_decode implementations found the same defect in seven more codecs.

The defect

A decoder parked in its terminal state that returns done: false with nothing consumed and nothing written — while the caller still holds input — is mapped by the RawDecoderDecoder bridge to Status::OutputFull, i.e. "drain and call me again", with nothing left to progress on. Any caller looping until StreamEnd spins: CPU-bound, no allocation, so neither an output-size cap nor a memory cap can stop it.

Fixed

codec why it stalled
zstd, xz Done arms returned false; both terminal (neither supports concatenated frames/streams)
lz4 block, lzo Done after the zero-length terminator block
lz4 frame only reachable by calling past StreamEnd, but the state is terminal either way
lzx, amiga_lzx Done arms returned false
rar5 returns on State::Done before the block that accepts caller bytes — a container passing the next header after the payload could never make progress
gzip see below

gzip is the subtle one. Its BetweenMembers arm sets phase = Done intending to reach the Done arm that swallows trailing bytes (added precisely to avoid this spin). But match arms do not fall through, so the loop's no-progress check returned first and the swallow never ran. It needs an explicit continue, like the adjacent 0x1F path already has.

Two call shapes

Only the first was covered before:

  1. trailing bytes present in the same call
  2. trailing bytes arriving after the payload completes — what a container does when handing over the next header

xz and gzip pass shape 1 and failed shape 2; both looked safe until the second shape was tested. Tests cover both.

What is not a bug

Codecs that only ever report InputEmpty and produce output on finish (brotli, lzma, and the other buffering decoders) are correct as-is — the trait permits it and it terminates. The invariant asserted is therefore not "must report StreamEnd" but "must never report OutputFull with zero progress".

Verification

Each fix was found by reading the state machine and confirmed with a probe driving real encoder output plus trailing bytes. A sweep over all 50 round-trippable algorithms now reports no stalls. tests/terminal_state.rs covers every fixed codec plus controls; rar5 is decoder-only so it gets equivalent tests against its own fixture.

cargo test --all-features: 1766 pass, 0 fail. fmt and clippy -D warnings clean.

…ling

Follow-up to #123. That fix covered deflate/deflate64/zlib; reviewing all
53 `raw_decode` implementations found the same defect in seven more.

A decoder parked in its terminal state that returns `done: false` with
nothing consumed and nothing written, while the caller still holds input,
is mapped by the RawDecoder->Decoder bridge to `Status::OutputFull` — "call
me again" — with nothing to progress on. Any loop waiting for StreamEnd
spins: CPU-bound, no allocation, so no output or memory cap catches it.

Fixed, each verified by reading its state machine and confirmed with a
probe that drives the real encoder output plus trailing bytes:

- zstd, xz          Done arms returned false; both are terminal (neither
                    supports concatenated frames/streams).
- lz4 block, lzo    Done after the zero-length terminator block.
- lz4 frame         only reachable by calling past StreamEnd, but the state
                    is terminal either way.
- lzx, amiga_lzx    Done arms returned false.
- rar5              returns on `State::Done` *before* the block that accepts
                    caller bytes, so a container passing the next header
                    after the payload could never make progress.
- gzip              subtler: the `BetweenMembers` arm sets `phase = Done`
                    intending to reach the Done arm that swallows trailing
                    bytes, but match arms do not fall through, so the loop's
                    no-progress check returned first and the swallow never
                    ran. Needs an explicit `continue`, like the 0x1F path.

Two call shapes matter and only the first was covered before: trailing bytes
in the same call, and trailing bytes arriving *after* the payload completes.
The second is what a container does when it hands over the next header, and
it is what xz and gzip failed — both looked safe under the first shape.

Codecs that only ever report `InputEmpty` and produce output on `finish`
(brotli, lzma, and the other buffering decoders) are correct as-is: the
trait permits that, and it terminates. The invariant asserted is not "must
report StreamEnd" but "must never report OutputFull with zero progress".

A sweep over all 50 round-trippable algorithms now reports no stalls.
tests/terminal_state.rs covers each fixed codec plus controls; rar5 is
decoder-only so it gets equivalent tests against its own fixture.
@MagicalTux
MagicalTux force-pushed the fix/decoder-terminal-state-stall branch from 62f36b5 to 8ecc1a9 Compare August 16, 2026 21:30
@MagicalTux
MagicalTux merged commit 66b7b06 into master Aug 16, 2026
46 checks passed
@MagicalTux
MagicalTux deleted the fix/decoder-terminal-state-stall branch August 16, 2026 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant