Release v1.23.0: poison routing on every transport (ADR-0036) (#12)

* Route identity-less and undecodable bodies as poison on every transport (ADR-0036)

Identity-less JSON objects and invalid UTF-8 inside JSON strings are now
poison instead of unknown-URN on AMQP, Pulsar, Artemis and SQS; Redis
dead-letters identity-less bodies unconditionally; Kafka without retry
tiers degrades to commit + drop per section 6.5. Vendors conformance
v1.7.0 and runs its identity_verdict and pulsar_exhaustion sections.

* Release v1.23.0

Author: muhammetsafak

.github/workflows/ci.yml

@@ -177,7 +177,7 @@ jobs:
           --health-retries 10
     env:
       # The pinned conformance release (ADR-0032). Bump together with tests/conformance/.
-      BQ_CONFORMANCE_VERSION: 1.6.0
+      BQ_CONFORMANCE_VERSION: 1.7.0
       # sha256 of that release's fetch.sh: the script runs on the runner, so it is pinned rather
       # than trusted from a mutable tag. Bump together with BQ_CONFORMANCE_VERSION.
       BQ_FETCH_SH_SHA256: 8bb6ba567b288fb0ac71a70fd17041b2503df119e01998d76fdad372570bf849

.github/workflows/release.yml

@@ -14,7 +14,7 @@ jobs:
     # Recommended for PyPI Trusted Publishing — configure a "pypi" environment +
     # a trusted publisher on PyPI (repo BabelQueue/babelqueue-python, this workflow).
     environment: pypi
-    # redis_layout (conformance 1.6.0) drives the real Lua scripts: a real redis:7 is mandatory,
+    # redis_layout (conformance 1.7.0) drives the real Lua scripts: a real redis:7 is mandatory,
     # exactly as in ci.yml's `conformance` job — without it the cases are reported as failures.
     services:
       redis:
@@ -28,7 +28,7 @@ jobs:
           --health-retries 10
     env:
       # The pinned conformance release (ADR-0032). Keep identical to ci.yml.
-      BQ_CONFORMANCE_VERSION: 1.6.0
+      BQ_CONFORMANCE_VERSION: 1.7.0
       BQ_FETCH_SH_SHA256: 8bb6ba567b288fb0ac71a70fd17041b2503df119e01998d76fdad372570bf849
     permissions:
       id-token: write   # PyPI Trusted Publishing (OIDC) — no API token needed

CHANGELOG.md

@@ -9,6 +9,76 @@ The envelope wire format is versioned separately by `meta.schema_version`
 
 ## [Unreleased]
 
+## [1.23.0] - 2026-10-25
+
+Implements ADR-0036 (post-R1 parity) decisions 2, 4, 5 and 6. A MINOR: every change is a
+consumer-side settle decision, so the envelope stays at `schema_version: 1`. The vendored
+conformance suite moves to v1.7.0 (new sections `identity_verdict` and `pulsar_exhaustion`, both
+passing; `conformance-known-gaps.json` stays empty).
+
+### Changed (behaviour)
+- **A body that does not decode, or has no identity, is poison on every transport (ADR-0036
+  decisions 4 and 6).** A JSON object that has neither a valid string `job` nor a valid string
+  `urn` (ADR-0032 section 6) used to be an *unknown URN*, and on SQS, AMQP, Pulsar, Azure Service
+  Bus, Artemis and the memory transport a body that did not decode was decoded leniently into an
+  empty envelope and retried as an unknown URN, finally dead-lettered as `reason: "unknown_urn"`
+  without the original body. Both are now **poison** and take the poison route of Go's
+  `routePoison`: they are never handed to a handler and never take the unknown-URN strategies
+  (those remain for a valid envelope whose URN has no handler). With the DLQ on (`dead_letter=True`
+  or `on_unknown_urn=DEAD_LETTER`) the message is written to the DLQ at once, with no retry, as the
+  ADR-0035 record (`dead_letter.reason: "poison"`, the original body preserved: a strict JSON
+  object spliced byte for byte, anything else as `raw`, base64 plus `raw_encoding` when it is not
+  valid UTF-8) and acknowledged after the write; `on_unknown_urn=DELETE` drops it (logged); with
+  the DLQ off `RELEASE` is replaced by the bounded `FAIL` path (re-published with `attempts + 1`
+  up to `max_attempts`, then dropped and logged). An operator who relied on a poison body being
+  retried through the unknown-URN strategy, or on a `reason: "unknown_urn"` record for it, sees the
+  new behaviour. The classification is `EnvelopeCodec.poison_cause(raw)` (new, pure): the typed
+  decode of the Go reference plus the identity rule; `decode` / `decode_lenient` are unchanged.
+- **Pulsar poison records carry `reason: "poison"`** (they were `unknown_urn`). The Pulsar
+  transport also reports the reconciled `bq-attempts` / redelivery count to the runtime
+  (`ReceivedMessage.native_attempts`, new), as does SQS (`ApproximateReceiveCount - 1`), so a poison
+  body, which cannot carry `attempts`, is still bounded by `max_attempts`.
+- **Redis: an identity-less body is dead-lettered unconditionally (ADR-0036 decision 4,
+  broker-bindings section 1.1 "Identity-less body").** In both layouts (legacy `:processing`
+  and Laravel-compatible zset) a body that decodes but names no identity (including a bare `null`,
+  which used to follow the bounded `FAIL` path) is handed over as transport poison: it is written
+  to `<queue>.dlq` as the ADR-0035 record with `reason: "poison"` in every configuration
+  (`on_unknown_urn=DELETE` and dead-lettering off change nothing), never handed to a handler and
+  never retried. Nothing else in the Redis transport changes.
+- **Kafka without retry tiers: a poison record no longer blocks the partition (ADR-0036 decision
+  5, broker-bindings section 6.5 degrade-to-delete).** With no `retry_tiers_ms`, dead-lettering off
+  and `on_unknown_urn` not `DELETE`, the bounded retry of a poison record (including an
+  identity-less body) needs a tier and raised, leaving the record uncommitted with `bq-attempts`
+  never advancing. It is now committed, dropped and logged, and the partition advances. A failing
+  handler and `RELEASE` of a valid envelope keep the section 6.5 rule (they still raise
+  `BabelQueueError` without tiers). The start-up warning for a tierless transport now says so; enable
+  `dead_letter` to keep such records in `<topic>.dlq`.
+- A message dropped after its retries are exhausted with no DLQ configured is now logged
+  (`babelqueue.app`, WARNING) instead of being dropped silently.
+
+- **Pulsar: a failed DLQ write or retry publish is negatively acknowledged (Go `Rewind`).**
+  `PulsarTransport` is now a `Rewinder` (`negative_acknowledge`), so the unsettled original is
+  redelivered by the broker instead of staying un-acknowledged for the life of the consumer. A
+  graceful-shutdown release of an in-flight Pulsar message is now a nack as well rather than a
+  re-publish (on a `Shared` subscription the native redelivery count advances; on `Exclusive` /
+  `Failover` it does not, which is harmless: `bq-attempts` carries the attempt limit).
+
+### Fixed
+- **Bytes that are not valid UTF-8 no longer crash the AMQP, Pulsar and Artemis consumers.** They
+  were decoded strictly in `pop`; they are now kept losslessly (`surrogateescape`). Invalid UTF-8
+  is poison wherever it sits, **also inside a JSON string** (`EnvelopeCodec.poison_cause`,
+  error-handling section 7): it is never handed to a handler and is dead-lettered as the ADR-0035
+  base64 record, like on Redis and Kafka.
+
+### Added
+- `EnvelopeCodec.poison_cause(raw, *, require_identity=True)`; `Transport.classifies_poison`
+  (Kafka and Redis classify poison themselves and opt out of the runtime's classification);
+  `ReceivedMessage.native_attempts`.
+- Conformance runners for `identity_verdict` (the core verdict on all 97 bodies, plus the fate on
+  the memory transport) and `pulsar_exhaustion` (all 29 rows, driven through the real
+  `PulsarTransport` and runtime over a duck-typed client), and per-transport poison tests.
+- `ci.yml` / `release.yml` pin conformance `1.7.0` (the `fetch.sh` SHA-256 is unchanged).
+
 ## [1.22.0] - 2026-10-24
 
 Implements ADR-0033 (Redis) and ADR-0034 (Kafka tiers, backoff, poison). The envelope stays at

README.md

@@ -259,9 +259,31 @@ the DLQ on (`dead_letter` or `on_unknown_urn=DEAD_LETTER`) it is dead-lettered o
 written to `<topic>.dlq` (which must exist) and its offset is committed once that write
 is confirmed; a failed write leaves it uncommitted and rewound. With the DLQ off nothing is written:
 `on_unknown_urn=DELETE` drops it, any other strategy retries it through the retry tiers up to
-`max_attempts` and then drops it (without tiers the record is redelivered in place, as above). On
-the Redis legacy (`<queue>:processing`) layout a body that Go's typed decode rejects (neither a JSON
-object nor `null`, or wrong member types) is dead-lettered raw on its first
+`max_attempts` and then drops it. **Without retry tiers** that bounded retry has no tier to use, so
+(with the DLQ off and `on_unknown_urn` not `DELETE`) the poison record is committed, dropped and logged
+instead of blocking the partition (section 6.5 degrade-to-delete, ADR-0036 decision 5); enable
+`dead_letter` to keep it in `<topic>.dlq`. A failing handler without tiers still cannot be retried, as above.
+
+### Poison messages on every other transport
+
+A message body that does not decode into an envelope (not a JSON object, malformed, a wrongly typed
+member, or bytes that are not valid UTF-8, even inside a JSON string) or that has neither a valid
+string `job` nor a valid string `urn` is **poison** on every transport (ADR-0036 decisions 2, 4 and
+6; conformance `identity_verdict`). It is never handed to a handler and never takes the
+unknown-URN strategies, which are for a valid envelope whose URN has no handler:
+
+- DLQ on (`dead_letter` or `on_unknown_urn=DEAD_LETTER`): written to `<queue>.dlq` at once, with no
+  retry, as the ADR-0035 record (`dead_letter.reason: "poison"`, the original body preserved,
+  base64 plus `raw_encoding` when it is not UTF-8), then acknowledged. A failed write leaves it
+  unacknowledged.
+- `on_unknown_urn=DELETE`: acknowledged and dropped (logged).
+- DLQ off: the bounded `FAIL` path (`RELEASE` is replaced by it): retried up to `max_attempts`, then
+  dropped and logged. The count comes from the broker where it has one (SQS receive count, Pulsar
+  `bq-attempts` / redelivery count), otherwise from a bare `{"attempts": n}` body the retry leaves.
+
+On Redis (both layouts) an identity-less body is dead-lettered as poison in **every** configuration,
+including `on_unknown_urn=DELETE` and dead-lettering off (broker-bindings section 1.1). On the legacy
+(`<queue>:processing`) layout a body that Go's typed decode rejects is dead-lettered raw on its first
 delivery and removed from `:processing`.
 
 ### Sharing a Redis queue with Laravel

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "babelqueue"
-version = "1.22.0"
+version = "1.23.0"
 description = "Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers."
 readme = "README.md"
 requires-python = ">=3.9"

src/babelqueue/__init__.py

@@ -42,7 +42,7 @@
     Transport,
 )
 
-__version__ = "1.22.0"
+__version__ = "1.23.0"
 
 __all__ = [
     "BabelQueue",

src/babelqueue/_envelope_shape.py

@@ -15,7 +15,7 @@
   decoder read them) and every occurrence of a duplicated key is checked; any other member (a
   differently-cased name or a non-string ``urn`` alias included) is only checked for syntax.
   A non-string ``job`` is **not** poison either: the identity resolver ignores it (ADR-0032 section 6),
-  so the body is readable and simply carries no identity (conformance ``kafka_poison``, v1.6.0).
+  so the body is readable and simply carries no identity (conformance ``kafka_poison``, v1.6.0, and ``identity_verdict``, v1.7.0).
 
 A well-typed but non-conformant envelope is not poison: Go's consume path never calls ``Accepts``.
 """