Skip to content

refactor(demo-26): S14 PR 1 — failure modes on resilience libraries - #80

Merged
patterncatalyst merged 16 commits into
mainfrom
refactor/resilience-libraries
Oct 13, 2026
Merged

patterncatalyst merged 16 commits into
mainfrom
refactor/resilience-libraries

Conversation

@patterncatalyst

Copy link
Copy Markdown
Owner

S14 PR 1 of 4 (plan: _plans/s14-resilience-plan.md, steps 0 and 1). Example 26's breaker, retry and bulkhead move from hand-written code to each stack's resilience library.

Stack Breaker Retry Bulkhead
Spring Boot Resilience4j 2.4.0 @CircuitBreaker + fallback method @Retry @Bulkhead
Quarkus SmallRye Fault Tolerance @CircuitBreaker + @Fallback @Retry + @ExponentialBackoff @Bulkhead
.NET Polly v8 pipelines (Microsoft.Extensions.Http.Resilience 10.10.0) HttpRetryStrategyOptions concurrency limiter, no queue
Python pybreaker 1.4.1 through calling() tenacity 9.2.1 AsyncRetrying asyncio.Semaphore
Go gobreaker v2 (unchanged; trial calls now 2) cenkalti/backoff v7 (unchanged) x/sync/semaphore TryAcquire
C++ hand-written (no maintained library on ConanCenter, D3) coroutine loop with sleepCoro std::counting_semaphore

Contract changes (user decisions D1, D2):

  • D1: a full bulkhead rejects at once with 429 in every language. Before, Spring, Quarkus, .NET and Python answered 200 with an error body, and Go and C++ answered 503. verify.sh adds a saturation check and a recovery check, taking 26 from 20 to 22 checks.
  • D2: /breaker-state is {name, state} plus whatever metrics the library exposes; the state is closed, open or half-open.
  • The shared breaker settings are 5 failures to open, 10 s open, and 2 trial successes to close. Polly lets 1 trial call through by design.
  • Quarkus and Go backends move to /process with X-Deadline-Ms, like the other four.

Clean-ups found on the way:

  • The C++ edge used blocking libcurl calls and sleep_for on Drogon event-loop threads. It now uses coroutine handlers and Drogon's async HttpClient, so libcurl is gone (lock cleaned).
  • The C++ backend's slow mode now answers from a timer.
  • Python opens one httpx.AsyncClient in the lifespan.
  • .NET pins Polly 8.8.0 directly, because Microsoft.Extensions.Http.Resilience only sets a >= 8.4.2 floor.

Docs:

  • Appendix M's timeout, retry and breaker codetabs and the 26 READMEs are now condensed excerpts of the examples.
  • In all six decks, the 18 Appendix M code slides became excerpts with matching chips and notes. They previously showed illustrative code that no example backed. All 18 were rendered and read, and deck_fit reports 0.

Verification:

  • Baseline on main was 36/36 (examples 02, 03, 11, 12, 18, 26).
  • verify-matrix.sh 26-failure-modes on the final commit: 6/6, 22/22 each.
  • validate_site, voice-scan and deck_text --check report clean.

…e FT, MEHR 10.10.0, tenacity, purgatory, x/sync)
…nce4j (S14)

The hand-written CircuitBreaker, retry loop and Semaphore are replaced by
@CIRCUITBREAKER (fallbackMethod), @Retry and @bulkhead on a BackendClient bean,
configured in application.properties: 5 failures open the breaker for 10 s,
2 trial successes close it; 3 attempts from 100 ms doubling with 50% jitter;
5 concurrent calls, the next refused at once and answered with 429.
/breaker-state is {name, state} plus Resilience4j's own metrics (D2).
…ult Tolerance (S14)

@Retry + @ExponentialBackoff, @CIRCUITBREAKER + @CircuitBreakerName + @fallback
(the fallback takes the cause) and @bulkhead on a Backend bean replace the
hand-written CircuitBreaker, retry loop and Semaphore. A full bulkhead's
BulkheadException is mapped to 429; /breaker-state reads
CircuitBreakerMaintenance (D2). The backend moves to /process with
X-Deadline-Ms and answers a too-small budget with 200, as in the other languages.
… v8 pipelines (S14)

Named pipelines registered with AddResiliencePipeline use the HTTP strategy
options from Microsoft.Extensions.Http.Resilience 10.10.0: retry (2 retries,
exponential from 100 ms with jitter), circuit breaker (5 of 5 failures open it
for 10 s; state through CircuitBreakerStateProvider), timeout (2 s) and a
concurrency limiter of 5 with no queue (429 when full). Polly.Extensions and
Polly.RateLimiting are pinned to 8.8.0, since the resilience package only sets
a >= 8.4.2 floor.
…(S14)

AsyncRetrying (3 attempts, exponential jitter from 100 ms to 2 s) replaces the
retry loop; pybreaker (fail_max 5, reset_timeout 10, success_threshold 2) wraps
the awaited call through its calling() context manager in place of the
hand-written class. One httpx.AsyncClient opened in the lifespan replaces a
client per request; the bulkhead semaphore answers 429 when full instead of
racing wait_for.
…and responses aligned (S14)

gobreaker keeps the breaker and backoff the retry; MaxRequests 2 so two trial
successes close it, as in the other languages, and the retry starts at 100 ms.
The bulkhead is a semaphore.Weighted with TryAcquire (429 when full) instead of
a buffered channel returning 503. The breaker endpoint answers with the shared
fallback body (reason circuit_open | upstream_<status> | upstream_unreachable,
plus the breaker state) and /breaker-state carries the name (D2). The backend
moves to /process with X-Deadline-Ms and answers a too-small budget with 200.
…ient (S14 D3)

No maintained C++ resilience library is on ConanCenter, so the breaker and the
bulkhead stay hand-written, now compact: 5 failures open the breaker for 10 s,
2 trial successes close it, and the fallback reports circuit_open,
upstream_<status> or upstream_unreachable like the other languages. Every
handler is a coroutine: backend calls co_await HttpClient::sendRequestCoro and
the retry delay co_awaits sleepCoro, so no event-loop thread blocks (the old
libcurl calls and sleep_for did). The full bulkhead answers 429 (was 503); the
unused retry_with_backoff helper is gone, and so is libcurl (lock cleaned with
--lockfile-clean). The backend's slow mode answers from a 5 s timer instead of
sleeping on the event loop.
…based examples (S14)

The timeout, retry and circuit-breaker codetabs are now condensed excerpts of
examples/26 in all six languages (Resilience4j, SmallRye Fault Tolerance,
Polly v8 pipelines, tenacity and pybreaker, cenkalti/backoff and gobreaker, and
the C++ coroutine version) instead of illustrative PaymentService snippets.
The READMEs name the libraries, the 429 bulkhead and the Drogon HttpClient.
… of examples/26 in all six decks (S14)

The 18 code slides showed illustrative PaymentService/QuoteService code (and in
C++ a gRPC deadline and an atomic breaker); they now carry condensed excerpts
of the library-based examples with matching chips and notes: Resilience4j,
SmallRye Fault Tolerance, Polly v8 pipelines, tenacity and pybreaker,
cenkalti/backoff and gobreaker, and the C++ coroutine version. Edit files
Spring.fix8, Quarkus.fix7, DotNet.fix9, Python.fix7, Go.fix5, Cpp.fix5; all 18
slides rendered and read, deck_fit 0.
@patterncatalyst
patterncatalyst merged commit 255f8f4 into main Oct 13, 2026
20 checks passed
@patterncatalyst
patterncatalyst deleted the refactor/resilience-libraries branch October 13, 2026 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant