Investigate Whether Alloy's Existing Fallback + Retry Logic Covers All Necessary Features and Makes RobustProvider Redundant

[0xNeshi]

@LeoPatOZ found out that fallback + retry logic actually already exists in alloy, one just had to dig deeper to find it (it's documented in the layers section).

If we verify that it supports fallbacks and retries for both one-off RPC calls and subscriptions, we could completely remove RobustProvider.

[LeoPatOZ]

This comment focuses on one-off RPC calls (not subscriptions).

Discovery

Alloy provides two key layers in alloy::transports::layers:

1. RetryBackoffLayer - Automatic retry with exponential backoff
2. FallbackLayer - Misleadingly named - this is actually more parallel racing with ranking

Finding: FallbackLayer

After analyzing the FallbackLayer source code, I found that it does the following:

Behavior	Traditional Fallback	Alloy's FallbackLayer
Request pattern	Try primary → on failure → try next	Race N transports in parallel
Resource usage	1 request at a time	active_transport_count simultaneous requests
Provider selection	Explicit primary/fallback order	Auto-ranked by latency/stability scores

/// **For methods with non-deterministic results** (e.g., `eth_sendRawTransactionSync`):
/// - Methods are tried sequentially on each transport
/// - Returns the first successful response
/// - Prevents returning wrong results (e.g., "already known" instead of receipt)
/// **For methods with deterministic results** (default - most methods):
/// - At the start of each request, we sort transports by score
/// - We take the top `self.active_transport_count` and call them in parallel
/// - If any of them succeeds, we update the transport scores and return the response
/// - If all transports fail, we update the scores and return the last error that occurred

/// What FallbackLayer actually does (simplified):
async fn make_request(&self, req: RequestPacket) {
    let top_transports = self.top_transports(); // Get best N by score
    let mut futures = FuturesUnordered::new();
    
    // Launch requests to ALL active transports in parallel
    for transport in top_transports {
        futures.push(transport.call(req.clone()));
    }
    
    // Return whichever succeeds first
    while let Some(result) = futures.next().await {
        if result.is_ok() { return result; }
    }
}

True Fallback Only for Specific Methods

FallbackLayer has a sequential_methods feature that provides true fallback behavior, but only for specific RPC methods (default: eth_sendRawTransactionSync, eth_sendTransactionSync) but there are methods which allows you to extend this hashset:

// Sequential fallback (true fallback) - only for configured methods
async fn make_request_sequential(&self, req: RequestPacket) {
    for transport in top_transports {
        match transport.call(req.clone()).await {
            Ok(response) => return Ok(response),  // Stop on first success
            Err(_) => continue,  // Try next
        }
    }
}

Transport Limitations

FallbackLayer does not support a mix of transport types (i.e cannot have HTTP and WS transport types)

As it expects a Vec of type S.

#[derive(Debug, Clone)]
pub struct FallbackService<S> {
    /// The list of transports to use
    transports: Arc<Vec<ScoredTransport<S>>>,
    /// The maximum number of transports to use in parallel
    active_transport_count: usize,
    /// Set of RPC methods that require sequential execution (non-deterministic results in
    /// parallel)
    sequential_methods: Arc<HashSet<String>>,

Feature Comparison

Feature	Alloy Native	RobustProvider	Notes
Retry with exponential backoff	✅	✅	Both use exponential backoff
Custom retry policy	✅	✅	Alloy: RetryPolicy trait, RobustProvider: is_retryable_error()
Transport error retry	✅	✅	Both retry on connection failures
True sequential fallback	⚠️	✅	Alloy: only for sequential_methods, RobustProvider: always
Multi-provider support	⚠️	✅	Alloy: parallel racing, RobustProvider: true failover
HTTP + WebSocket mixed	❌	✅	FallbackLayer requires homogeneous transport types
Total call timeout	❌	✅	RobustProvider wraps entire retry cycle with call_timeout
Primary provider preference	❌	✅	RobustProvider has explicit primary + periodic reconnect
Subscription failover	❌	✅	FallbackLayer only works for request/response

Code Examples

Alloy Native: Custom Retry Policy

The retry policy / layer is roughly feature equivalent to the robust provider. As mentioned the RP has additional total call timeout, but this can easily be added by the callee. Whats interesting is that we can add our own retry policies that we defined in the RP

use std::{num::NonZeroUsize, time::Duration};
use alloy::{
    providers::{Provider, ProviderBuilder},
    rpc::client::RpcClient,
    transports::{
        http::{reqwest::Url, Http},
        layers::{FallbackLayer, RetryBackoffLayer, RetryPolicy},
        RpcError, TransportErrorKind,
    },
};
use tower::ServiceBuilder;

// Define a custom retry policy
#[derive(Debug, Clone, Default)]
pub struct MyRetryPolicy;

impl RetryPolicy for MyRetryPolicy {
    fn should_retry(&self, error: &RpcError<TransportErrorKind>) -> bool {
        match error {
            // Retry if the error response is marked as retryable (e.g., 429)
            RpcError::ErrorResp(err) if err.is_retry_err() => true,
            // Retry on transport errors (connection failures, etc.)
            RpcError::Transport(tr_err) => tr_err.is_retry_err(),
            // Default: retry unknown errors
            _ => true,
        }
    }

    fn backoff_hint(&self, _error: &RpcError<TransportErrorKind>) -> Option<Duration> {
        Some(Duration::from_millis(100))
    }
}

// Build provider with retry layer
let retry_layer = RetryBackoffLayer::new_with_policy(
    10,   // max_retries
    1000, // initial_backoff_ms
    100,  // compute_units_per_second
    MyRetryPolicy::default(),
);

let client = RpcClient::builder()
    .layer(retry_layer)
    .http(Url::parse("https://eth.llamarpc.com")?);

let provider = ProviderBuilder::new().connect_client(client);

Interesting Findings

The example shows how you could add your own layer.

[LeoPatOZ]

Alloy WebSocket Subscription Resilience: Current State

Overview

Alloy provides basic resilience for WebSocket subscriptions through keepalive checks and automatic reconnection. This document covers what's currently available and its limitations.

Current Features

1. Keepalive (Connection Health Check)

WsBackend sends WebSocket pings every 10 seconds of inactivity to detect dead connections:

const KEEPALIVE: u64 = 10;

// In WsBackend::spawn()
tokio::select! {
    biased;
    
    // Outgoing messages (highest priority)
    inst = self.interface.recv_from_frontend() => { ... }
    
    // Keepalive ping
    _ = &mut keepalive => {
        if expecting_pong {
            // Missed pong = connection dead
            errored = true;
            break;
        }
        self.socket.send(Message::Ping(...)).await;
        expecting_pong = true;
    }
    
    // Incoming messages
    resp = self.socket.next() => {
        if item.is_pong() {
            expecting_pong = false;
        }
    }
}

Behavior:

• Ping sent if no messages for 10 seconds
• If pong not received before next ping interval → connection considered dead
• Triggers reconnection flow

2. Reconnection

When connection drops, PubSubService handles reconnection:

async fn reconnect_with_retries(&mut self) -> TransportResult<()> {
    let mut retry_count = 0;
    let max_retries = self.handle.max_retries;
    let interval = self.handle.retry_interval;
    
    loop {
        match self.reconnect().await {
            Ok(()) => break Ok(()),
            Err(e) => {
                retry_count += 1;
                if retry_count >= max_retries {
                    break Err(e);
                }
                sleep(interval).await;
            }
        }
    }
}

Configuration via WsConnect:

WsConnect::new("wss://eth.llamarpc.com")
    .with_max_retries(10)              // Default: 10
    .with_retry_interval(Duration::from_secs(3))  // Default: 3s

3. Subscription Persistence

On successful reconnection, subscriptions are automatically re-established:

async fn reconnect(&mut self) -> TransportResult<()> {
    // 1. Create new backend connection
    let old_handle = self.get_new_backend().await?;
    
    // 2. Drain old connection
    while let Ok(item) = old_handle.from_socket.try_recv() {
        self.handle_item(item)?;
    }
    
    // 3. Re-issue pending requests
    for (_, in_flight) in self.in_flights.iter() {
        self.handle.to_socket.send(...)?;
    }
    
    // 4. Re-subscribe all active subscriptions
    self.subs.drop_server_ids();  // Old server IDs invalid
    for (_, sub) in self.subs.iter() {
        // Send fresh eth_subscribe requests
    }
    
    Ok(())
}

What's NOT Available

Feature	Status	Notes
Multi-provider failover	❌	Retries same provider only
Custom retry policy	❌	Fixed linear backoff
Configurable keepalive interval	❌	Hardcoded 10 seconds
Health scoring	❌	No tracking of provider reliability
Primary provider preference	❌	N/A (single provider)
Parallel provider racing	❌	N/A (single provider)

Extension Point: PubSubConnect::try_reconnect()

The PubSubConnect trait provides a hook for custom reconnection logic:

pub trait PubSubConnect: Sized + Send + Sync + 'static {
    fn is_local(&self) -> bool;
    fn connect(&self) -> impl Future<Output = TransportResult<ConnectionHandle>>;
    
    // Override this for custom reconnection (e.g., failover)
    fn try_reconnect(&self) -> impl Future<Output = TransportResult<ConnectionHandle>> {
        self.connect()  // Default: same provider
    }
}

Using the Extension Point: FailoverWsConnect

Implement a custom PubSubConnect that cycles through multiple providers:

pub struct FailoverWsConnect {
    providers: Vec<WsConnect>,
    current: AtomicUsize,
    failures: RwLock<Vec<u32>>,
}

impl PubSubConnect for FailoverWsConnect {
    fn is_local(&self) -> bool { false }

    async fn connect(&self) -> TransportResult<ConnectionHandle> {
        self.providers[self.current.load(Ordering::Relaxed)].connect().await
    }

    async fn try_reconnect(&self) -> TransportResult<ConnectionHandle> {
        let start_idx = self.current.load(Ordering::Relaxed);
        
        // Cycle through all providers
        for i in 0..self.providers.len() {
            let idx = (start_idx + i) % self.providers.len();
            if let Ok(handle) = self.providers[idx].connect().await {
                self.current.store(idx, Ordering::Relaxed);
                return Ok(handle);
            }
        }
        Err(TransportErrorKind::custom_str("All providers exhausted"))
    }
}

Usage:

let connector = FailoverWsConnect::new(vec![
    WsConnect::new("wss://primary.example.com"),
    WsConnect::new("wss://fallback1.example.com"),
    WsConnect::new("wss://fallback2.example.com"),
]);
let client = ClientBuilder::default().pubsub(connector).await?;
let provider = ProviderBuilder::new().connect_client(client);

This could be extended to mimic our current failover logic

Two Approaches to Subscription Resilience

Approach	Layer	Trigger	Scope	Primary Recovery
FailoverWsConnect	Transport (connection)	Connection drop detected by PubSubService	All subscriptions	Optional (add to try_reconnect)
RobustSubscription	Application (wrapper)	recv() timeout	Single subscription	Built-in periodic attempts

FailoverWsConnect (Transport Layer)

• Implements PubSubConnect::try_reconnect()
• Reactive: Triggers when WebSocket connection dies
• Transparent: All subscriptions automatically migrate
• Limitation: Won't catch silent failures (node alive but not sending data)

RobustSubscription (Application Layer)

• Wraps Subscription<T> with timeout-based failover
• Proactive: Triggers when no data received within timeout
• Per-subscription: Each subscription managed independently
• Catches silent failures: Timeout detects unresponsive providers

Summary

Current state: Single-provider reconnection with automatic subscription restoration.

Gap: No built-in failover to alternate providers when the primary is unavailable.

Solution: Implement PubSubConnect with custom try_reconnect() logic (e.g., FailoverWsConnect) for transport-level failover, optionally combined with application-level wrappers like RobustSubscription for timeout-based resilience.

[0xNeshi]

@LeoPatOZ

re: retry + fallback analysis

What I'm getting from all of this is that a viable alternative to RP would be to implement our own fallback layer to supplant alloy's "fallback" (parallel fetch) layer.
Setting aside the technical challenges, would it even be possible to replace alloy's inherent "fallback" layer?

re: subscription analysis

1. Does that mean it could be possible to replace RobustSubscription with a custom PubSubConnect implementation?
2. Would that work for all types of subscriptions (blocks, pending tx, logs...)?
3. Would it be possible to support mixed WS+HTTP subscriptions (as RP does)?