Skip to content

feat(firehose): add async emission mode, shutdown drain, and cursor writing#2

Draft
Copilot wants to merge 3 commits into
mainfrom
copilot/releasereth-1x-reimplement-feature
Draft

feat(firehose): add async emission mode, shutdown drain, and cursor writing#2
Copilot wants to merge 3 commits into
mainfrom
copilot/releasereth-1x-reimplement-feature

Conversation

Copy link
Copy Markdown

Copilot AI commented May 1, 2026

Ports the async emission feature from PR #1 onto release/reth-1.x with a strategy to minimize future merge conflicts: all custom code lives in separate files/crates.

Note: This PR's base branch should be changed to release/reth-1.x.

Changes

  • crates/firehose/ — new reth-firehose crate containing:
    • FirehoseArgs clap struct (--firehose.emission-mode, --firehose.channel-capacity, --firehose.live-threshold, --firehose.cursor-path)
    • init_tracer(Config) -> Option<ShutdownHandle> with process-wide OnceLock
  • bin/reth/src/main.rs — minimal diff wrapping the existing node launch:
    Cli::<EthereumChainSpecParser, FirehoseArgs>::parse().run(async move |builder, args| {
        let cfg = args.to_tracer_config(&data_dir);
        let shutdown_handle = reth_firehose::init_tracer(cfg);
        builder.node(EthereumNode::default())
            .on_component_initialized(move |node| { /* wire drain */ Ok(()) })
            .launch_with_debug_capabilities().await?;
    })
  • vendor/rbase64/ — patched rbase64 2.0.3 with #[global_allocator] (MiMalloc) removed; wired via [patch.crates-io]
  • Bumped firehose-tracer 5.0.0 → 5.1.0 (adds EmissionMode, ShutdownHandle, cursor file support)

Deliberately not ported

Gap detection, re-trace on async gap, and everything related to startup re-emission — per requirements, only cursor writing is included.

Original prompt

Ok we had a great problem working with you where we worked together but I forgot to tell you on which branch to do that. This yielded https://github.com/streamingfast/reth/pull/1/changes#r3174592773 based on main branch.

But I actually wanted that to be based on release/reth-1.x. The merge conflicts would be really hard here. What I suggest is that we open a new PR and replicate the general idea done in the wrong based one. Crates like reth-firehose already exists as well as integration with evm-firehose-tracer-rs and inspector and all block tracing. Two things of context to add on this job:

  • You must take a strategy to minize future diffs. To achieve that, you goal is to implement the feature we need with minimal future diff production. The best way to achieve that is having our custom code all in separate files so we can't create conflict with those and we put smaller call.

Here I propose we changed almost no code and simply wrap .node(EthereumNode::default()) with something like .node(firehose_node::wrap(EthereumNode::default(), firehose_args))

  • Re-trace on async gap, gap detection and everything related to that outside of cursor writing should not be ported.

Created from VS Code.

Copilot AI and others added 2 commits May 1, 2026 19:19
…riting

Bump firehose-tracer to 5.1.0 which adds EmissionMode, ShutdownHandle, and
cursor file support. Add FirehoseArgs CLI args in a separate crate
(reth-firehose) to minimize future merge conflicts. Wire shutdown handle
drain into the node lifecycle via on_component_initialized hook.

Vendor rbase64 with global allocator removed to avoid jemalloc conflict.

Agent-Logs-Url: https://github.com/streamingfast/reth/sessions/92ea0d4c-b97f-43b8-b6de-cb95c46058f1

Co-authored-by: maoueh <123014+maoueh@users.noreply.github.com>
Copilot AI changed the title [WIP] Implement feature with minimal future diffs feat(firehose): add async emission mode, shutdown drain, and cursor writing May 1, 2026
Copilot finished work on behalf of maoueh May 1, 2026 19:25
Copilot AI requested a review from maoueh May 1, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants