Skip to content

Proposer silently skips its slot after every proposer-rotation wrap (slot ≡ 1 mod 512); no failure counted #602

Description

@pablodeymo

Observed

On a 512-validator / 64-node devnet (image devnet5-leanvm-main, rev 691df39, 8 validators/node, 8 subnets), validator 1 has missed 100% of its proposal slots: 8/8 at first measurement (slots 1, 513, 1025, 1537, 2049, 2561, 3073, 3585 — every slot ≡ 1 mod 512), and still 5/5 when re-sampled four days and ~170 rotations later (slots 84481…86529 all absent, adjacent control slots present).

The node never logs Building block for these slots, and lean_block_building_failures_total stays 0 fleet-wide: the interval-4 build tick never fires, so nothing counts the miss.

Context

  • The rotation-wrap block (slot ≡ 0 mod 512, proposer = validator 0) is imported by peers 7–8 s after its slot start at every observed wrap (7.3 s / 7.2 s / 7.8 s at slots 512 / 3072 / 3584).
  • node_1 (host of validator 1, also an aggregator) is seconds behind real time when its slot arrives: its attestations for slot 512k publish 3–7 s late and its aggregation sessions run ~3 s.
  • At 3 of 5 recently sampled wraps the following proposer (validator 2's node) also lost its block — net cost 1–2 blocks per 512-slot cycle, invisible to every failure metric.

Expected

A proposer that cannot build in time should at least increment a failure counter; a deterministic 100%-miss pattern tied to the proposer-rotation wrap should be visible in metrics.

Evidence

  • Chain: devnet-eth35, genesis 2026-08-28 16:53:53Z (GENESIS_TIME 1787936033), hosts ethlambda-3/ethlambda-5.
  • Missed-slot verification via GET /lean/v0/blocks/<slot> (404 for every s ≡ 1 mod 512 sampled; 200 for controls).
  • Full measurement context: 3.9 h instrumented window + 4-day soak; per-node logs and Prometheus/Loki history retained on the hosts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions