Conversation
Support story
Add nft header
support ZSTD compression
generate the python code from proto
transform Hash and Account Address into Tronscan format
add test case for Tron address convertion;
Fix/add tron fields
INF-59 Integrate with CircleCI
Merged latest upstream changes from Coinbase
Enable support for launching multiple backfillers concurrently
Collaborator
🟡 Heimdall Review Status
|
Resolve CircleCI build failures
use python grpcio-tools to generate python code
The hourly retention probe has been failing with "canceling statement due to statement timeout" since 04:00Z, so the cron cannot select any cohort and retention is stalled. The due query filters storage generation, but wrote it as `col IS NOT DISTINCT FROM NULLIF($n, '')`, and that form cannot use a btree index at all. Verified against production: height = 439200000 -> Index Only Scan height IS NOT DISTINCT FROM 439200000 -> Parallel Seq Scan So every probe examined all ~20.9M legacy-generation shadow rows to find ~768k v2 rows. Adding a generation column to an index would have been inert while the query kept that operator, which is why this changes the query and the schema together. The generation is known when the query is built, so render it as plain equality for a concrete generation and IS NULL for the legacy one (stored as NULL). Both are seekable. The generation argument is bound only when the rendered SQL references it, so the placeholder and argument lists cannot drift. Add idx_block_consolidation_shadow_retention_due_generation leading with (tag, single_block_storage_generation, ...) so the probe seeks straight to the active write generation instead of walking superseded ones. The existing due index stays: it still serves probes that do not constrain generation, and removing it belongs in a follow-up once production plans confirm the new one is chosen. Equivalence checked against production before shipping: over the current probe window both forms return the same 146 cohorts, the same first key and the same 145,554 rows, and both return zero for the legacy generation. Tests pin the rendered predicates, fail if IS NOT DISTINCT FROM returns to the selector, assert the index columns match what the probe filters, and check that each generation mode binds exactly the placeholders its SQL references. Claude-Session: https://claude.ai/code/session_012Ge3KEUU82pkYvQ1GC1Chi Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Endpoint, EndpointGroup, EndpointConfig and StickySessionConfig only
carried json tags, which cover the path where a group arrives as a JSON
string (the CHAINSTORAGE_..._ENDPOINT_GROUP env var, decoded by
EndpointGroup.UnmarshalText). A group written as a YAML mapping is
decoded by mapstructure instead, which without a tag falls back to a
case-insensitive field-name match and therefore silently dropped every
key containing an underscore: rps_count_batch, provider_id, extra_urls,
endpoints_failover, use_failover and endpoint_config.
The visible symptom was an endpoint configured with
rps: 100
rps_count_batch: true
still tripping the provider's rate limit, because the limiter kept
charging one token per HTTP request instead of one per call inside a
JSON-RPC batch, i.e. up to rps * tx_batch_size provider-side calls per
second.
Add the mapstructure tags so both decoders agree, and log the endpoint
settings that are actually in effect at startup (host only, since
providers embed the API key in the url path), including a warning for an
empty group. Note the rate limiter is per endpoint per group, so the
same url configured in master, slave and validator gets an independent
budget in each.
TestEndpointGroupParsedFromYAML loads a YAML config end to end through
config.New to lock the behavior; TestDumpEndpoints is an opt-in
diagnostic that prints the effective endpoint settings for a given
environment.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rror (#209) The syncer logged the per-block fetch error and dropped it: reprocessChannel carried only the block metadata, so the aggregate error could name which blocks failed but not why. During the 2026-08-31 robinhood-mainnet incident that left Temporal with "failed to get 10 blocks after attempting twice: [<metadata>]" while the actual cause -- 25,200 HTTP 429s and a provider-side loss of the debug_* namespace -- existed only in the worker pod stdout, which sent the incident response after a block-data gap that was never the problem. - reprocessChannel now carries the {metadata, err} pair. - The aggregate error summarizes the distinct causes, deduplicated by error class and endpoint so the message stays bounded when many blocks fail for one reason, e.g. "10/10 RateLimitError (HTTPError 429) on endpoint=nownodes-jsonrpc-slave; heights=[50881296..50881305]". - The jsonrpc client counts every throttled attempt as jsonrpc.rate_limited+endpoint=<name>, including the ones absorbed by a retry, so an endpoint approaching its quota alerts while the poller is merely slow rather than after it fails. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t in bounded chunks (#210)
Move ownership of the chainstorage image into this repo so the monorepo can consume a prebuilt, sha-pinned tag instead of git-cloning and rebuilding on every deploy. Dockerfile: adopt the monorepo's semantics minus the git clone. - ARG BASE_REGISTRY so CI pulls base images through the ECR docker-hub pull-through cache; local builds keep using Docker Hub. - Build on golang:1.26-trixie with CGO_ENABLED=0 (static binaries), with a separate go mod download layer. - Deploy on ubuntu:22.04 with ca-certificates; ship /app/bin/* plus the postgres migrations at /app/migrations; CMD /app/worker. - Add .dockerignore so .git, bin/, local compose data and secrets never enter the build context. CircleCI: keep build_and_test untouched and add - orb circleci/aws-cli@5.4.2 - command build_image: OIDC assume-role via CIRCLECI_OIDC_ARN, ECR login, docker build --platform linux/amd64 tagged with the 7-char short sha, no_output_timeout 30m (the Go build is silent for minutes). - job build-image: every branch, build only, no push. - job publish-image: master only, requires build_and_test, pushes the single sha tag and prints the pushed ref. No latest/master tags. Auth needs no new secrets: cipherowl_build_context already carries CIRCLECI_OIDC_ARN, the circleci_oidc role trust has no project subject condition, and the ECR repo policy grants that role push. Claude-Session: https://claude.ai/code/session_01RvJf1meAams4fNAFtpyLB1 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
PersistBlockMetas issued two statements per block inside one transaction, about 5,000 sequential round trips per 2,500-block backfill batch. It now persists in height-ordered chunks of 1,000 blocks with UNNEST-based upserts: one statement per group (regular, skipped) plus one canonical upsert per chunk, so a 2,500-block batch takes 9 statements. The ON CONFLICT clauses and placement guards are reused verbatim, ids are mapped back by conflict key, and same-key duplicates are collapsed Go-side because Postgres rejects them inside one statement. The sort is now stable so "last block wins" honors caller order. Local docker-compose benchmark, 2,500 blocks: 4.0s -> 62ms fresh, 4.2s -> 78ms replay. Adds a legacy-vs-batched benchmark, a toxiproxy compose overlay and a benchmark-postgres make target. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed? Why?
2.1 Adding LTC / Tron Support,
2.2 Adding ZSTD support
2.3 Making ChainStorage more friendly for opensource / k8s environment
How did you test the change?