Skip to content

beholder: log partial delivery per-event errors with error_code and reason#2210

Draft
pkcll wants to merge 3 commits into
mainfrom
chipingress-partial-delivery-cl-nodes
Draft

beholder: log partial delivery per-event errors with error_code and reason#2210
pkcll wants to merge 3 commits into
mainfrom
chipingress-partial-delivery-cl-nodes

Conversation

@pkcll

@pkcll pkcll commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Type-assert the send callback error to `*batch.PublishError` so partial delivery failures (individual events rejected server-side while the batch RPC itself succeeded) are logged at WARN with structured `error_code` and `reason` fields.
  • Adds `error_code` label to the `chip_ingress.events_dropped` OTel counter for `partial_delivery` failures so dashboards and alerts can track rejection types without relying on logs (e.g. `PUBLISH_ERROR_CODE_VALIDATION_FAILED`).
  • Adds `failure_type` attribute (`partial_delivery` vs `rpc_error`) to the same counter to distinguish per-event server rejections from whole-batch RPC failures.
  • Whole-batch RPC failures keep the existing ERROR log and increment the counter without `error_code` (no structured server code available).

Jira: INFOPLAT-13901 | Epic: INFOPLAT-7177

Must merge before the chainlink PR that bumps this dep.

Changes

  • `pkg/beholder/batch_emitter_service.go`:
    • Type-assert `*batch.PublishError`; WARN log with `error_code`, `reason`, `domain`, `entity`
    • `dropMetricAttrsFor` now accepts `errorCode` — included for `partial_delivery`, omitted for `rpc_error`
  • `pkg/beholder/batch_emitter_service_test.go`:
    • `TestChipIngressBatchEmitterService_PartialDeliveryError` — covers log fields (`error_code`, `reason`, `domain`, `entity`) and metric label (`failure_type=partial_delivery`, `error_code=PUBLISH_ERROR_CODE_VALIDATION_FAILED`)

Test plan

  • `go test ./pkg/beholder/... -count=1` passes
  • Partial delivery log entry is WARN with `error_code` and `reason` fields
  • `chip_ingress.events_dropped{failure_type="partial_delivery", error_code="PUBLISH_ERROR_CODE_VALIDATION_FAILED"}` increments for per-event errors
  • `chip_ingress.events_dropped{failure_type="rpc_error"}` increments for whole-batch failures (no `error_code` label)

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

📊 API Diff Results

No changes detected for module github.com/smartcontractkit/chainlink-common

View full report

@pkcll pkcll force-pushed the chipingress-partial-delivery-cl-nodes branch 3 times, most recently from bcb946f to 59011ac Compare July 2, 2026 22:18
…code label

- Type-assert send callback error to *batch.PublishError; log at WARN
  with error_code, reason, domain, entity for per-event server rejections.
- Add error_code label to chip_ingress.events_dropped metric so dashboards
  and alerts can distinguish rejection types without relying on logs.
- Whole-batch RPC failures keep the existing ERROR log; their drop metric
  omits error_code (no structured server code available).
- Also adds failure_type label (partial_delivery / rpc_error) to the
  dropped counter to distinguish the two failure modes.

Jira: INFOPLAT-13901
@pkcll pkcll force-pushed the chipingress-partial-delivery-cl-nodes branch from 59011ac to d16297c Compare July 3, 2026 20:41
pkcll added 2 commits July 13, 2026 16:02
- Add batch.ClassifyDropFailure for partial_delivery, rpc_error,
  buffer_full, and client_error paths with shared metric dimensions.
- Replace failure_type/grpc_code labels with error_type and error_code
  (PUBLISH_ERROR_CODE_* or gRPC code name) plus error_reason.
- Add ErrMessageBufferFull and ErrClientShutdown sentinels for queue drops.

Jira: INFOPLAT-13901
@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

✅ API Diff Results - github.com/smartcontractkit/chainlink-common/pkg/chipingress

✅ Compatible Changes (8)

batch (8)
  • ClassifyDropFailure — ➕ Added

  • DropFailure — ➕ Added

  • ErrClientShutdown — ➕ Added

  • ErrMessageBufferFull — ➕ Added

  • ErrorTypeBufferFull — ➕ Added

  • ErrorTypeClientError — ➕ Added

  • ErrorTypePartialDelivery — ➕ Added

  • ErrorTypeRPCError — ➕ Added


📄 View full apidiff report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant