OBE-10327: rejection reports for splunk hec / in general#114
Open
akshayakumar-t wants to merge 8 commits into
Open
OBE-10327: rejection reports for splunk hec / in general#114akshayakumar-t wants to merge 8 commits into
akshayakumar-t wants to merge 8 commits into
Conversation
…ice rejection paths Co-Authored-By: Akshaya's Agent <akshaya.kumar+agent@sentinelone.com>
Co-Authored-By: Akshaya's Agent <akshaya.kumar+agent@sentinelone.com>
Sinks no longer emit a category field in rejection logs — vector's own structured logging already identifies the component. The trait method is removed so implementations are not forced to provide one. Co-Authored-By: Akshaya's Agent <akshaya.kumar+agent@sentinelone.com>
- Inline rejected counter directly into HecRejectionContext, removing the Telemetry wrapper struct that collided with the ES Telemetry name - Parse Splunk JSON error body in error_message to surface the text field - Remove #OBSERVO_STYLE_TELEMETRY# cross-reference comment Co-Authored-By: Akshaya's Agent <akshaya.kumar+agent@sentinelone.com>
…itialisers Co-Authored-By: Akshaya's Agent <akshaya.kumar+agent@sentinelone.com>
Co-Authored-By: Akshaya's Agent <akshaya.kumar+agent@sentinelone.com>
- Use typed SplunkErrorBody struct instead of serde_json::Value to parse the Splunk error body — avoids full allocation for one field - Replace &*context with context.as_ref() for consistency - Add unit tests for HecRejectionContext::error_message covering the text-field-present and fallback paths Co-Authored-By: Akshaya's Agent <akshaya.kumar+agent@sentinelone.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extracts rejection-report infrastructure from the Elasticsearch sink into a shared
src/sinks/util/rejection_report.rsmodule, then ports the same capability to bothSplunk HEC sinks (logs + metrics).
Before: Only the ES sink logged details about rejected batches. HEC silently dropped
rejections with no structured log output and no counter.
After: Both ES and HEC emit a structured
error!log on rejection/error, incrementa
hec_rejected{endpoint=...}counter, and expose arejection_reportconfig fieldthat controls how much detail is logged.
What changed
New:
src/sinks/util/rejection_report.rsRejectionReportenum (moved fromelasticsearch/mod.rs, re-exported there viapub useso existing configs and imports are unaffected):stats(default) — increment counters only, no bodies loggedresponse— also log the HTTP response bodyrequest_response— log both the request payload and response body; use with smallbatch sizes during debugging only, as event payloads may contain sensitive data
RejectionContexttrait — sink-specific plug-in points:error_code,error_message,record_rejection. Each sink implements this to wire in its owncounters and response parsing without touching the shared logging logic.
emit_rejection_error— generic free function that handles all threeRejectionReportbranches, decompresses the request body when needed, and emits astructured
error!log.Elasticsearch (
elasticsearch/service.rs)ElasticsearchRejectionContextimplementingRejectionContext; holds theexisting
Telemetry{rejected, indexed}counters.emit_bad_response_errorremoved;get_event_statusnow delegates toemit_rejection_error.err_summarysignature changed from(&Response<Bytes>)to(u16, &Bytes),decoupling it from the HTTP response type.
Splunk HEC (
splunk_hec/common/service.rs)HecRejectionContextimplementingRejectionContext; holds a singlerejected: Counter. Parses Splunk's JSON error body ({"text":"..."}) to surfacea human-readable message; falls back to the status code.
HecService<S>gains three new fields:rej_rpt,compression,context.call()clones the request body before sending (only whenneeds_request()is true,i.e.
RequestResponsemode) and callsemit_rejection_erroron 4xx/5xx.RequestResponse→Response(request payload is not usefulfor server-side failures).
ResponseExttrait extended withstatus_code() -> u16so the generic service canexpose the HTTP status without depending on
http::Response<Bytes>directly.Config (
logs/config.rs,metrics/config.rs)Both HEC sink configs gain: