Conversation
Filter 2.0.1. go-sdk v1.1.36 defines a grok pattern with the keys
fieldName and pattern only (plugins/plugins.proto), and the engine
loads every pipeline file with protojson DiscardUnknown
(plugins/config.go). The field_name key was therefore dropped without
any error or log line: the grok step wrote nothing, and the json step
failed on every generic event with "source was not found". No generic
event kept log.message or any key parsed from its text.
- Line 9: field_name becomes fieldName, so grok again copies the
trimmed first line of raw into log.message.
- Line 14: the json step runs only when log.message starts with "{".
The filter's own comment names syslog as expected input; without
the guard, every line that is not a JSON object would be stored with
a JSON parse error. The guard reads the grok output, which is
already trimmed, and changes nothing but that error. It must ship
with the key fix: on the old key it would only hide the failure.
- Line 1: version 2.0 becomes 2.0.1.
The only rule for this data type,
rules/generic/generic/cross_source_lateral_movement.yml, is not
changed. It requires top-level origin.ip and origin.user, which no
generic producer writes, so it cannot match a generic event before or
after this fix. filters/syslog/syslog-generic.yml has the same key and
belongs to the syslog review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
generic_filter_test.go checks filters/generic/generic.yml with go-sdk v1.1.36, the version plugins/alerts pins: - a strict protojson decode finds no key the SDK does not define; - plugins.GetCfg, the loader the engine calls, keeps the grok target log.message and the pattern (.*). It runs in a child process with WORK_DIR set to a private temporary folder, and the filter is staged where the config plugin writes it (pipeline/filters/<id>.yaml); - the json step's where clause, taken from that loaded definition and evaluated with the SDK CEL cache the engine uses for step conditions, is true for a JSON object (also after leading spaces, which grok trims) and for text that starts with a brace; false for syslog text, a syslog header before JSON, key=value text, a JSON array and a number; and false, without an error, when grok wrote no log.message. All three checks fail against the original file: the strict decode reports unknown field "field_name", the loaded pattern has an empty fieldName, and the json step has no where clause. A guard on raw, an exists guard, and the guard without the key fix each fail the matching check. Every event is fabricated and uses RFC 5737 addresses. Full plugins/alerts suite: 44 pass, 11 skip, 0 fail, 2205 subtests pass. The untouched branch gives 43 pass, 11 skip, 0 fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Evidence basis (no generic data on any searched instance; production runs the unfixed file byte for byte; how the SDK loader drops the key, from source and by running it), the three filter changes and their proof, why the rule stays unchanged and why retiring or re-scoping it is a maintainer decision, the playground, Go test and SDK replay results, the deferred items and the known limits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The generic filter never parses anything. Its grok pattern uses the key
field_name, but go-sdk'sPatternmessage only hasfieldName. The engine loads filter definitions withprotojson.UnmarshalOptions{DiscardUnknown: true}(go-sdkplugins/config.go), so the key is silently dropped. The grok step then writes nothing, and the json step fails with "source was not found". Every generic event is stored with only its raw line and one error.This draft renames the key and guards the json step so that plain text lines no longer add an error.
Built and tested on the latest versions: v11
d2479c1a, go-sdk v1.1.36, and EventProcessor8a3ade7. That engine revision is the same source as today'seventprocessor:v11.2.14image andeventprocessor/base:1.1.7.Predecessor: #2605 (closed, historical context only).
Evidence
genericrecords. Two instances run the deployed filter, and it is byte-identical to this file, includingfield_name.plugins.GetCfg) shows the behavior:field_name;fieldName;fieldNamekeeps it.fieldName, the grok plugin writes nothing.Change (
filters/generic/generic.yml, 2.0 → 2.0.1)field_name: log.message→fieldName: log.message.where: 'startsWith("log.message", "{")'.Validation
log.message, 29 with an error.log.message(all non-blank inputs), 12 with parsed keys (every one-line JSON object), and 2 errors (text starting with{that is not JSON, and pretty-printed multi-line JSON).plugins/alerts/generic_filter_test.go:fieldName.plugins/alertssuite passes.Rule
rules/generic/generic/cross_source_lateral_movement.ymlis unchanged. It requires top-levelorigin.ipandorigin.user, which no producer writes for generic input: the json step puts every sender key underlog.. So the rule cannot fire on generic input, with or without this fix. Retiring or re-scoping it is a maintainer decision, recorded in the audit.Deferred
filters/syslog/syslog-generic.yml:9has the samefield_namekey. It will get its own review.See
filters/audits/generic.md.🤖 Generated with Claude Code