Skip to content

fix(generic): use fieldName so the generic filter actually parses input - #2749

Draft
kryonsx wants to merge 4 commits into
utmstack:v11from
kryonsx:codex/v11-generic-review-20260924
Draft

kryonsx wants to merge 4 commits into
utmstack:v11from
kryonsx:codex/v11-generic-review-20260924

Conversation

@kryonsx

@kryonsx kryonsx commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Summary

The generic filter never parses anything. Its grok pattern uses the key field_name, but go-sdk's Pattern message only has fieldName. The engine loads filter definitions with protojson.UnmarshalOptions{DiscardUnknown: true} (go-sdk plugins/config.go), so the key is silently dropped. The grok step then writes nothing, and the json step fails with "source was not found". Every generic event is stored with only its raw line and one error.

This draft renames the key and guards the json step so that plain text lines no longer add an error.

Built and tested on the latest versions: v11 d2479c1a, go-sdk v1.1.36, and EventProcessor 8a3ade7. That engine revision is the same source as today's eventprocessor:v11.2.14 image and eventprocessor/base:1.1.7.

Predecessor: #2605 (closed, historical context only).

Evidence

  • No reachable v11 instance holds generic records. Two instances run the deployed filter, and it is byte-identical to this file, including field_name.
  • A small program calling go-sdk's own loader (plugins.GetCfg) shows the behavior:
    • strict decoding rejects field_name;
    • the production path drops it with no error or log line, leaving an empty fieldName;
    • the same file spelled fieldName keeps it.
  • With an empty fieldName, the grok plugin writes nothing.

Change (filters/generic/generic.yml, 2.0 → 2.0.1)

  • field_name: log.message → fieldName: log.message.
  • json step: where: 'startsWith("log.message", "{")'.
    • Only a line that starts like a JSON object is parsed as JSON.
    • This is shipped together with the key fix. On its own, the guard would hide the defect.
    • Maintainers can drop it if they prefer the smallest change.

Validation

  • Playground (engine 8a3ade7), 29 fabricated inputs:
    • Unchanged filter: 0 events with log.message, 29 with an error.
    • Fixed filter: 27 with log.message (all non-blank inputs), 12 with parsed keys (every one-line JSON object), and 2 errors (text starting with { that is not JSON, and pretty-printed multi-line JSON).
    • Without the guard, 16 events would carry errors.
  • Go test plugins/alerts/generic_filter_test.go:
    • It loads the file with go-sdk's loader and checks fieldName.
    • It evaluates the guard with the SDK CEL evaluator.
    • All three checks fail on the original file.
    • The full plugins/alerts suite passes.

Rule

rules/generic/generic/cross_source_lateral_movement.yml is unchanged. It requires top-level origin.ip and origin.user, which no producer writes for generic input: the json step puts every sender key under log.. So the rule cannot fire on generic input, with or without this fix. Retiring or re-scoping it is a maintainer decision, recorded in the audit.

Deferred

  • filters/syslog/syslog-generic.yml:9 has the same field_name key. It will get its own review.
  • Multi-line input keeps only the first line.
  • Indexing when senders use different types for the same key.

See filters/audits/generic.md.

🤖 Generated with Claude Code

kryonsx and others added 4 commits September 24, 2026 20:47
Filter 2.0.1. go-sdk v1.1.36 defines a grok pattern with the keys
fieldName and pattern only (plugins/plugins.proto), and the engine
loads every pipeline file with protojson DiscardUnknown
(plugins/config.go). The field_name key was therefore dropped without
any error or log line: the grok step wrote nothing, and the json step
failed on every generic event with "source was not found". No generic
event kept log.message or any key parsed from its text.

- Line 9: field_name becomes fieldName, so grok again copies the
  trimmed first line of raw into log.message.
- Line 14: the json step runs only when log.message starts with "{".
  The filter's own comment names syslog as expected input; without
  the guard, every line that is not a JSON object would be stored with
  a JSON parse error. The guard reads the grok output, which is
  already trimmed, and changes nothing but that error. It must ship
  with the key fix: on the old key it would only hide the failure.
- Line 1: version 2.0 becomes 2.0.1.

The only rule for this data type,
rules/generic/generic/cross_source_lateral_movement.yml, is not
changed. It requires top-level origin.ip and origin.user, which no
generic producer writes, so it cannot match a generic event before or
after this fix. filters/syslog/syslog-generic.yml has the same key and
belongs to the syslog review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
generic_filter_test.go checks filters/generic/generic.yml with go-sdk
v1.1.36, the version plugins/alerts pins:

- a strict protojson decode finds no key the SDK does not define;
- plugins.GetCfg, the loader the engine calls, keeps the grok target
  log.message and the pattern (.*). It runs in a child process with
  WORK_DIR set to a private temporary folder, and the filter is staged
  where the config plugin writes it (pipeline/filters/<id>.yaml);
- the json step's where clause, taken from that loaded definition and
  evaluated with the SDK CEL cache the engine uses for step
  conditions, is true for a JSON object (also after leading spaces,
  which grok trims) and for text that starts with a brace; false for
  syslog text, a syslog header before JSON, key=value text, a JSON
  array and a number; and false, without an error, when grok wrote no
  log.message.

All three checks fail against the original file: the strict decode
reports unknown field "field_name", the loaded pattern has an empty
fieldName, and the json step has no where clause. A guard on raw, an
exists guard, and the guard without the key fix each fail the matching
check. Every event is fabricated and uses RFC 5737 addresses.

Full plugins/alerts suite: 44 pass, 11 skip, 0 fail, 2205 subtests
pass. The untouched branch gives 43 pass, 11 skip, 0 fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Evidence basis (no generic data on any searched instance; production
runs the unfixed file byte for byte; how the SDK loader drops the key,
from source and by running it), the three filter changes and their
proof, why the rule stays unchanged and why retiring or re-scoping it
is a maintainer decision, the playground, Go test and SDK replay
results, the deferred items and the known limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant