fix(paloalto): align PAN-OS parsing and correlation contracts

[kryonsx]

PAN-OS SYSTEM records arriving in an RFC3164 envelope were indexed without the standard authentication fields, and the deployed filter emitted errors for two empty steps. The source also used incorrect CSV properties/offsets and inconsistent field/value mappings. This draft updates the Palo Alto filter and its consuming rules together. It was reviewed with the SDK v1.1.33 pinned by the v11 base of that time and re-validated with go-sdk v1.1.36, which v11 now pins.

Replacement for closed #2614. This extends the initial configuration-only correction with raw extraction tests, SDK predicate checks and history tests requested in the previous review.

Changes

• Parse native CSV by its actual type column and support the documented envelope/layouts. Repair CSV configuration and field positions, including GTP and the differing URL/Data Filtering versus generic THREAT tail order. Preserve ambiguous legacy tails as vendor data.
• Parse tested seven-field CEF layouts based on the PAN-OS 10.0 guide, retaining case-sensitive vendor keys and final extension values. Keep reporter, session, NAT and forwarded addresses distinct; validate addresses, ports, counters, hashes and endpoint fields before standard promotion.
• Normalize documented actions/severities without treating alert-only or THREAT allow as a successful connection. Recover only explicitly identified authentication users/source IPs. Prevent CONFIG placeholder hostnames and retain scheme-less requests outside the SDK's full-URL field.
• Align seven existing detections and four direction-specific counterparts with the produced fields. Preserve event endpoint direction while assigning alert adversary/target correctly. Remove unsupported intelligence-feed, execution, zero-day and blanket C2 claims.
• Scope five histories to recomputed candidates and relevant collector/firewall/virtual-system identities. Preserve thresholds and impacts, and clear input-supplied candidate markers.
• Write every field name in camelCase with letters and digits only (for example log.paType). go-sdk up to v1.1.34 removed underscores from the names the parser plugins write; v1.1.35 and later keep them. camelCase names are stored unchanged by both, so conditions and rules find them.

See the source audit for exact SDK/vendor references, producer/consumer behavior and rollout limits.

Validation

• 125 fabricated raw fixtures cover normalization, negative controls and every one of the eleven rule predicates. Final Event decoding is strict; alert-side and required-placeholder expectations are checked separately.
• Read-only sampling obtained 34 distinct records plus mounted filter/rule configurations from two deployments. Twenty-two were native SYSTEM records; twelve were unrelated Cortex XDR records and remain negative controls. All sampled stored records carried the two empty-step errors. Private records and identifying metadata are not committed.
• The actual SDK evaluates all seven deployed predicates over those stored records with zero matches and zero CEL errors. Extraction of the same raw records by the corrected filter, in the offline model and in the real 8a3ade7 playground, followed by the revised predicates identifies all nine authentication failures; the three successes, ten other SYSTEM records and twelve Cortex records remain negative.
• Five actual SDK history-request tests use loopback mocks to verify thresholds, time windows, required identities and unrelated-event exclusion. Broader native classes and CEF are documentation/fixture validated; the SYSTEM sample does not establish their deployed behavior.

Re-validated on the latest versions (2026-09-24): official v11 d2479c1a (merged into this branch, no conflicts; plugins/alerts now pins go-sdk v1.1.36) and EventProcessor main 8a3ade7, the same source revision as the newest published engine image (eventprocessor:v11.2.14), compiled natively here. The camelCase names and every regexMatch (all on text fields) behave the same on v1.1.36.

What was wrong in the earlier version, and fixed here (filter 3.1.1). Run through the real 8a3ade7 parser plugins, most committed lines did not parse, although the Go suite passed; the older engine gave the same result, so the SDK update did not cause it:

• The CSV layouts list every documented column (130 for TRAFFIC), and the csv plugin fails the whole step when a line has fewer columns. Older firmware sends shorter lines, and so did 75 of the 122 committed lines; those events kept only the envelope. Each layout is now applied in tiers (for example TRAFFIC 103, then 130), each longer tier only when the line has that many columns, counted as the csv reader counts them.
• The 613 CEF/LEEF extension steps read each value with a lazy pattern and checked its end with a separate pattern. grok matches each pattern on its own, so the value matched empty text and nothing was stored. Each value is now read with its boundary and trimmed once.
• The SYSTEM authentication address pattern also took the final period, so user and address were never set: on the real engine the 9 real failed logins matched the authentication rule 0 times. The pattern now ends at the last address character.
• Lines without <PRI> were never parsed (an optional first pattern matched empty text); they now have their own step.

The offline model hid this: it joined each grok step's patterns into one expression and let csv skip missing columns. It now follows the 8a3ade7 plugins (pattern-by-pattern grok, csv failing on short lines, errors recorded like the engine). Against the previous filter it fails 104 of the 122 committed lines and 12 of the 34 real records, and its events equal the real engine's on all 122 lines. Two new tests pin this down. No fixture or expectation changed.

Empty CEF header values (filter 3.1.2). A check of every grok pattern on its own, the way the 8a3ade7 grok plugin runs it, found three more patterns that match empty text: the CEF/LEEF header's sender software version, signature and severity. When one of those header values is empty (for example |PAN-OS||auth|), the whole header step failed and the event kept only the envelope. Each value is now read together with the | that ends it, and one trim step removes that |. An empty value is stored as empty text, as the csv plugin stores empty columns, and the later steps already skip an empty signature. On the real engine, three new fixtures (cef-system-auth with one empty header value each) had no parsed field with 3.1.1 and have every expected field and the authentication rule match with 3.1.2. A new test tries every pattern alone on texts that start with each printable character and fails on the old filter.

Now, on the real engine (8a3ade7 playground, go-sdk v1.1.36, filter 3.1.2):

• The 125 committed lines (122 plus the three header fixtures): 125 events, no parser error, every expected field and absence on all 125 (before 3.1.1: 75 of 122 with the csv error, 16 complete). The Go model gives identical events on all 125.
• go-sdk v1.1.36 rule replay over those events: exactly the 41 expected rule matches, no other, every history placeholder resolved (before 3.1.1: 2 of the first 38).
• The 34 private real records: all parse with every expected field, with events identical to filter 3.1.1; the authentication rule matches exactly the 9 real failed logins and the other 25 records stay negative (before 3.1.1: 0 of 9).
• The alerts-module suite passes: 49 tests pass, 12 skip (private evidence), 0 fail; with the 34 private records 50 pass, 11 skip, 0 fail (2,424 tests/subtests). Run go test -count=1 ./... in plugins/alerts; optionally provide PALOALTO_PRIVATE_FIXTURES for the private raw contract suite.
• The playground has no OpenSearch, so history searches cannot complete there; the two history rules matching five or more lines raise a Circuit Breaker in these local runs for that reason. History queries are covered by the SDK tests against loopback mocks.

Limits

Raw extraction in the Go suite uses an offline model of the 8a3ade7 parser plugins, checked against the real playground on all 125 committed lines. SDK predicate/mock-history success does not prove production alert creation, throughput or alert volume. No customer system was modified. Exercise the parser with real export profiles and observe alerts before rollout.

Ambiguous CEF byte orientation and numeric-severity mapping remain vendor data. Standard/custom LEEF and arbitrary CEF profiles remain unverified. Cortex routing requires a separate source review. New candidate fields need up to thirty minutes of fresh history; install the filter/rules together and reconcile renamed installed definitions and custom consumers. The shared grouping fix #2627 is already in the reviewed v11 base; actual grouping behavior still needs rollout validation.

On updated head 21bb6201, CI run 35867008451 fails its architecture, bugs and security AI-review jobs before review: each log reports THREATWINDS_API_KEY is required at ai-review.sh:29. Prompt discovery passed; the Go-dependency job was still running at the last check. This missing CI credential is separate from the passing local regression suite and is not reviewer approval.

Leave this PR as a draft for team review. Nothing is merged or deployed by this change.

🤖 Generated with Claude Code

[kryonsx]

Change pushed (dd27214a): field names the parser actually keeps

What changed

• filters/paloalto/pa_firewall.yml: 239 field names that contained _ or other special characters are renamed to camelCase, in 1,147 places. Examples: log.pa_type → log.paType, log.pa_subtype → log.paSubtype, log.thr_category → log.thrCategory. Two steps that only copied log.pa_type into log.paType are removed, because they would now copy a field onto itself.
• All 11 rules under rules/paloalto/pa_firewall/ now read log.paType, log.paSubtype, log.thrCategory and log.paThreatid.
• The Go test model (paloalto_contract_test.go) now cleans every name that grok, add, rename and csv write, and json top-level keys, exactly like the real plugins. Run against the previous files, it fails 92 fixture cases, so it now catches this kind of defect.

Why

• The EventProcessor parser keeps only letters, digits and dots in the field names it writes (utils.SanitizeField), so log.pa_type was stored as log.patype.
• Every filter condition and rule that tested the underscored name never matched. On real logs this meant the 70 type-specific filter steps never ran (outcome, addresses, URL and configuration handling, the brute-force marker), and none of the 11 rules could fire.
• Until the EventProcessor handles these characters, filters and rules should use only letters, digits and dots in field names.

Validation (local EventProcessor playground, commit 497bf53, rule engine library v1.1.34)

• 46 real Palo Alto records from 2 production instances, this draft before and after the change:
  • All records parse without errors.
  • The 31 recognized SYSTEM records now carry log.paType and log.paSubtype, and action and target.host are filled.
  • 16 real login failures now get actionResult failure and 3 successes get success; before, none did.
  • No rule matches the real sample either way. The login failures carry no source address ("User is not in allowlist"), which the brute-force rule requires, and the sample has no THREAT records.
• On this draft's 122 fixtures, rule matches go from 0 to 2.
• go test ./... in plugins/alerts passes, and a field-name audit finds no names with special characters left.

Still blocking most detections (not fixed here; the old test model hid them)

1. The real csv step fails when a line has fewer columns than the header list, and the engine then drops everything that step wrote. 75 of the 122 fixtures hit this, and so would firewalls on older PAN-OS versions that send fewer columns.
2. The real grok step matches one pattern at a time from the start of the remaining text and stops at an empty match. Lines without a <priority> prefix get no type (22 fixtures), and CEF key=value extensions are never extracted (20 of 21 CEF fixtures).
3. 15 of the 46 real records, all from one instance, use a field order that no layout recognizes.
   The fixtures expect 38 rule matches; the real parser currently gives 2. These need a parser redesign checked against the real plugins, not the Go model.