Skip to content

fix(cisco-switch): map real message addresses and stop MAC rule circuit breakers - #2689

Merged
osmontero merged 8 commits into
utmstack:v11from
kryonsx:codex/v11-cisco-switch-review-20260924
Sep 25, 2026
Merged

osmontero merged 8 commits into
utmstack:v11from
kryonsx:codex/v11-cisco-switch-review-20260924

Conversation

@kryonsx

@kryonsx kryonsx commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Real Cisco switch logs show that the MAC spoofing rule keeps disabling itself in production. The filter never writes the MAC address that the rule's history search needs. This draft maps the addresses that real switch messages carry. It also keeps MAC flap notifications out of the spoofing rule, guards the ARP rule's history search, and fixes a raw CEL condition that errors on every non-switch line.

Predecessor: #2599 (closed, historical context only).

Evidence (read-only, described without identifying data)

  • Two instances send real Cisco IOS/IOS XE switch syslog:
    • about 226,000 records from 5 switches, and about 2,600 from one switch;
    • 79% are %SW_MATM-4-MACFLAP_NOTIF;
    • the rest are LINK/LINEPROTO, SISF, DHCPD, SSH, SYS, PKI, IP and wireless-controller messages.
  • A third instance sends about 2.36M non-switch lines (Firepower Threat Defense events and Firepower Management Center system lines) to this data type.
  • The deployed filter and rules (v11.2.13) match this repository.
  • Production alert indices contain 12 Circuit Breaker: MAC Address Spoofing Detection alerts (2026-08-19 to 09-21) and no real MAC spoofing alert.
    • The error: expression value cannot be nil after placeholder resolution.
    • The events that triggered it: MAC flap notifications.
  • Cisco's documentation site refused automated access (HTTP 403). Changes rest on real records, the filter, EventProcessor (commit 497bf53; the latest 8a3ade7 differs only in SDK versions and the CEL plugin's OpenSearch settings) and go-sdk behavior (unchanged for these points in v1.1.36).

Filter (filters/cisco/cs_switch.yml, 3.0.2 → 3.1.0)

  • Condition fix: where: log.severity=="4" → equals("log.severity", "4"). The raw form fails to compile on events without a log object ("undeclared reference to 'log'"). It stored that error on about 1.12M non-switch records. Severity results are unchanged.
  • MAC flap notifications: write origin.mac, log.vlan, log.firstPort and log.secondPort. Interface names stay under log.* because origin.port is numeric. The names do not claim which port is the previous one.
  • Other address mappings, taken from the fixed text shapes seen in real records:
    • SISF excessive ARP activity: client MAC → origin.mac;
    • SSH session close and unexpected message: client address → origin.ip;
    • DHCP ping conflict: pinged address → target.ip;
    • logging-host start/fail: target.ip and a numeric target.port.
  • Outcome values and severity words are unchanged; a separate outcome review owns them.

Rules

  • mac_address_spoofing.yml:
    • requires origin.mac;
    • excludes flap notifications and corrects its description;
    • reads log.msg, which the filter writes, instead of log.message, which nothing writes;
    • uses deduplicateBy: [adversary.mac].
  • arp_poisoning_detection.yml: requires origin.ip before its history search and reads log.msg.
  • vlan_hopping_attempts.yml is unchanged; see the deferred items.

Why flaps are excluded (owner decision the reviewer can revisit)

Nothing establishes that a flap means an address was copied. In the real data, flaps look like network paths:

  • one address alternates between the same two ports about every 15 seconds for 31 days;
  • on the other instance, five port pairs carry 93% of flaps with 73–116 different addresses each.

Estimated 30-day alert volumes from real timestamps (not observed alerts):

Option Instance A Instance B
Flaps in, deduplicated, 3 in 10 min 5 (from ~175,000 firings) 115–119
Flaps in, deduplicated, 10 in 10 min 5 9
Flap storms only (50–100 in 10 min) 2–5 0
Flaps excluded (this draft) 0 0

Validation

Re-checked on 2026-09-24 on the latest versions: official v11 d2479c1a (merged into this branch, no conflicts; plugins/alerts now pins go-sdk v1.1.36) and EventProcessor main 8a3ade7 (playground and every plugin on go-sdk v1.1.36).

  • SDK change: go-sdk v1.1.35+ keeps underscores in field names the parser plugins write, and v1.1.36 makes regexMatch match strings only. Neither changes this draft: none of the 25 names the filter writes contains an underscore (SW_MATM and similar are values, not names), and every regexMatch/contains call reads a text field. No file needed a change.
  • Isolated EventProcessor playground (latest, real parser):
    • 398 inputs: 327 real, kept private, and 71 fabricated. Both runs are identical, event by event, to the results recorded on the older build.
    • The unchanged filter reproduces all stored records.
    • The changed filter gives 0 errors (from 25), unchanged severity and only intended new fields.
    • All real flap, SISF, SSH, DHCP and logging-host records get their fields, and no fabricated near-miss gets one.
    • The 2,795 distinct real texts and 28 fabricated lines give the same output as on the older build, line for line.
    • Committed plugins/alerts/testdata/cisco-switch/replay.py: 44 events, 0 errors, every field as in expected.json, 6 VLAN alerts, no circuit breaker, no history search.
  • go-sdk v1.1.36 replay: the changed MAC and ARP rules match 0 of 229,017 real records, with no unresolved placeholder. The original MAC rule matches all 182,326 flap records and, with the deployed filter, cannot resolve origin.mac; with this filter's new mapping it would instead run a history search on every flap, which is why flaps are excluded. 15/15 synthetic checks pass.
  • Playground rule run with fabricated lines (latest):
    • Changed rules: 0 MAC, 0 ARP and 0 circuit-breaker alerts, and VLAN alerts only on the six VLAN/DTP lines (14/14 checks).
    • Original rules: they reproduce the circuit breaker.
  • Go tests: the full plugins/alerts suite passes on go-sdk v1.1.36: 51 pass, 11 skip (other technologies' private-evidence tests, same as the base), 0 fail. The eight new Cisco Switch tests all fail against the original files.

Deferred

These need Cisco's documentation or an owner decision:

  • whether flaps should alert at all (as a separate network-health rule);
  • which flap port is the previous one;
  • address mappings for SW_DAI, IP DUPADDR and SOURCEGUARD;
  • the VLAN rule's text branches, which have no threshold or guard;
  • new rules for excessive ARP activity, logging-host failures, DHCP conflicts and PKI failures;
  • access-list fields and device time conversion;
  • severity words and outcome values.

For the owner, not a filter change

On one instance, Firepower and Firepower Management Center devices send their logs to the Cisco Switch input. Those events never reach the Firepower filter or rules. The fix is to point those devices at the Firepower input.

Limits

  • The playground (EventProcessor 8a3ade7) and this module both use go-sdk v1.1.36; neither is claimed to match a customer deployment.
  • History searches, deduplication, indexing and notifications were not tested against OpenSearch.
  • Alert volumes are estimates from real timestamps.

See filters/audits/cisco-switch.md.

🤖 Generated with Claude Code

kryonsx and others added 8 commits September 24, 2026 12:06
Names, impact, category, technique, adversary side, references,
thresholds and windows are unchanged. The VLAN hopping rule is
unchanged.

- mac_address_spoofing (v1.0.1): require origin.mac, leave out MAC flap
  notifications (SW_MATM-4-MACFLAP_NOTIF), read log.msg instead of
  log.message, and deduplicate by adversary.mac instead of grouping.
  The filter never wrote origin.mac, so the {{.origin.mac}} history
  placeholder failed on every flap, and after five failures the CEL
  plugin disabled the rule with a 'Circuit Breaker' alert; the rule has
  never produced a detection. The filter change that follows maps
  origin.mac on every flap. Nothing shows that a flap means an address
  was copied, and without this change every flap would run a history
  search. The description now says that flaps are not used.
- arp_poisoning_detection (v1.0.1): require origin.ip before the rule
  can match, and read log.msg instead of log.message. No step writes
  origin.ip for SW_DAI, IP DUPADDR/SOURCEGUARD or the text branches, so
  any match would fail its {{.origin.ip}} history search the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Filter 3.1.0. Every change rests on real switch records, the filter's
own patterns and the EventProcessor and go-sdk behaviour. Cisco's
system message guide could not be read, so nothing that depends on
what a message means is changed.

- Line 206: write the 'medium' severity condition as
  equals("log.severity", "4") instead of log.severity=="4". A line
  without a %FACILITY-SEVERITY-MNEMONIC header has no log object or no
  log.severity, so the raw comparison failed and the engine stored the
  error on the event. The helper returns false instead; severity is
  unchanged on every tested line.
- SW_MATM-4-MACFLAP_NOTIF: write the flapping address to origin.mac, the
  VLAN to log.vlan, and the two interfaces, in the order the message
  gives them, to log.firstPort and log.secondPort. Interface names stay
  under log.*; origin.port holds numbers only.
- SISF-4-EXCESS_ARP_ACTIVITY: the client address to origin.mac.
- SSH-4-SSH2_UNEXPECTED_MSG and SSH-5-SSH_CLOSE: the client address to
  origin.ip.
- DHCPD-4-PING_CONFLICT: the pinged address to target.ip.
- SYS-3-LOGGINGHOST_FAIL and SYS-6-LOGGINGHOST_STARTSTOP: the logging
  host to target.ip and its port to target.port as a number.

Each new step runs only for its facility and mnemonic and writes nothing
unless the whole text shape matches. The actionResult steps and the
severity words are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cisco_switch_filter_test.go checks, with go-sdk v1.1.33: that no where
clause compares log.* directly, and that every clause evaluates without
an error on a draft without a log object, one without a severity and a
parsed one; that the severity steps keep their result on every level
from 0 to 7; that a model of the engine's step plugins reproduces the
playground result for every stored field of 44 fabricated lines, with
a positive and a near-miss line for each new mapping; that interface
names never reach origin.port or target.port; the unchanged names,
metadata, impact and history searches of the three rules, the MAC
rule's deduplication and the unchanged VLAN condition; 28 synthetic
rule cases; that the MAC and ARP rules match none of the fabricated
lines and the VLAN rule exactly the six SW_VLAN/DTP lines; and that a
rule with a history search never matches an event that lacks its
placeholder fields. All eight tests fail against the original filter
and rules.

testdata/cisco-switch/replay.py runs the same 44 lines through the
public EventProcessor playground with the filter, the three rules and
the shared grok definitions, and checks every field and alert. The
rules' OpenSearch address is a closed local port, so a history search
would fail and be reported. All inputs are invented: MAC addresses in
the locally administered 02:00:00:xx:xx:xx range in Cisco's dotted
form, RFC 5737 and RFC 3849 addresses, and example names. None is
taken from Cisco's documentation, which could not be read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Evidence basis (real switch records on two instances, given as counts
and header shapes only; the production circuit-breaker alerts of the MAC
rule; Cisco's documentation unavailable), the basis and proof of each
change, the MAC rule decision with the estimated volumes of each option
as an owner decision, the playground, SDK and Go test results, the
deferred items with what would unblock each, the customer-side routing
note and the known limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the playground run in which two contrived lines gave the committed
MAC and ARP rules a true condition: both rules reached their history
search with the value resolved, and both searches failed because no
OpenSearch was listening. State that no history search completed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in go-sdk v1.1.36 for plugins/alerts (field names keep underscores; regexMatch matches strings only) and the coordinated vendor re-baseline. No file overlaps this draft.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d engine

Official v11 d2479c1 (go-sdk v1.1.36 in plugins/alerts) and EventProcessor main 8a3ade7 (every playground plugin on v1.1.36). Field names now keep underscores and regexMatch matches strings only; neither changes this draft, because none of the 25 names the filter writes contains an underscore and every text-search call reads a string field. Full plugins/alerts suite: 51 pass, 11 skip, 0 fail. replay.py: 44 events, 6 VLAN alerts, no Circuit Breaker, no history search. The 398 private inputs and the 2,823 distinct texts give event-for-event the same output as the original review; rule runs 14 of 14; the original rules still trip the Circuit Breaker. v1.1.36 replay: committed MAC and ARP rules match none of 229,017 real records; 15 of 15 synthetic checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion build

The newest published engine image, eventprocessor:v11.2.14 (built 2026-09-24 19:13 UTC), carries EventProcessor revision 8a3ade7 with go-sdk v1.1.36, built with go1.26.8 for linux/amd64. The local build used for the re-validation is the same source compiled natively for darwin/arm64 with go1.25.7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@osmontero
osmontero marked this pull request as ready for review September 25, 2026 15:19
@osmontero
osmontero requested a review from a team September 25, 2026 15:19
@osmontero
osmontero merged commit 31c3b66 into utmstack:v11 Sep 25, 2026
3 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants