Skip to content

Map Azure action results to final operation outcomes - #2664

Draft
kryonsx wants to merge 2 commits into
utmstack:v11from
kryonsx:codex/data-engine-azure-20260923
Draft

kryonsx wants to merge 2 commits into
utmstack:v11from
kryonsx:codex/data-engine-azure-20260923

Conversation

@kryonsx

@kryonsx kryonsx commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Azure Event Hub currently can label an unfinished or contradictory operation as actionResult: success. For example, a sign-in with one zero error-code alias and another nonzero alias, or an Event Grid record whose event name ends in Success while its status is Started, can be treated as successful. Since the normalized result and IP feed threat-intelligence correlation, these cases can create false alerts.

This change derives a single final success, failure, or denied result from the relevant Azure event class. Denial and failure take precedence over success; pending, unknown, and conflicting signals leave the result unset. Sign-ins require consistent final codes, HTTP responses use explicit final status codes, Kubernetes audit events require ResponseComplete, and WAF actions distinguish requests passed to the backend from blocks and intermediate rule matches. The filter also extracts the needed scalar leaves from embedded HTTP, Kubernetes, and request-body JSON. It restores the authoritative outer event before normalization, so embedded fields cannot replace its category or outcome.

The mappings follow the Microsoft sign-in error-code reference, Azure asynchronous-operation statuses, HTTP status definitions, Application Gateway WAF log action definitions, and the standard event schema. The reviewed UTMStack snapshot is ab96ae9a685c54c974253ec517c688a7557019e9, with SDK v1.1.33 and schema wiki revision c18b54bd5ea5a34abb0e690458d73f89835edd29.

Validation: 69 fabricated raw regression cases pass in the isolated parser, with seven expected alerts from shipped rule controls; the original filter passed 32 of 69 and produced 17 alerts on the same controls. Another 185 existing Azure raw cases and the full plugins/alerts test module pass. The parser's writer can concatenate complete JSON objects without NDJSON separators; assertions strictly decoded the complete object stream and recorded the framing warnings separately. The isolated parser has a different SDK/plugin dependency set from the reviewed snapshot, so this is not proof of deployed equivalence. Ten bounded read-only production samples were from a different event class, and the sampled worker did not have the Azure filter loaded; deployment and live Azure outcome behavior remain unverified. No customer records are included in this PR.

The shared TI outcome consumer has a separately tested draft correction so it alerts only on confirmed successful operations. Both changes should be reviewed together before relying on normalized outcomes for TI alerts.

kryonsx and others added 2 commits September 23, 2026 11:24
The parser plugins remove every character except letters, digits and dots
from the field names they write. The Azure filter added its 10 correlation
markers with underscores (for example
log.correlationCandidate.key_vault_access_spikes), so they were stored
without them (log.correlationCandidate.keyvaultaccessspikes) and the 10
history rules that count them never matched.

Rename the markers to camelCase (for example
log.correlationCandidate.keyVaultAccessSpikes) in the filter and in every
rule that reads them. Rule conditions, thresholds and windows are unchanged.

Tests: the Azure parse model now stores grok, rename and add targets the
way the parser does, the history test counts each marker by its new name,
and a new test checks that every marker the filter adds and every marker an
Azure rule reads are the same name, made only of letters, digits and dots.
Fixtures that forge a marker in the input now use the new name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kryonsx

kryonsx commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

Change pushed (ace3e6a0): history markers renamed to names the parser keeps

What changed

  • filters/azure/azure-eventhub.yml: the 10 log.correlationCandidate.* markers are now camelCase. For example, key_vault_access_spikes is now keyVaultAccessSpikes.
  • The 10 rules in rules/cloud/azure/ that read them use exactly the same names. Their # Azure rule contract review v2.1.0 header is not a rule version line, so it was left as it is.
  • plugins/alerts:
    • New azure_marker_names_test.go. It reads the filter and every Azure rule, and checks that each marker the filter adds and each marker a rule reads have the identical name, made only of letters, digits and dots. It also checks that each history rule counts the value the filter adds.
    • The parse model in the tests now cleans grok, rename and add target names the way the parser does, and the history tests and test inputs use the new names.

Why

  • The EventProcessor parser keeps only letters, digits and dots in the field names it writes (utils.SanitizeField). The filter wrote log.correlationCandidate.key_vault_access_spikes, which is stored as log.correlationCandidate.keyvaultaccessspikes, while the rules counted the underscored name. So the history rules never counted anything.
  • The old tests missed this because their parse model kept the underscores. Against the previous names, the new test and the history threshold tests fail for every marker. On this commit they pass.
  • The same underscored markers are already in v11 from Fix Azure event mappings, outcomes and correlated detections #2637, so v11 has this problem until this merges.
  • Until the EventProcessor handles these characters, filters and rules should use only letters, digits and dots in field names.

Validation

  • Local EventProcessor playground on 47 real Azure records from production instances, comparing this draft before and after the change. All 47 events are parsed on both sides, with no errors and no alerts. The only marker these records trigger is the Key Vault one: before, it was stored as keyvaultaccessspikes on 3 events; after, it is keyVaultAccessSpikes on the same 3 events, which is the name the rule counts. Nothing else changed.
  • The playground also ran on the 185 cases in this pull request's Azure test data. All 10 markers are stored under their camelCase names, on 24 events, and every one of the 10 rules now has between 1 and 6 events carrying its marker, up from 0.
  • go test ./... in plugins/alerts passes. The Azure filter and rules have no marker or top-level field names with _ or other special characters. Nested vendor keys, such as the claim addresses and riskEventTypes_v2, are left as they are, because nested JSON keys keep their characters.

Not tested: the history thresholds themselves, because counting past events needs a search index. After deployment, only new events count toward the history windows.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant