Skip to content

Commit ff651dd

Browse files
kryonsxkryonsx
committed
fix(macos): align native log fields and correlation consumers
1 parent 01589d2 commit ff651dd

42 files changed

Lines changed: 6014 additions & 345 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎filters/audits/macos.md‎

Lines changed: 126 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,42 +1,139 @@
1-
# macOS normalization and rule review
1+
# macOS normalization and correlation review
22

3-
Scope Endpoint Security history and grouping to the documented agent dataSource; reject missing/unknown host identity.
3+
This draft targets UTMStack `v11` and changes both macOS filter configurations
4+
and all 36 macOS rules. The pinned `go-sdk v1.1.31` Event/Side protobuf is the
5+
schema authority. The local dictionary is supporting material.
46

5-
This draft targets UTMStack `v11`. It contains 0 filter changes
6-
and 1 rule changes for this technology only. Review covered
7-
2 filter configurations and 36 matching shipped rule files.
8-
Unchanged rules are listed in the regression manifest; they are not duplicated in the diff.
7+
## Producer and consumer corrections
98

10-
## Contract and validation
9+
The 30 sampled native raw records contain JSON with `message`, `process`,
10+
`class_name`, `level`, `timestamp`, and identifiers whose names contain underscores.
11+
The Go agent source supplies the dataType/dataSource envelope and forwards lines
12+
from an external collector; the collector implementation was not available in
13+
the inspected source tree. Its JSON shape is established by observed records,
14+
while Apple documentation establishes class and level semantics. SDK JSON field
15+
sanitization removes underscores and preserves letter case. The existing filter
16+
already produces `log.message`; 35 rules instead read `log.eventMessage`. Those
17+
rules now read the emitted message. The filter copies the process name to
18+
`origin.process`, the raw timestamp to `deviceTime`, and meaningful agent
19+
`dataSource` to `origin.host`. Every native record gets `dataSource` from the
20+
agent hostname; the native producer uses `unknown` if hostname lookup fails.
1121

12-
- Compared exact standard names/types with go-sdk v1.1.31 and the supplied UTMStack dictionaries.
13-
- Checked documented pipeline ordering, rename/move behavior, open vendor log fields,
14-
event-side versus alert-side fields, and surviving fields used by affected rule predicates/history/grouping.
15-
- Strict SDK configuration decoding and actual CEL compilation pass for this scope.
16-
- 2 synthetic normalization cases pass, including SDK Event conversion and any
17-
trigger predicate assertions recorded in the manifest.
18-
- The scoped alerts module tests and `git diff --check` pass with the shared contract runner applied.
22+
Legacy `eventMessage` and `messageType` remain available while supplying
23+
`log.message` and `log.level` when the native fields are absent. Native values
24+
win if both forms exist. `class_name` becomes the compatible `log.className`
25+
alias; the original sanitized `log.classname` remains available. The four
26+
existing activity/process/store/thread identifier aliases remain unchanged.
27+
There is no PID field in the SDK Side message, so the numeric process identifier
28+
stays in `log.processIdentifier`. No non-schema `origin.pid` is introduced.
1929

20-
The shared alert-contract PR supplies the reusable Go runner for the manifest in
21-
`plugins/alerts/testdata/filter-contracts/macos.json`. Apply that support before running `go test ./...` in `plugins/alerts`.
30+
Apple documents `OSLogEntryLog` as an entry generated by the logging API. Three
31+
class-gated rules accept this native class alongside their legacy `eventType`
32+
checks; the filter does not invent `eventType`. Keychain and Endpoint Security
33+
consume `log.level`. Apple debug, info/notice, error, and fault map respectively
34+
to standard severity debug, info, error, and critical; undefined remains unmapped.
35+
Vendor levels remain intact. A log level does not establish operation success,
36+
so the filter does not synthesize `actionResult`.
2237

23-
The changed rules also require the shared alert-grouping fix to resolve `lastEvent.*` values correctly at runtime.
38+
A native process string identifies the process name, not an executable path.
39+
Only an explicitly supplied legacy `processImagePath` is copied to `origin.path`,
40+
with its vendor alias preserved. Neither process nor sender is fabricated into
41+
a path. All 36 rules retain `adversary: origin`; the host/process promotions feed
42+
those existing actor roles. Host grouping now has an actual source identity;
43+
process grouping uses `adversary.process`. Optional user grouping remains empty
44+
when the producer has no user. Existing message grouping uses `lastEvent.log.message`.
2445

25-
The model starts from synthetic extraction results. It does not run complex grok,
26-
JSON/KV/XML/CSV extraction, time conversion, dynamic plugins, historical OpenSearch
27-
queries, or the closed EventProcessor. Raw vendor logs and resulting alerts must
28-
still be checked in staging before rollout. No customer false-positive reduction
29-
has been measured and no production rollout is included.
46+
## Wrappers and history
3047

31-
The shipped macOS filters do not write origin.host. The history query and grouping use the input dataSource instead; agent ingestion must supply its documented per-host identity. Missing/unknown dataSource is rejected. The normalization test supplies this input metadata explicitly.
48+
The optional forwarding prefix is anchored at the start of raw input. It first
49+
captures `log.datasourceoverride`, then promotes only a nonempty, whitespace-free
50+
value other than `unknown` to `dataSource`. Invalid overrides stay in their
51+
vendor field without replacing ingress identity. JSON `datasource_override`
52+
uses the same sanitized spelling and validation. The native filter handles the
53+
wrapper itself and both filter execution orders are tested. Wrapper and legacy
54+
message copies preserve newlines. The compatibility filter skips extraction
55+
when a message is already available, avoiding duplicate capture. The original
56+
noise-drop policy runs after JSON extraction and legacy level normalization,
57+
before the remaining normalization work.
58+
59+
Endpoint Security and ransomware history searches use the meaningful per-agent
60+
`dataSource` and an exact candidate marker computed by the filter. Without that
61+
marker, any earlier macOS event from the same source could satisfy a security
62+
threshold. Marker conditions repeat the complete rule trigger; tests check their
63+
parity. Untrusted incoming markers are removed and recomputed. The rules retain
64+
their existing counts/windows and reject empty/unknown identities. Filter and
65+
rules must deploy together; old indexed events do not have candidate markers,
66+
so each history window must warm up after deployment.
67+
68+
The kernel-extension rule no longer treats a missing or non-system executable
69+
path as evidence that the loaded extension is untrusted. Its first branch
70+
requires explicit unsigned or invalid-signature message evidence. Process
71+
exclusions in kernel-extension, XProtect and TCC checks require a present process
72+
name. The other negative predicates were checked for positive same-field gates.
73+
74+
## Validation and bounded evidence
75+
76+
- Standalone `macos_contract_test.go` tests 99 synthetic raw fixtures in three
77+
configurations: native filter alone, compatibility filter first, and
78+
compatibility filter last. This includes a native positive and benign negative
79+
for each of the 36 rules, wrapper edge cases, multiline/legacy messages,
80+
severity/identity mappings, noise drops and absent-field regressions.
81+
- All 36 real SDK CEL predicates are compiled/evaluated. Forty-two positive
82+
fixtures assert intended matches, source grouping identities, and available
83+
history placeholders. The two candidate-marker predicates must agree with
84+
their consumers. Negative fixtures assert that none of the 36 rules matches.
85+
- Real SDK historical requests execute against a local mock OpenSearch service.
86+
Both history rules are checked below/at count thresholds, against another
87+
source, benign non-candidate history, and expired events. The mock test runs
88+
in a subprocess because the SDK OpenSearch client is a process-wide singleton.
89+
- The shared contract manifest covers 90 raw JSON cases with opt-in decoding and recursive
90+
history-placeholder preflight supplied by draft #2590. The standalone macOS
91+
tests also cover wrappers, which the shared harness deliberately does not model.
92+
- Thirty distinct raw/normalized records from three instances were inspected
93+
read-only. All use native JSON; all lack `log.eventMessage`, `origin.host` and
94+
`origin.process`; all carry a raw timestamp differing from stored `deviceTime`.
95+
The relevant deployed repository filters match the baseline mappings. Private
96+
evidence retains instance/document anchors, raw records and configuration
97+
hashes without publishing customer payloads.
98+
- Replaying those 30 raw records through the offline parser model and actual SDK
99+
CEL yields zero candidates. No wrapper override was observed in bounded
100+
30-day searches on those instances. Wrapper defects are supported by source,
101+
deployed filter configuration and synthetic tests, not observed wrapper events.
102+
103+
Filter extraction and transformations are an offline model using SDK field
104+
sanitization and documented steps. Configuration decoding, CEL, Event conversion,
105+
placeholder expansion, query construction and count decisions use the actual
106+
SDK. This does not execute the closed EventProcessor or validate live alert
107+
creation/delivery. No customer writes or production rollout were performed.
108+
109+
## Activation and remaining validation
110+
111+
Correcting 35 message consumers can materially increase alert candidates.
112+
Populating host grouping can also consolidate events that previously had no
113+
usable grouping identity. The bounded sample had no candidates; it does not
114+
measure false-positive rates or establish detection coverage. Before rollout,
115+
staging must compare actual raw/output/alert records and per-rule volumes,
116+
identity grouping, time/count behavior, and expected benign traffic.
117+
118+
Native samples do not include a process image path. Legacy branches comparing
119+
that executable path to a TCC database or Gatekeeper resource path remain
120+
semantically uncertain and unexercised by the native positive fixtures. No
121+
resource path was invented as a process image to make those branches pass;
122+
message-based branches supply the tested matches. Further producer evidence is
123+
required before changing those resource-specific heuristics.
124+
125+
The shared alert-grouping correction in draft #2590 is required for runtime
126+
resolution of the remaining `lastEvent.*` grouping keys. Review and deploy the
127+
filter, its consumers and that grouping support together. This draft does not
128+
claim that existing heuristic matches prove malware or compromise.
32129

33130
## References
34131

35-
- [SDK schema](https://github.com/threatwinds/go-sdk/blob/v1.1.31/plugins/plugins.proto)
132+
- [SDK schema v1.1.31](https://github.com/threatwinds/go-sdk/blob/v1.1.31/plugins/plugins.proto)
133+
- [SDK sanitization](https://github.com/threatwinds/go-sdk/blob/v1.1.31/utils/fields.go)
134+
- [Native macOS producer](https://github.com/utmstack/UTMStack/blob/v11/agent/collector/platform/darwin.go)
36135
- [Filter steps](https://github.com/threatwinds/go-sdk/wiki/Filter-Steps-Reference)
37136
- [Standard event schema](https://github.com/threatwinds/go-sdk/wiki/Standard-Event-Schema)
38-
- [Rule implementation](https://github.com/threatwinds/go-sdk/wiki/Implementing-Rules)
39-
40-
`afterEvents`, empty noncapturing grok names, supported numeric strings, and custom
41-
`log.*` fields are accepted. Existing textual protocol casing and vendor action names
42-
are preserved unless a concrete consumer mismatch requires correction.
137+
- [Correlation rules](https://github.com/threatwinds/go-sdk/wiki/Implementing-Rules)
138+
- [Apple OSLogEntryLog](https://developer.apple.com/documentation/oslog/oslogentrylog)
139+
- [Apple log entry levels](https://developer.apple.com/documentation/oslog/oslogentrylog/level-swift.enum)

‎filters/macos/macos-syslog.yml‎

Lines changed: 21 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,29 @@
1+
# Compatibility wrapper extraction; macos.yml also handles this format.
12
pipeline:
2-
- dataTypes:
3-
- macos
3+
- dataTypes: [macos]
44
steps:
5-
# 1. Handle Agent Wrapper (Optional)
5+
# Optional forwarding wrapper. Anchoring prevents payload text from
6+
# overriding the source, and the message is captured exactly once.
67
- grok:
78
source: raw
89
patterns:
9-
- fieldName: log.wrapper_open
10-
pattern: '\[utm_stack_agent_ds='
11-
- fieldName: log.datasource_override
12-
pattern: '{{.data}}'
13-
- fieldName: log.wrapper_close
10+
- fieldName: ''
11+
pattern: '^\[utm_stack_agent_ds='
12+
- fieldName: log.datasourceoverride
13+
pattern: '[^\]\r\n]*'
14+
- fieldName: ''
1415
pattern: '\]-'
16+
- fieldName: log.wrapperMessage
17+
pattern: '(?s:.*)'
18+
where: startsWith("raw", "[utm_stack_agent_ds=") && !exists("log.message")
19+
- json:
20+
source: log.wrapperMessage
21+
where: regexMatch("log.wrapperMessage", "^\\s*\\{") && !exists("log.message")
22+
- grok:
23+
source: log.wrapperMessage
24+
patterns:
1525
- fieldName: log.message
16-
pattern: '{{.greedy}}'
17-
- fieldName: log.message
18-
pattern: '{{.greedy}}'
19-
20-
# 2. Cleanup
26+
pattern: '(?s:.*)'
27+
where: exists("log.wrapperMessage") && !regexMatch("log.wrapperMessage", "^\\s*\\{") && !exists("log.message")
2128
- delete:
22-
fields: [log.wrapper_open, log.wrapper_close]
29+
fields: [log.wrapperMessage]

0 commit comments

Comments
 (0)