|
| 1 | +# Deceptive Bytes v11 filter review |
| 2 | + |
| 3 | +The filter previously extracted a command into a root `command` field. The v11 alert |
| 4 | +module pins go-sdk v1.1.36 (v1.1.33 when this was first reviewed); neither version's |
| 5 | +`Event` has such a field. Finalization discards it. |
| 6 | +The corrected grok and both quote trims use `origin.command`, the documented |
| 7 | +`Side.command` field. Three KV inputs are optional products of different grok |
| 8 | +branches; the public KV plugin returns an error when a source is absent. Each KV |
| 9 | +step now runs only when its source exists. Present-source separators and outputs |
| 10 | +are unchanged. The filter's explicit `log.action=blocked` or `prevented` case |
| 11 | +now yields the documented `actionResult=denied` value. It retains the vendor |
| 12 | +action in `log.action`, and no shipped Deceptive Bytes rule reads root |
| 13 | +`actionResult`. |
| 14 | + |
| 15 | +The following raw inputs are **fabricated parser fixtures**, not captured Deceptive |
| 16 | +Bytes records: |
| 17 | + |
| 18 | +| Case | Raw input | Original output | Corrected output | |
| 19 | +|---|---|---|---| |
| 20 | +| Command | `<14>2026-09-23T12:00:00Z,123,-,45,source,67,path,platform,/tmp/fake.bin,"synthetic --flag"` | `origin.path=/tmp/fake.bin`; command absent; three missing-source KV errors | same path; `origin.command=synthetic --flag`; zero errors | |
| 21 | +| Present KV source | `<14>1 2026-09-23T12:00:00Z host 2 foo:1 sampleKey=sampleValue` | `log.sampleKey=sampleValue`; two missing-source KV errors | same vendor fields; zero errors | |
| 22 | +| Blocked action | `<14>1 2026-09-23T12:00:00Z host 2 foo:1 action=blocked` | `actionResult=blocked`; two missing-source KV errors | `actionResult=denied`, `log.action=blocked`; zero errors | |
| 23 | +| Unrelated raw | `not a Deceptive Bytes message` | three missing-source KV errors | zero errors; no command | |
| 24 | + |
| 25 | +These four input pairs first produced eight events with the public EventProcessor |
| 26 | +playground at commit `497bf53dbd1ae096f7b2dbc7bce77a6bf9f22ce1`, whose parser and |
| 27 | +writer link go-sdk v1.1.26. On 2026-09-24 they were run again with EventProcessor |
| 28 | +`main` at `8a3ade72bd9d12db21f6b273200588fb49540f14`, whose playground and plugins all |
| 29 | +link go-sdk v1.1.36, the version the v11 alerts module now pins: every original and |
| 30 | +corrected output in the table is unchanged. The common regex patterns |
| 31 | +were read from a deployed UTMStack configuration. The complete staged inputs, |
| 32 | +binary hashes and resulting events are retained in the private Data Engine review |
| 33 | +run. SDK CEL evaluated the diagnostic predicate |
| 34 | +`equals("dataType", "deceptive-bytes") && equals("origin.command", "synthetic --flag")` |
| 35 | +on all eight resulting events: only the corrected command case matched. The focused |
| 36 | +contract test and full `plugins/alerts` Go suite pass with go-sdk v1.1.36 (48 tests |
| 37 | +pass, 11 skip because they need other technologies' private evidence, none fail). |
| 38 | +The builds are not asserted to be identical to a customer deployment. |
| 39 | + |
| 40 | +The newest published engine image, `ghcr.io/utmstack/utmstack/eventprocessor:v11.2.14` |
| 41 | +(built 2026-09-24 19:13 UTC on base image `eventprocessor/base:1.1.7`), embeds Go build |
| 42 | +information showing that its playground and plugin binaries come from the same |
| 43 | +EventProcessor revision `8a3ade7` with go-sdk v1.1.36, built with go1.26.8 for |
| 44 | +linux/amd64. The local build used here is that source revision compiled natively for |
| 45 | +darwin/arm64 with go1.25.7; only the Go toolchain and platform differ. |
| 46 | + |
| 47 | +All 16 shipped Deceptive Bytes rules were checked for consumers of `command`, |
| 48 | +`origin.command` and the three temporary KV inputs. None reads them, so the |
| 49 | +command fix itself needs no rule rewrite. |
| 50 | + |
| 51 | +Six shipped rules read 18 vendor keys that contain an underscore and reach the |
| 52 | +event through KV, for example `log.event_type`, `log.decoy_sensitivity` and |
| 53 | +`log.source_ip`, in predicates, history fields and placeholders, and grouping |
| 54 | +paths. The KV parser passes every key through go-sdk `utils.SanitizeField`. Up to |
| 55 | +v1.1.34 that function removed underscores, so KV stored `log.eventtype` and these |
| 56 | +rules could not match; an earlier revision of this draft therefore respelled the 18 |
| 57 | +names without underscores. Since v1.1.35 the function keeps underscores, and `v11` |
| 58 | +now pins v1.1.36, so KV stores the vendor's own spelling and those respelled names |
| 59 | +would never match. This revision restores the original names. Three of the six rules |
| 60 | +are again identical to `v11`; `data_theft_attempt_indicators` keeps only its |
| 61 | +`origin.ip` guard, `ransomware_behavior_patterns` only its placeholder guard (below) and |
| 62 | +`nation_state_tactic_detection` only its `"true"` comparisons. |
| 63 | +KV also stores values as strings, so ten boolean comparisons across five rules use |
| 64 | +`"true"`; go-sdk v1.1.36 CEL still accepts a native boolean `true` for those |
| 65 | +comparisons. Literal event labels, thresholds and source-IP requirements are |
| 66 | +unchanged. Six source rules needed neither field nor boolean changes. |
| 67 | + |
| 68 | +Three rules run a history search on `{{.origin.ip}}` (and `{{.log.tacticName}}` or |
| 69 | +`{{.log.processName}}`), but this filter never writes `origin.ip`. A missing |
| 70 | +placeholder makes the search fail, and five failures switch a rule off with a |
| 71 | +Circuit Breaker alert, so the data theft, advanced threat tactic and zero-day |
| 72 | +conditions now also require those fields, as the other seven `origin.ip` rules of |
| 73 | +this source already do. For the same reason `ransomware_behavior_patterns`, which |
| 74 | +searches on `{{.log.process}}` and `{{.log.source_ip}}`, now requires both fields. The |
| 75 | +history-guard test checks each of the four rules: no match without the fields it needs or |
| 76 | +without any one of them, a match with them, and every placeholder resolved. Without the |
| 77 | +ransomware guard it fails; the go-sdk v1.1.36 replay over the playground events plus two |
| 78 | +copies of the ransomware line that each lack one of those fields then matched both copies |
| 79 | +with unresolved placeholders. With the guard the same replay passes 65 of 65 checks: each |
| 80 | +rule matches only its intended case, and the ransomware rule only the line that carries |
| 81 | +both fields. |
| 82 | + |
| 83 | +The committed fabricated lines carry all 18 keys. On EventProcessor `8a3ade7` the KV |
| 84 | +plugin stored every one with its underscore (for example `log.event_type`, |
| 85 | +`log.process_name` and `log.deceptive_target`) and none without it. With the |
| 86 | +respelled rules the Living Off The Land positive yielded no alert. With the restored |
| 87 | +rules the playground raised exactly one alert for each of the Living Off The Land, |
| 88 | +nation-state and privilege-escalation positives, containing only that line's event |
| 89 | +ID, and none for the near-miss negative or any other line; these three rules need no |
| 90 | +history search. The go-sdk v1.1.36 rule replay over the same events, plus copies |
| 91 | +given an `origin.ip` and synthetic events for the boolean rules, matched each of the |
| 92 | +six restored rules and the four other rules this draft edits only on its intended |
| 93 | +case, resolved every history placeholder on those matches, and matched nothing with |
| 94 | +the six other rules or with the respelled names (89 of 89 checks). This proves the |
| 95 | +local parser and rule contract for these fabricated cases, not the frequency or |
| 96 | +semantics of real vendor detections. |
| 97 | + |
| 98 | +No retained Deceptive Bytes documents were found in 29 successful source-index |
| 99 | +discovery queries across the accessible v11 estate; two discovery attempts failed. |
| 100 | +Thus raw vendor syntax, deployed parser version, production rule coverage, alert |
| 101 | +grouping and notification remain unverified. The available official product pages |
| 102 | +do not define the severity-letter crosswalk, actor roles, or the vendor event |
| 103 | +labels assumed by these rules. Those semantic mappings are left for |
| 104 | +a separate review with relevant source logs or a technical export specification. |
| 105 | + |
| 106 | +Sources: [SDK Event schema](https://github.com/threatwinds/go-sdk/blob/v1.1.36/plugins/plugins.proto), |
| 107 | +[standard field meanings](https://github.com/threatwinds/go-sdk/wiki/Standard-Event-Schema), |
| 108 | +[filter steps](https://github.com/threatwinds/go-sdk/wiki/Filter-Steps-Reference), |
| 109 | +[public KV parser](https://github.com/utmstack/EventProcessor/blob/8a3ade72bd9d12db21f6b273200588fb49540f14/plugins/kv/main.go), |
| 110 | +[SDK field sanitizer](https://github.com/threatwinds/go-sdk/blob/v1.1.36/utils/fields.go) (keeps letters, |
| 111 | +digits, dots and underscores since v1.1.35). |
| 112 | + |
| 113 | +## Reproduce the raw parser and local alert checks |
| 114 | + |
| 115 | +`plugins/alerts/testdata/deceptive-bytes/` contains eleven fabricated raw inputs, |
| 116 | +the twelve common regex definitions needed by this filter, and `replay.py`. |
| 117 | +It stages the current filter and the three shipped rules that need no history search |
| 118 | +(Living Off The Land, nation-state, privilege escalation), runs the actual |
| 119 | +playground, and requires all eleven finalized events, zero parser errors, every |
| 120 | +vendor key stored with its underscore, and exactly one alert from each rule, on its |
| 121 | +positive line. The unrelated, near-miss and other cases must not alert. It records |
| 122 | +the binary build information, input/configuration hashes and outputs in a fresh |
| 123 | +local directory. It uses no customer connection or index writer. On EventProcessor |
| 124 | +`8a3ade7` it passes: 11 events and 3 alerts. |
| 125 | + |
| 126 | +Build a separately checked-out EventProcessor at `8a3ade72bd9d12db21f6b273200588fb49540f14`, |
| 127 | +preserving each module's checked-in dependencies. With `EP` set to that checkout's |
| 128 | +absolute path: |
| 129 | + |
| 130 | +```sh |
| 131 | +mkdir -p "$EP/test-bin" "$EP/test-plugins" |
| 132 | +(cd "$EP" && go build -mod=readonly -o "$EP/test-bin/playground" ./cmd/playground) |
| 133 | +for plugin in add grok delete sew kv trim cel saw; do |
| 134 | + (cd "$EP/plugins/$plugin" && go build -mod=readonly -o "$EP/test-plugins/$plugin.plugin" .) |
| 135 | +done |
| 136 | +``` |
| 137 | + |
| 138 | +From this UTMStack checkout, use a Python environment with PyYAML installed: |
| 139 | + |
| 140 | +```sh |
| 141 | +python3 plugins/alerts/testdata/deceptive-bytes/replay.py \ |
| 142 | + --playground "$EP/test-bin/playground" --plugins "$EP/test-plugins" |
| 143 | +(cd plugins/alerts && go test ./... -count=1) |
| 144 | +``` |
| 145 | + |
| 146 | +The Python replay and Go suite are separate checks. The Go suite alone does not |
| 147 | +execute raw extraction. Playground startup can take several minutes. At `8a3ade7` |
| 148 | +the CEL plugin reads its OpenSearch address from separate `host`, `port`, `user` and |
| 149 | +`password` settings, so the single loopback URL in `replay.py` leaves it an empty |
| 150 | +address; the staged rules have no history request. This does not validate any other |
| 151 | +rule's history, production grouping or notification. |
0 commit comments