Skip to content

Commit 00bac09

Browse files
osmonteroosmontero
authored
Merge pull request #2661 from kryonsx/codex/data-engine-o365-20260923
Reviewed 2026-09-25: workload-specific canonical outcomes (success/failure/denied), handles the AAD UserLoginFailed ResultStatus:Success quirk via RecordType 15 + LogonError; 4 new low-sev awareness rules key on actionResult=success + Exchange ops that are ingested (not in the drop list); forwarding rules key on the new log.o365* markers (with structured-Parameter fallback for older filters). DEPENDENCY: writes failure where v11 wrote failed - the EventProcessor feeds (TI) plugin skips only denied/blocked/failed, so companion EventProcessor #19 should land first to avoid a Known-Malicious-IP false-positive window on failed O365 sign-ins. 122 fabricated + 328 real records pass in the author's playground.
2 parents bf2c4aa + 007c722 commit 00bac09

21 files changed

Lines changed: 4771 additions & 158 deletions

‎filters/office365/o365.yml‎

Lines changed: 167 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Microsoft 365 filter, version 1.2.3
1+
# Microsoft 365 filter, version 1.4.1
22

33
pipeline:
44
- dataTypes:
@@ -8,6 +8,45 @@ pipeline:
88
source: raw
99
- drop:
1010
where: oneOf("log.Operation", ['AINotesUpdate', 'AcceptedSharingLinkOnFolder', 'AccessRequestUpdated', 'AccessedOdataLink', 'AddCommentToIncident.', 'AddFormCoauthor', 'AddRemediatedData', 'AddTagsToIncident', 'AdminThreadMuted', 'AdminThreadUnmuted', 'AlertExcelDownloaded', 'AlertNotificationsRecipientAdded', 'AllowShareFormForCopy', 'AppPublishedToCatalog', 'AppUpdatedInCatalog', 'AppUpgraded', 'ApplyRecordLabel', 'ApproveDisposal', 'AssignUserToIncident', 'AuditSearchCancelled', 'AuditSearchCompleted', 'AuditSearchCreated', 'AuditSearchExportJobCompleted', 'AuditSearchExportJobCreated', 'AuditSearchExportResultsDownloaded', 'BackupItemAdded', 'BackupItemRemoved', 'BackupItemRestoreCompleted', 'BackupItemRestoreTriggered', 'BackupPolicyActivated', 'BackupPolicyPaused', 'BotAddedToTeam', 'BreakEnded', 'BurnJob', 'CanceledQuery', 'CaseUpdated', 'ChannelAdded', 'ChannelOwnerResponded', 'ChatRetrieved', 'ChatUpdated', 'ClassificationAdded', 'ClassificationDefinitionCreated', 'ClientViewSignaled', 'ClockedIn', 'ClockedOut', 'CloseConversation', 'CollectionCreated', 'CollectionHardDeleted', 'CollectionRenamed', 'CollectionSoftDeleted', 'CollectionUpdated', 'ComplianceManagerAutomationChange', 'ComplianceSettingChanged', 'ConnectToExcelWorkbook', 'ConnectorAdded', 'CopilotInteraction', 'Copy', 'Create', 'CreateComment', 'CreateCopilotPlugin', 'CreateCopilotPromptBook', 'CreateForm', 'CreateResponse', 'CreateTag', 'CreateUpdateRequest', 'CreateWorkingSet', 'CreateWorkingSetSearch', 'DataExport', 'DataShareCreated', 'DataShareDeleted', 'DeleteCopilotPlugin', 'DeleteCopilotPromptBook', 'DeleteSummaryLink', 'DeleteTag', 'DeleteWorkingSetSearch', 'DeletedResult', 'DisableCopilotPlugin', 'DisableCopilotPromptBook', 'DisallowShareFormForCopy', 'DocumentSensitivityMismatchDetected', 'DomainControllerCoverageExcelDownloaded', 'DownloadCopyOfLakeData', 'DownloadDocument', 'DownloadedReport', 'DraftRestoreTaskCreated', 'DraftRestoreTaskDeleted', 'DraftRestoreTaskEdited', 'EditUpdateRequest', 'EnableCopilotPlugin', 'EnableCopilotPromptBook', 'EntityCreated', 'ErrorRemediationJob', 'ExchangeDataProactivelyPreserved', 'ExecutedQuery', 'ExportForm', 'ExportJob', 'ExtendRetention', 'FailedValidation', 'FileCheckOutDiscarded', 'FileCheckedIn', 'FileCheckedOut', 'FileCopied', 'FileDeletedFirstStageRecycleBin', 'FileDeletedSecondStageRecycleBin', 'FileModifiedExtended', 'FileRecycled', 'FileRestored', 'FileSyncDownloadedFull', 'FileSyncUploadedFull', 'FileUpdateDescription', 'FileUploaded', 'FileVersionRecycled', 'FileVersionsAllMinorsRecycled', 'FileVersionsAllRecycled', 'FileVisited', 'FolderCopied', 'FolderCreated', 'FolderDeletedFirstStageRecycleBin', 'FolderDeletedSecondStageRecycleBin', 'FolderRecycled', 'FolderRestored', 'FolderSharingLinkShared', 'GenerateCopyOfLakeData', 'GetAllRestoreArtifactsInTask', 'GetBackupItem', 'GetRestoreTaskDetails', 'GetSummaryLink', 'GlossaryTermAssigned', 'GlossaryTermCreated', 'HoldRemoved', 'HoldUpdated', 'HubSiteJoined', 'HubSiteOrphanHubDeleted', 'HubSiteRegistered', 'InformationBarriersInsightsReportCompleted', 'InformationBarriersInsightsReportOneDriveSectionQueried', 'InformationBarriersInsightsReportSchedule', 'InformationBarriersInsightsReportSharePointSectionQueried', 'InviteSent', 'InviteeResponded', 'LabelContentExplorerAccessedItem', 'LegacyWorkflowEnabledSet', 'LinkedEntityCreated', 'ListAllBackupItemsInPolicies', 'ListAllBackupItemsInTenant', 'ListAllBackupItemsInWorkload', 'ListAllBackupPolicies', 'ListAllRestorePoints', 'ListAllRestoreTasks', 'ListColumnCreated', 'ListColumnUpdated', 'ListContentTypeCreated', 'ListContentTypeDeleted', 'ListContentTypeUpdated', 'ListCreated', 'ListForms', 'ListItemCreated', 'ListItemRecycled', 'ListItemRestored', 'LiveNotesUpdate', 'LockRecord', 'LogsCollection', 'ManagedSyncClientAllowed', 'MarkedMessageChanged', 'MeetingDetail', 'MeetingParticipantDetail', 'MessageCreatedHasLink', 'MessageCreatedNotification', 'MessageCreation', 'MessageDeleted', 'MessageDeletedNotification', 'MessageEditedHasLink', 'MessageHostedContentsListed', 'MessageRead', 'MessageUpdated', 'MessageUpdatedNotification', 'MonitoringAlertNotificationRecipientAdded', 'MonitoringAlertUpdated', 'MoveForm', 'MovedFormIntoCollection', 'MovedFormOutofCollection', 'NewAdaptiveScope', 'NewBackupPolicyCreated', 'NewComplianceTag', 'NewRetentionCompliancePolicy', 'NewsFeedEnabledSet', 'OffShiftDialogAccepted', 'OfficeOnDemandSet', 'OpenConversation', 'OpenShiftAdded', 'PagePrefetched', 'PageViewed', 'PageViewedExtended', 'PeopleResultsScopeSet', 'PerformedCardAction', 'PlanCopied', 'PlanListRead', 'PreviewForm', 'PreviewItemDownloaded', 'PreviewItemListed', 'PreviewModeEnabledSet', 'PreviewWorkingSetSearch', 'ProInvitation', 'ProjectCreated', 'ProjectListAccessed', 'PulseCancel', 'PulseCreate', 'PulseCreateDraft', 'PulseDeleteDraft', 'PulseExtendDeadline', 'PulseInvite', 'PulseShareResults', 'PulseSubmit', 'QuarantinePreview', 'QuarantineReleaseRequest', 'QuarantineReleaseRequestDeny', 'QuarantineViewHeader', 'RecordDelete', 'RelabelItem', 'RemediationActionAdded', 'RemediationActionUpdated', 'RemoveAdaptiveScope', 'RemoveComplianceTag', 'RemoveCuratedTopic', 'RemoveFormCoauthor', 'RemoveTagsFromIncident', 'RemovedSearchExported', 'RemovedSearchPreviewed', 'RemovedSearchResultsPurged', 'RemovedSearchResultsSentToZoom', 'ReportDownloaded', 'RequestAdded', 'RequestRespondedTo', 'RestoreTaskActivated', 'RestoreTaskCompleted', 'RoadmapAccessed', 'RoadmapCreated', 'RoadmapItemAccessed', 'RoadmapItemCreated', 'RunAntiVirusScan', 'ScheduleGroupAdded', 'ScheduleShared', 'SearchPermissionUpdated', 'SearchQueryPerformed', 'SearchRemoved', 'SearchReport', 'SearchReportRemoved', 'SearchResultsSentToZoom', 'SearchStopped', 'SearchUpdated', 'SearchViewed', 'Send', 'SensitivityLabelApplied', 'SensitivityLabelChanged', 'SensorActivationMethodConfigurationUpdated', 'SensorCreated', 'SensorDeploymentAccessKeyReceived', 'SensorDeploymentAccessKeyUpdated', 'SetAdvancedFeatures', 'SetRestrictiveRetentionUI', 'SharePointDataProactivelyPreserved', 'SharingInvitationRevoked', 'SharingInvitationUpdated', 'ShiftAdded', 'SiteAdminChangeRequest', 'SiteCollectionCreated', 'SiteCollectionQuotaModified', 'SiteColumnCreated', 'SiteColumnDeleted', 'SiteColumnUpdated', 'SiteContentTypeCreated', 'SiteContentTypeDeleted', 'SiteContentTypeUpdated', 'SoftDeleteSettingsUpdated', 'SubTaskCreated', 'SubmitResponse', 'SubmitUpdate', 'SubscribedToMessages', 'SupervisionRuleMatch', 'SupervisoryReviewTag', 'TabAdded', 'TabUpdated', 'TagFiles', 'TagJob', 'TaggingConfigurationUpdated', 'TaskAccessed', 'TaskAssigned', 'TaskCompleted', 'TaskCreated', 'TaskListCreated', 'TaskListRead', 'TaskRead', 'TeamCreated', 'ThreadAccessFailure', 'ThreadViewed', 'TimeClockEntryAdded', 'TimeOffAdded', 'UnlockRecord', 'Update', 'UpdateCopilotPlugin', 'UpdateCopilotPromptBook', 'UpdateCopilotSettings', 'UpdateIncidentStatus', 'UpdateResponse', 'UpdateTag', 'UpdateUsageReportsPrivacySetting', 'UpdateWorkingSetSearch', 'UpdatedPolicyConfigPriority', 'UploadedOrgData', 'UsagePolicyAcceptance', 'ViewBackupPolicyDetails', 'ViewDocument', 'ViewForm', 'ViewResponse', 'ViewResponses', 'ViewRuntimeForm', 'ViewUpdate', 'ViewedExplore', 'ViewedSearchExported', 'ViewedSearchPreviewed', 'WorkforceIntegrationAdded', 'WorkspaceCreated', 'updateddeviceconfiguration'])
11+
# Capture documented Name/Value parameter identities before the legacy string
12+
# cast loses their structure. The inbox rule and send-on-behalf flags mean a
13+
# setting was supplied, including clearing it. The mailbox flag needs a non-empty
14+
# forwarding address: Exchange records a cleared address as an empty value, and
15+
# DeliverToMailboxAndForward alone sets no destination. No flag establishes an
16+
# external recipient. Never trust markers supplied by the input record.
17+
- delete:
18+
fields:
19+
- log.o365InboxForwardingChange
20+
- log.o365MailboxForwardingSet
21+
- log.o365SendOnBehalfChange
22+
- add:
23+
function: string
24+
params:
25+
key: log.o365InboxForwardingChange
26+
value: "true"
27+
where: >-
28+
equals("log.Workload", "Exchange") &&
29+
oneOf("log.Operation", ["New-InboxRule", "Set-InboxRule"]) &&
30+
(exists("log.Parameters.#(Name==ForwardTo).Name") ||
31+
exists("log.Parameters.#(Name==ForwardAsAttachmentTo).Name") ||
32+
exists("log.Parameters.#(Name==RedirectTo).Name"))
33+
- add:
34+
function: string
35+
params:
36+
key: log.o365MailboxForwardingSet
37+
value: "true"
38+
where: >-
39+
equals("log.Workload", "Exchange") && equals("log.Operation", "Set-Mailbox") &&
40+
(regexMatch("log.Parameters.#(Name==ForwardingAddress).Value", "(?s)^.+$") ||
41+
regexMatch("log.Parameters.#(Name==ForwardingSmtpAddress).Value", "(?s)^.+$"))
42+
- add:
43+
function: string
44+
params:
45+
key: log.o365SendOnBehalfChange
46+
value: "true"
47+
where: >-
48+
equals("log.Workload", "Exchange") && equals("log.Operation", "Set-Mailbox") &&
49+
exists("log.Parameters.#(Name==GrantSendOnBehalfTo).Name")
1150
- cast:
1251
fields:
1352
- log.Parameters
@@ -73,18 +112,125 @@ pipeline:
73112
from:
74113
- log.DestFolder.Path
75114
to: log.destFolderPath
115+
# ResultStatus is workload-specific. Preserve it and derive only established
116+
# operation outcomes; partial, pending and unknown results remain unset.
117+
- delete:
118+
fields:
119+
- actionResult
76120
- add:
77121
function: string
78122
params:
79123
key: actionResult
80124
value: success
81-
where: oneOf("log.ResultStatus", ["Succeeded", "Success", "Successful", "True"]) && !equals("action", "UserLoginFailed")
125+
where: >-
126+
!oneOf("log.RecordType", [1, 15, 41, 188, 189, 190, 191, 192, 193, 194, 231, 386, 387, 401, 402]) &&
127+
oneOf("log.ResultStatus", ["Succeeded", "Success", "Successful"]) &&
128+
!equals("action", "UserLoginFailed")
129+
# ExchangeAdmin uses string True/False; these are not universal outcome codes.
130+
- add:
131+
function: string
132+
params:
133+
key: actionResult
134+
value: success
135+
where: equals("log.RecordType", 1) && equals("log.ResultStatus", "True")
136+
# STS HTTP success and UserLoggedIn alone do not prove completed authentication.
137+
# Require an explicit zero result code and reject contradictory/unknown errors.
138+
# ErrorNumber is the corresponding field in observed STS audit records.
139+
- add:
140+
function: string
141+
params:
142+
key: actionResult
143+
value: success
144+
where: >-
145+
equals("log.RecordType", 15) &&
146+
(equals("log.ErrorCode", 0) || equals("log.ErrorNumber", 0)) &&
147+
(!exists("log.ErrorCode") || equals("log.ErrorCode", 0)) &&
148+
(!exists("log.ErrorNumber") || equals("log.ErrorNumber", 0)) &&
149+
(!exists("log.LogonError") || regexMatch("log.LogonError", "^$")) &&
150+
(!exists("log.ResultStatus") || oneOf("log.ResultStatus", ["Succeeded", "Success", "Successful"])) &&
151+
!equals("action", "UserLoginFailed")
152+
# Planner defines its own result enum. Audit records also emit its member names.
153+
- add:
154+
function: string
155+
params:
156+
key: actionResult
157+
value: success
158+
where: >-
159+
oneOf("log.RecordType", [188, 189, 190, 191, 192, 193, 194, 231, 386, 387, 401, 402]) &&
160+
(equals("log.ResultStatus", 1) || equals("log.ResultStatus", "Success"))
161+
# Safe Links reports the navigation decision separately from event processing.
162+
# Values 4/5 mean the user overrode the page and navigated; 3 remains pending.
163+
- add:
164+
function: string
165+
params:
166+
key: actionResult
167+
value: success
168+
where: equals("log.RecordType", 41) && oneOf("log.URLClickAction", [4, 5])
169+
- add:
170+
function: string
171+
params:
172+
key: actionResult
173+
value: failure
174+
where: >-
175+
!oneOf("log.RecordType", [41, 188, 189, 190, 191, 192, 193, 194, 231, 386, 387, 401, 402]) &&
176+
(oneOf("log.ResultStatus", ["Failure", "Failed"]) ||
177+
(equals("log.RecordType", 1) && equals("log.ResultStatus", "False")))
82178
- add:
83179
function: string
84180
params:
85181
key: actionResult
86-
value: failed
87-
where: oneOf("log.ResultStatus", ["Failure", "Failed"])
182+
value: failure
183+
where: >-
184+
equals("log.RecordType", 15) &&
185+
(greaterThan("log.ErrorCode", 0) || regexMatch("log.LogonError", "(?s)^.+$"))
186+
- add:
187+
function: string
188+
params:
189+
key: actionResult
190+
value: failure
191+
where: >-
192+
oneOf("log.RecordType", [188, 189, 190, 191, 192, 193, 194, 231, 386, 387, 401, 402]) &&
193+
(equals("log.ResultStatus", 2) || equals("log.ResultStatus", "Failure"))
194+
- add:
195+
function: string
196+
params:
197+
key: actionResult
198+
value: denied
199+
where: >-
200+
!oneOf("log.RecordType", [41, 188, 189, 190, 191, 192, 193, 194, 231, 386, 387, 401, 402]) &&
201+
equals("log.ResultStatus", "Blocked")
202+
- add:
203+
function: string
204+
params:
205+
key: actionResult
206+
value: denied
207+
where: >-
208+
oneOf("log.RecordType", [188, 189, 190, 191, 192, 193, 194, 231, 386, 387, 401, 402]) &&
209+
(equals("log.ResultStatus", 3) || equals("log.ResultStatus", "AuthorizationFailure"))
210+
# These exact operation names describe prevented user access, not successful
211+
# administrative changes to a blocking policy.
212+
- add:
213+
function: string
214+
params:
215+
key: actionResult
216+
value: denied
217+
where: >-
218+
oneOf("action", ["SharingInvitationBlocked", "UnmanagedSyncClientBlocked", "URLNavigationBlocked", "AccessRequestDenied"]) ||
219+
(equals("log.RecordType", 41) && equals("log.URLClickAction", 2))
220+
# KmsiInterrupt is the expected "Keep me signed in" prompt, not a final result.
221+
- delete:
222+
fields:
223+
- actionResult
224+
where: >-
225+
equals("log.RecordType", 15) &&
226+
(equals("log.ErrorCode", 50140) || equals("log.ErrorNumber", 50140))
227+
# A failed login wins even when its HTTP request or audit operation succeeded.
228+
- add:
229+
function: string
230+
params:
231+
key: actionResult
232+
value: failure
233+
where: equals("action", "UserLoginFailed")
88234
- dynamic:
89235
plugin: com.utmstack.geolocation
90236
params:
@@ -141,24 +287,23 @@ pipeline:
141287
- log.ItemName
142288
to: target.filename
143289
where: equals("log.Workload", "OneDrive") || equals("log.Workload", "SharePoint")
144-
- add:
145-
function: string
146-
params:
147-
key: actionResult
148-
value: success
149-
where: equals("log.ResultStatus", "PartiallySucceeded")
150-
- add:
151-
function: string
152-
params:
153-
key: actionResult
154-
value: blocked
155-
where: equals("log.ResultStatus", "Blocked")
156-
- add:
157-
function: string
158-
params:
159-
key: actionResult
160-
value: failed
161-
where: equals("action", "UserLoginFailed")
290+
# Safe Links supplies the clicking user's IP and destination URL under its own
291+
# documented fields. Copy them so vendor-specific queries retain the originals.
292+
- grok:
293+
source: log.UserIp
294+
patterns:
295+
- fieldName: origin.ip
296+
pattern: (?s:.*)
297+
where: >-
298+
equals("log.RecordType", 41) &&
299+
(inCIDR("log.UserIp", "0.0.0.0/0") || inCIDR("log.UserIp", "::/0")) &&
300+
!inCIDR("log.UserIp", "0.0.0.0/32") && !inCIDR("log.UserIp", "::/128")
301+
- grok:
302+
source: log.URL
303+
patterns:
304+
- fieldName: target.url
305+
pattern: (?s:.*)
306+
where: equals("log.RecordType", 41) && regexMatch("log.URL", "(?s)^.+$")
162307
- delete:
163308
fields:
164309
- log.AppAccessContext

0 commit comments

Comments
 (0)