[zanetworker]

Summary

Native OCSF 1.8.0 events failed the official OCSF validator: HTTP Activity emitted an attribute the class doesn't define, Process Activity events had no actor, connection_info lacked direction_id, and SSH Activity events named no endpoint. The vendored-schema tests missed these because they checked only top-level, non-profile attributes.

This makes every builder's native output pass https://schema.ocsf.io/api/1.8.0/v2/validate (apart from top-level container, tracked on #4915) and adds structural checks that keep it that way, including an e2e test that validates the supervisor's real JSONL. The checks cover required and defined attributes, profiles, and at_least_one constraints, but not value types or enum ranges; the official validator stays the reference for those (see schemas/ocsf/README.md).

The schema downgrade is out of scope. #4284 tracks picking the one version OpenShell emits (1.1 or 1.8) and removing the downgrade; until then the existing downgrade is unchanged, so output with ocsf_schema_version set is still not conformant. This PR is needed either way:

• If 1.8 stays, this is the conformance work, and bug(ocsf): schema downgrade labels events 1.1 or 1.3 that do not conform to those versions #4284 only removes the downgrade and deprecates the setting.
• If OpenShell moves to 1.1, the validator and e2e show exactly which emitters need work once the 1.1 schemas are vendored and OCSF_VERSION changes.

Related Issue

Closes #4283

With #4677 (device.os.type_id), this meets #4283's acceptance items apart from top-level container, which this PR leaves unchanged. #4915 tracks container.

Changes

• HTTP Activity no longer emits is_src_dst_assignment_known, which no OCSF version defines on that class.
• ProcessActivityBuilder requires an acting process before build() (type-state, like the HTTP and Network builders). Sandbox process events name the openshell-sandbox launcher with its real pid.
• connection_info carries the required direction_id and direction, defaulting to Unknown.
• SshActivityBuilder requires a source or destination endpoint before build(). All eight SSH events had none; they now name the supervisor's SSH server as dst_endpoint (svc_name: "ssh" and the listen socket path in name), using a new Endpoint::svc_name field. Each SSH server builds its endpoint from its own socket and passes it to its connection handlers, so a shared supervisor hosting several sandboxes (refactor(supervisor): isolate state per sandbox instance #4701) names the right socket on each sandbox's events.
• validate_required_fields and validate_known_fields share one recursive walker. Both reject undefined attributes at every level and, when validating an event, attributes from profiles the event doesn't declare. validate_required_fields also checks required attributes and at_least_one constraints, and fails any event not labelled 1.8.0. New test every_builder_emits_schema_conformant_events covers one event per builder, including Authentication and a fully populated Detection Finding with an ATT&CK mapping (tactic and technique are vendored for it).
• New e2e test ocsf_jsonl_conformance (Podman CI set) enables JSONL, generates denied TCP traffic and a plain HTTP request through the sandbox proxy, waits until Network, HTTP and SSH Activity records are all present, copies the supervisor's JSONL with podman cp, and validates every record. It covers those production call sites rather than builders, and found the SSH endpoint gap. When records can't be copied, the failure names the podman command and its error. The e2e crate takes openshell-ocsf (with test-support) as a dev-dependency so both suites share one validator.
• With test-support, vendored schemas are compiled into the binary (validation/embedded.rs, drift-checked by embedded_schemas_match_vendored_files), so archived e2e runs on CI machines need no source files. The fallback EventContext carries a sandbox identity, so producer tests exercise the top-level container exception as a supervisor with a sandbox ID emits it. Top-level container is the single named exception in the validator, linked to bug(ocsf): top-level container is undefined in OCSF 1.8.0 event classes #4915.
• schemas/ocsf/README.md documents the embedded table, regeneration, and validation with the official OCSF server through its v2 endpoint (/api/v2/validate locally, /api/1.8.0/v2/validate hosted), which reports constraint failures such as at_least_one that the older /api/validate misses. The unused v1.7.0 schemas are removed, so only 1.8.0 is vendored and embedded.
• embedded_schemas_match_vendored_files normalizes line endings, so CRLF checkouts on Windows pass, and skips when the source checkout isn't present, as in a relocated nextest archive.
• Docs: ocsf-json-export.mdx describes the per-class fields (SSH dst_endpoint, Process actor.process, connection_info.direction_id) and notes that top-level container is the one attribute OCSF 1.8.0 doesn't define.

User-visible: HTTP events drop is_src_dst_assignment_known; SSH events gain dst_endpoint; connection_info gains direction_id; Process Activity events gain actor.process (the openshell-sandbox launcher).

Deferred: top-level container is undefined in every OCSF version but identifies the affected sandbox on gateway events (#4915). The SIGTERM-failure Process Activity event still uses the Process::new("unknown", 0) fallback.

Testing

Rebased on main at fdce6ef26, which includes #4677 (device.os.type_id), the gateway authentication events from #4728, #4761 and #4762, the per-sandbox supervisor state from #4701, the Windows build fix from #4862, and the agent OTLP trace relay from #4276.

• Real run on Podman (OPENSHELL_E2E_PODMAN_TEST=ocsf_jsonl_conformance mise run e2e:podman, gateway and supervisor images built from this branch): supervisor_jsonl_conforms_to_the_native_schema passes. It runs an L7 policy against a local test server, requires Network, HTTP and SSH Activity records, and validates every supervisor JSONL record against 1.8.0. An earlier version of the test, with raw sockets only, produced no HTTP Activity at all; requiring the classes caught that. Before the SSH fix it failed with "SSH Activity" 1.8.0 "SSH connection accepted on supervisor Unix socket": Missing at_least_one field from [dst_endpoint, src_endpoint].
• cargo test for openshell-ocsf (with test-support), openshell-server, openshell-supervisor, openshell-supervisor-network, openshell-supervisor-process, openshell-sandbox, openshell-otel: all pass (4783 tests across the seven crates). The embedded-schema drift test also passes against a CRLF copy of embedded.rs.
• cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, and no clippy findings in the new e2e file
• Unit tests added/updated

ssh::exec_input_tests::eof_preserves_queued_input_until_the_child_reads has a 5 s deadline and hit it on this machine on both main and this branch in one run; it passed in the final run.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[johntmyers]

/ok to test 78621e8

[github-actions]

Label test:e2e applied for 78621e8. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

gator-agent

PR Review Status

The independent review found no blocking defects in the OCSF conformance and schema downgrade changes. @zanetworker, I checked your note about top-level container: it is present in the base, remains an acknowledged design question, and the downgrade preserves its data under unmapped.

Current-head testing has started with test:e2e and test:windows. Trivy and Helm are green; Branch Checks, E2E, and both Windows lanes are pending. The E2E Label Help bot requests Re-run all jobs, but the identified run still has an active first attempt.

Action required: rerun Branch E2E Checks after the active attempt finishes. Gator has operator authorization and the narrow sandbox permission to perform that rerun in a subsequent cycle.

Blocking findings: None.

Carried findings: None.

Gator metadata

• Validation: Focused OCSF correctness fixes for linked issues #4283 and #4284; operator requested review and relevant tests.
• Docs: Fern documentation updates describe OS identity, mixed-version JSONL, downgrade preservation, gateway configuration, and the kept-native metric.
• Checks: DCO, OpenShell / Helm Lint, and OpenShell / Trivy Changes pass; Branch Checks and both Windows MSVC PR lint and test lanes are pending on the reviewed head.
• E2E: test:e2e applied; current-head mirror verified; Branch E2E Checks run 37657018590 attempt 1 is active. Bot-required rerun remains outstanding. Relevant E2E coverage includes the new Podman JSONL conformance test.
• Head SHA: 78621e8bb1fb9af4c91f9cf8dd5c3ebffc62b963
• Base SHA: 3fc93e28273fe304db634e6d5e9a52d57200389e
• Merge base SHA: 834b79a8c2351370e7c6c363423f5be18d440256
• Patch ID: 33dfd9e90bc20c4d4b8493efc716d9a8d3c32edb
• Gator payload: 11
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:blocked
• Blocked reason: test_dispatch_required

[johntmyers]

gator-agent

Author Follow-Up Nudge

This PR has been in gator:in-review for more than 48 business hours and required CI still needs attention.

@zanetworker, please make the vendored OCSF schemas available to the relocated Podman E2E test and push an update. The rootless Podman run fails in supervisor_jsonl_conforms_to_the_version_each_record_claims because the loader uses the build machine's CARGO_MANIFEST_DIR and cannot find schemas/ocsf/v1.1.0/classes/base_event.json on the remote test machine. Verify that the archived test can load its schemas and validate the JSONL records there.

@NVIDIA/openshell-maintainers, both Windows lanes also need a gate resolution: Clippy reports unnecessary_wraps and needless_pass_by_ref_mut in crates/openshell-cli/src/ssh.rs at lines 369, 376, and 423. That file is unchanged by this PR.

Gator metadata

• Head SHA: 78621e8bb1fb9af4c91f9cf8dd5c3ebffc62b963
• Gator payload: 11
• Next state: gator:in-review
• Nudge scope: Existing CI follow-up; no new code-review disposition.

[zanetworker]

Fixed in aae174742: the vendored schemas are now compiled into the test binary (crates/openshell-ocsf/src/validation/embedded.rs, generated with a drift test), so archived binaries need no files from the build machine. Verified by running the ocsf_jsonl_conformance nextest archive from an unrelated directory with --workspace-remap and the repo's schemas/ hidden, against a Podman gateway: 1 passed. The Windows Clippy findings are in crates/openshell-cli/src/ssh.rs, which this PR doesn't change.

@johntmyers could you take a look, and /ok to test aae17474233af3ddd272f43557a01a7cf0c7f677 when you get a chance?

[johntmyers]

/ok to test aae1747

[zanetworker]

Thanks @houman44, this is a useful before-picture, and it matches what the PR targets: container (#4283, unchanged here) and the SSH Activity endpoint gap.

You're right about the recipe. The v1 endpoint misses at_least_one: for an SSH record without endpoints, /api/1.8.0/validate reports only container, while /api/1.8.0/v2/validate also reports the constraint failure. I've switched the README to the versioned v2 endpoint and to checking error_count (unversioned /api/v2/validate validates against 1.9.0, so the version is pinned).

A rerun on this branch would help, especially for HTTP and Process Activity, which your capture didn't produce. The e2e gets HTTP Activity by running an L7 policy against a local server and sending a request the policy denies; plain TCP to a denied host only produces Network Activity.

[krishicks]

/ok to test 8dbc022

[krishicks]

A few remaining items found by the bot on re-review:

1. Duplicate tests that use the deleted field (crates/openshell-ocsf/src/validation/schema.rs:279 and :473): rejects_attributes_the_class_does_not_define and known_fields_rejects_attributes_outside_the_schema both inject is_src_dst_assignment_known into a base event and expect a rejection. Now that both validators share one walker, these run the same code. Please keep one, and use a neutral name such as "not_an_ocsf_attribute" so the test doesn't refer to a field we removed.
2. Docs describe a field we don't emit (docs/observability/ocsf-json-export.mdx:155): "HTTP Activity records omit is_src_dst_assignment_known…" reads like a changelog entry; it tells readers about something that isn't in the output. Please drop this bullet. The SSH dst_endpoint, Process actor and connection_info direction bullets are useful and should stay.
3. Test of str::replace (schema.rs:411): embedded_table_comparison_ignores_crlf_checkouts only tests with_lf_line_endings, which is a single replace("\r\n", "\n"). Please keep the normalization in the drift test but remove this test.
4. Drift check can skip without anyone noticing (schema.rs:426): when the source checkout is missing, embedded_schemas_match_vendored_files prints a message and passes. That makes sense for relocated nextest archives. But if the CI job that runs this test only uses archived binaries, the drift check never actually runs. Can you confirm it runs from a full checkout in at least one CI job?
5. Test comment doesn't match what it shows (schema.rs:342): in object_schemas_are_validated_as_objects_not_events, the comment says "The top-level container exemption applies to events only". The device half passes because device defines container, not because of the exemption. Only the user half (from :346) shows what the comment describes. Please either move the comment to the user half or drop the device half.

[zanetworker]

Thanks @krishicks, fixes addressed.

[krishicks]

/ok to test 58dd3e0

[krishicks]

This failed Windows MSVC lint/tests, but not because of any changes here; the build is broken. Fixed here: #4862

[zanetworker]

@krishicks any missing bits here? if not can we merge, as there are other issues/PR that are depending on this work being in.

[rhuss]

Second review round

All of the comments below are non-blocking. They come from an additional agent-assisted review round with updated review agents, run on the squashed head 1255f6d01. None of them blocks the merge; pick up whatever you find useful here or in a follow-up.

The failure scenarios in the comments were derived from reading the code and were not executed.

The points from my earlier review are settled. The downgrade and JSONL layer changes are no longer part of this PR, and the enum and type checks moved to #4596. The review agents also found krishicks' Oct 19 and Oct 21 items addressed in the current code.

What went well

• One recursive walker in validation/schema.rs now backs both validators, and its errors carry the full path (event.device.container), which makes producer test failures easy to locate.
• The type-state markers in builders/ssh.rs and builders/process.rs turn the required actor and the at_least_one endpoint rule into compile errors at every call site.
• ssh_server_endpoint and the Arc<Endpoint> threaded through the SSH handlers mean a shared supervisor names the right socket on each sandbox's events, and ssh_events_name_their_own_server_socket_and_conform_to_ocsf checks two sockets against the schema.
• The e2e only validates once Network, HTTP and SSH records are all present, so it can't pass on an empty capture, and supervisor_records tolerates a line caught mid-write.

Findings

Severity	File	Description
Minor	schemas/ocsf/README.md:30	connection_info.json is a 404 stub but is counted, fetched and embedded
Minor	builders/ssh.rs:139	Setters hand-copied from the shared macros; macro docs are stale
Minor	builders/process.rs:39	compile_fail doctests can pass for an unrelated compile error
Minor	ctx.rs:21	test-support now changes the fallback context at runtime
Minor	validation/schema.rs:90	validate_known_fields is a subset of validate_required_fields
Minor	validation/schema.rs:193	No negative test for array elements in the walker
Minor	ocsf-json-export.mdx:156	Example after the conformance claim lacks type_uid
Minor	ocsf_jsonl_conformance.rs:226	exec errors are dropped, so traffic failures are hard to triage

One more observation without an inline comment: class_schema_name in the e2e duplicates the uid that each vendored class schema already carries. A load_class_schema_by_uid helper would remove the table, but it's fine as is.

[zanetworker]

@rhuss can we address the minor comments after merging the PR? bots will always have a better solution for all sorts of things :)

[krishicks]

I have some ideas of how to rework this a bit that can incorporate @rhuss's feedback after this lands. @zanetworker, please rebase and fix the merge conflicts and I'll /ok to test it and get it merged.

[rhuss]

@zanetworker yes, sure! as mentioned above:

All of the comments below are non-blocking. They come from an additional agent-assisted review round with updated review agents, run on the squashed head 1255f6d. None of them blocks the merge; pick up whatever you find useful here or in a follow-up.

[krishicks]

/ok to test 6786fb5