Skip to content

feat: verify single-file fMP4 fragments at their byte offset - #3

Open
mstattma wants to merge 10 commits into
integration/upstream-base-192023c3from
feat/upstream-fragment-at-offset
Open

mstattma wants to merge 10 commits into
integration/upstream-base-192023c3from
feat/upstream-fragment-at-offset

Conversation

@mstattma

@mstattma mstattma commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Fork-first preparation, not yet submitted to ContentAuth. Based on upstream
Python 192023c3 (integration/upstream-base-192023c3). Depends on
contentauth/c2pa-rs#2832 and the C FFI in mstattma/c2pa-rs#36.

Feature

  • Optional bindings for c2pa_reader_with_fragment_at_offset and its capability
    probe; older supported native libraries still import.
  • has_fragment_at_offset(), Reader.with_fragment_at_offset(...), and
    Reader.from_fragment_at_offset(..., context=None).
  • Offset/format/stream validation happens before new native allocations;
    rejected arguments leave existing readers unchanged.
  • The existing fragment lock, stream retention, cache invalidation, and
    consuming-reader ownership machinery are shared with with_fragment.
  • With a valid init and parseable fragment, hash failures are validation
    mismatches. Loading/parsing errors still raise. Original-file offset zero
    preserves separate-file behavior.

GitHub-Only Qualification

Final Python head: 6adcc40861c9c91d7cd7802686e508e1ca30491b.

Qualification run 39018483548
is green on Ubuntu 24.04 and Windows 2022, using Python 3.12 and Rust 1.96.0.
No local Cargo build was used for this qualification.

The non-publishing, fork-only workflow builds native release libraries from
exact source mstattma/c2pa-rs@20d7c45972976c32ab24d5d4ac33e7be826ce1be
(SDK 0.92.0-dev, ZIP remediation included). It checks loaded-library path,
binary digest, Python source provenance, and SDK version before running tests.

Final PR-source result Linux Windows
Candidate suites, top-level tests 569 passed 568 passed, 1 platform skip
Required focused offset identities 18 passed, no skips 18 passed, no skips
Additional passing subtests 82 82
Stock 0.91 compatibility PASS PASS
No new regressions against same-binary baselines PASS PASS

The Windows platform skip is the real-os.fork test on an OS without
os.fork, not any offset test. Stock-native focused coverage requires the
11 binding tests to pass and exactly seven unsupported-native skips.

The previously failing builder tests assumed actions shared one assertion and
ingredients retained insertion order. Upstream Rust #2720 deliberately changed
those layouts. The tests now retain v2 checks and assert exact required actions,
parameters, parent/component identity, metadata, and opened-parent references
without depending on array order. Both pass on stock 0.91 and candidate 0.92.
Unmodified-baseline failures remain visible in raw reports rather than being
relabeled as passes.

The qualification gate pins required test identities, rejects missing tests
and new skips, compares failure identities/types/messages, and independently
requires every native offset test to run. Nineteen lightweight helper tests
include counterexamples for constant-count test replacement and hidden stock
skips. Review-agent findings were addressed and re-reviewed.

Evidence

Actions artifacts retain native binaries, generated lockfile, toolchain/source
identities, loaded-library records, raw JUnit, logs, and comparison summaries
for 14 days. Their archive digests and retained native binaries were independently
rehash-verified after download; the artifacts were not executed locally.

  • Linux artifact: 12096510002.
  • Windows artifact: 12096206301.
  • Resolved native lock SHA-256: 9e38eba78f67c2309bab511e1144e273cc4d9bb4611ce2f9fd7836b8dd2c5d58.

These remain preparation artifacts, not release wheels or publication approval.
The fork-only workflow/helper commits are separate from the feature commits.
Before upstream release, select an accepted native dependency containing the
Rust and C APIs and repeat qualification without relying on fork candidates.

Bind c2pa_reader_with_fragment_at_offset and its probe
c2pa_reader_fragment_at_offset_supported as optional native symbols, so
libraries without them still import.

- has_fragment_at_offset(): True when both symbols exist and the probe
  reports support. The probe runs on first use, not at import.
- Reader.with_fragment_at_offset(format, stream, fragment_stream,
  fragment_offset) and Reader.from_fragment_at_offset(..., context=None).
  with_fragment and with_fragment_at_offset share one helper, so the
  offset variant gets with_fragment's locking, stream retention, cache
  invalidation and consume/swap handling unchanged.
- Unsupported libraries raise C2paError.NotSupported, and offsets that are
  not an int in 0..2**64-1 (bool included) raise TypeError/ValueError,
  before any native call; the reader is left untouched.
- A wrong cut or offset is reported as assertion.bmffHash.mismatch in the
  validation results, not raised.

Tests: feature detection and validation run against any native library;
native tests use the signed DASH fixtures (offset 0 matches with_fragment,
other offsets are a mismatch) and are skipped without the native API,
or fail under C2PA_FRAGMENT_OFFSET_REQUIRED=1.
@mstattma

mstattma commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner Author

Implemented and pushed 72e7b37. Automatic PR qualification completed: https://github.com/mstattma/c2pa-python/actions/runs/39011093245 (overall FAIL, intentionally not blanket green).

Verdict Ubuntu 24.04 Windows 2022
Native release build, pinned SHA / Rust 1.96.0 PASS PASS
Focused offsets, exactly 18 passes / zero skips PASS PASS
No new regressions vs base Python, identical candidate binary PASS PASS
Candidate full suite FAIL: 2 baseline failures FAIL: 3 baseline failures
Stock 0.91 compatibility / combined gate PASS FAIL: existing fork-only test

Identical baseline/candidate failures:

  • tests.test_unit_tests.TestBuilderWithSigner::test_builder_minimal_manifest_add_actions_and_sign_with_auto_add: builtins.AssertionError, False is not true.
  • tests.test_unit_tests.TestBuilderWithSigner::test_builder_sign_with_ingredients_edit_intent: builtins.KeyError, missing title.
  • Windows only, also stock: tests.test_unit_tests_threaded.TestForkedChildDoesNotDeadlock::test_parent_copy_unaffected: builtins.AssertionError; subprocess fails because Windows has no os.fork.

Exact messages/types/IDs and raw JUnit are retained with native binaries/hashes, resolved lock and source/toolchain/features evidence for 14 days: Linux artifact, Windows artifact. No failure-hiding skips were added. Local validation: 12 stdlib tests plus YAML/Python syntax checks. No local Cargo commands, workflow dispatch, release, merge or default-branch changes.

@mstattma

Copy link
Copy Markdown
Owner Author

Follow-up a8d85a4 adds a per-test requires-os.fork guard only to TestForkedChildDoesNotDeadlock::test_parent_copy_unaffected; Unix assertions and native offset coverage are unchanged. Comparison accepts only this exact Windows-without-fork baseline failure -> exact applicability skip, and records it in comparison.json plus the summary. Passing baselines, other IDs/errors/reasons and other platforms remain protected.

Automatic qualification completed successfully on Ubuntu 24.04 and Windows 2022: https://github.com/mstattma/c2pa-python/actions/runs/39012252478. Both have stock compatibility PASS, no-new-regressions PASS and 18 focused offsets PASS with zero skips. Applicability adjustments: Linux 0, Windows exactly 1. Candidate full-suite status remains FAIL for the same two raw builder failures (AssertionError: False is not true; KeyError for title), preserved alongside baseline results.

Local verification: 15 stdlib helper tests passed; no local native/Cargo commands. Evidence retained for 14 days: Linux, Windows.

@mstattma

Copy link
Copy Markdown
Owner Author

Review P2 fixes pushed in 68d838f. Focused qualification now pins the 18 full raw-JUnit IDs, permits extra passing tests but rejects replacements/missing IDs and every extra failure/skip. Stock focused checks require all 11 binding passes and the seven exact native unsupported skips (type and reason).

Added unmodified base Python + the identical stock 0.91 binary as a fourth lane. Stock and native comparisons detect missing tests/new skips; all PR stock suites must still exit zero. Only the exact known Windows fork failure may exempt a failing stock baseline, recorded separately with stock/native contexts. Both stock lanes clear C2PA_PREFLIGHT_RUN.

Automatic Linux/Windows qualification passed: https://github.com/mstattma/c2pa-python/actions/runs/39015210336. Candidate full status remains FAIL for the same two raw builder failures. Windows retains the expected stock-baseline threaded exit 1 and two contextual fork applicability records; PR stock exits zero. Evidence: Linux, Windows.

Verification: 19 stdlib tests, including counterproofs for replacing one ID while keeping count 18 and a stock pass-to-skip change with exit zero. Both mutations were also rejected against actual retained CI JUnit. No local Cargo/native builds, release/default-branch/secret changes, or dispatch.

@mstattma

Copy link
Copy Markdown
Owner Author

Separate test-only fix ebf6fcc updates only the two requested methods in tests/test_unit_tests.py, following deliberate created/gathered ordering from contentauth/c2pa-rs#2720 (commit 309b99abe81b88acb24da28ea0558dae83b768d6).

The auto-action test now searches all action assertions and requires one c2pa.created inception with the exact source type at index zero of its own assertion, plus one brightnesscontrast color adjustment. The EDIT test selects exactly one parentOf and one componentOf by relationship, preserves their existing field checks, and verifies opened-action references resolve to the parent label. Signing inputs are unchanged; no created:true workaround, skips, version gates or default-value masking were added.

Automatic qualification passed on both Ubuntu 24.04 and Windows 2022, including candidate full-suite PASS: https://github.com/mstattma/c2pa-python/actions/runs/39016909576. Both changed tests also pass stock 0.91. The untouched baseline SHA 192023c still records the original two native unit failures in raw JUnit; no qualification policy changed. Evidence: Linux, Windows.

Local verification was stdlib-only: 19 helper tests, AST scope/input checks, and synthetic JSON assertions for both layouts and negative cases. No local Cargo/native tests or default-branch/release/pin changes. Ready for parent review.

@mstattma

Copy link
Copy Markdown
Owner Author

Final-head GitHub qualification is green on Linux and Windows: https://github.com/mstattma/c2pa-python/actions/runs/39018483548 at 6adcc40861c9c91d7cd7802686e508e1ca30491b. Candidate full suites, stock compatibility, exact focused-ID/no-skip gates, and baseline comparisons all pass. All 18 required offset tests execute on both platforms. The two old builder failures were stale ordering assumptions from upstream Rust #2720; tests now retain v2 checks and assert required actions/relationships rather than positions, without changing signing inputs. Evidence artifacts and raw unchanged-baseline failures remain linked in the PR body. No local Cargo builds or releases were performed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant