Conversation
Build training samples from one materialized frame table with per-column frame or time offsets, two-sided padding and per-column masks. Add tensor and image transforms, adapt the ACT benchmark, and preserve legacy windows. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-6 Co-Authored-By: Codex <noreply@ai-tool.com> AI-Contributed/Feature: 643/643 AI-Contributed/UT: 365/365
Use importorskip for torch and Pillow, matching the other optional training suites. CI environments without Pillow can collect the remaining tests. Verified 44 related tests, collection without Pillow or torch, and flake8. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-6 Co-Authored-By: Codex <noreply@ai-tool.com> Co-Authored-By: Codex <noreply@openai.com> AI-Contributed/Feature: 0/0 AI-Contributed/UT: 6/6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Extend
ContiguousWindowDataset(CWD) into a training reader for a materialized, frame-level multimodal table. Previously, CWD used one forward window for all non-anchor columns. A sample containing state history, a current camera image, and future actions needs independent windows for those fields.The two Dataset APIs serve different data layouts:
They should converge on training capabilities while retaining these distinct data-source responsibilities. This PR focuses on CWD; PLD only reuses the image conversion helpers here.
This PR adds:
frame_offsetssupports history, future, sparse, and repeated positions, preserving their order. Unspecified selected columns default to[0]. For example,frame_offsets={"state": [-2, -1, 0], "action": range(16)}combines state history and future actions while a selected image column stays at the anchor.delta_timestampsconverts offsets using an explicitfpsand validates frame-grid alignment withtolerance_s.boundary="drop"|"pad"|"error"applies at both ends of each group. Padding repeats the nearest endpoint or uses per-column rawpad_values, with a<column>_is_padmask. Padding precedes transforms.Existing
window_size/anchor_columns/tailcalls retain their output shapes and globalis_padmask. New offset-based calls use per-column masks. Pillow is added to the optionaltorchextra for image conversion.This does not yet unify PLD's window API or add CWD video decoding. CWD continues to require contiguous integer frame positions; irregular-timestamp interpolation and automatic cross-table metadata/statistics discovery are outside this change.
Tests
152 tests passed across the affected CWD, transform, PLD, and ACT suites:
cd paimon-python python -m pytest -q \ pypaimon/tests/contiguous_window_dataset_test.py \ pypaimon/tests/window_transforms_test.py \ pypaimon/tests/multimodal_lerobot_test.py \ pypaimon/tests/act_runner_test.py \ pypaimon/tests/act_benchmark_test.pyCoverage includes offset conversion and validation, both group boundaries, per-column masks and projections, batch/slice semantics, mutable-value isolation, worker serialization, distributed sampling, legacy compatibility, and HDF5/Paimon ACT tensor and loss parity.
Flake8 with the repository configuration and
git diff --checkalso passed.