python(feat): add export api for sift client - #490

Merged

wei-qlu merged 60 commits into

mainfrom

python/data-export-high-level-interface

Mar 24, 2026

wei-qlu commented Mar 10, 2026 •

edited

Loading

Contributor

What was changed

Adds a high-level data export API to sift_client to reach parity with sift_py (soon to be deprecated).

DataExportAPIAsync with a single export() method that accepts domain objects or raw string IDs for runs, assets, or time-range-based exports and returns a Job handle
Job.wait_and_download() polls until complete, downloads results, and optionally extracts zip files (delegates to JobsAPIAsync)
ExportsLowLevelClient wraps gRPC protobuf calls, the high-level client has zero proto imports
Utility helpers added under _internal/util/:
- channels.py - resolve_calculated_channels() resolves name-based channel identifiers to UUIDs via the channels API (the export API requires UUIDs)
- file.py - download_file() streams downloads in 4 MiB chunks; extract_zip() extracts and cleans up archives
- executor.py - run_sync_function() runs blocking calls off the event loop via run_in_executor
Sync wrappers and type stubs registered. Wired up as client.data_export and client.async_.data_export

Verification

Unit tests for input validation, domain object resolution, low-level delegation, wait/download status handling, and calculated channel resolution
Integration tests for async and sync export paths

wei-qlu changed the title ~~Python/data export high level interface~~ python(feat): add export api for sift client

wei-qlu requested review from alexluck-sift, marc-sift, nathan-sift and solidiquis

March 12, 2026 19:18

wei-qlu marked this pull request as ready for review

March 12, 2026 21:19

nathan-sift requested changes

View reviewed changes

python/lib/sift_client/resources/exports.py Outdated

python/lib/sift_client/resources/exports.py Outdated

python/lib/sift_client/sift_types/export.py

python/lib/sift_client/sift_types/export.py Outdated

python/lib/sift_client/sift_types/export.py Outdated

python/lib/sift_client/resources/exports.py Outdated

python/lib/sift_client/resources/exports.py

python/lib/sift_client/resources/exports.py Outdated

python/lib/sift_client/resources/exports.py Outdated

python/lib/sift_client/resources/exports.py Outdated

wei-qlu requested a review from nathan-sift

March 16, 2026 19:52

wei-qlu marked this pull request as draft

March 17, 2026 00:39

nathan-sift previously approved these changes

View reviewed changes

nathan-sift left a comment

Contributor

This looks good, thanks for the changes! Approving, but it looks like you might need to re-update the stubs

wei-qlu dismissed nathan-sift’s stale review via

March 17, 2026 20:57

wei-qlu marked this pull request as ready for review

March 17, 2026 22:49

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/_internal/_grpc/_async_interceptors/__init__.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/client.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/resources/exports.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/resources/exports.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/resources/exports.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/resources/exports.py Outdated

alexluck-sift requested changes

View reviewed changes

python/lib/sift_client/resources/exports.py Outdated

python/lib/sift_client/resources/exports.py Outdated

python/lib/sift_client/resources/exports.py Outdated

python/lib/sift_client/resources/exports.py Outdated

python/lib/sift_client/resources/exports.py Outdated

python/lib/sift_client/sift_types/export.py

python/lib/sift_client/_internal/low_level_wrappers/exports.py Outdated

python/lib/sift_client/resources/exports.py

python/lib/sift_client/_tests/resources/test_exports.py Outdated

python/lib/sift_client/_tests/resources/test_exports.py

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/resources/exports.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/_internal/util/download.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/resources/exports.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/_tests/resources/test_exports.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/_tests/resources/test_exports.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/_tests/resources/test_exports.py Outdated

wei-qlu force-pushed the python/data-export-high-level-interface branch from edc48f6 to 920e5ca Compare

March 23, 2026 18:22

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/_internal/util/file.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/_internal/util/file.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/_tests/_internal/test_sync_wrapper.py Outdated

alexluck-sift reviewed

View reviewed changes

python/lib/sift_client/sift_types/job.py Outdated

wei-qlu added 22 commits

March 23, 2026 15:22


          pyright fix

cc2ab5b


          move _resolve_calculated_channels and download_and_extract_zip to _in…

763e40f

…ternal/util


          add extract parameter to wait_and_download can keep the zip without e…

fe6d4fe

…xtracting


          refactored export methods to a single entry point

d15f661


          mypy fix

722d61a


          python(refactor): use rest_client instead of raw request

58dc781


          python(refactor): added dict support for calc channels

2b4e84f


          python(refactor): class rename to DataExportsAPI

cab9888


          mypy fix

5ea51ea


          pyright fix

d724ebc


          timeout increased

b7705ed


          python(refactor): split up the download_and_extract function and rena…

8aebbc5

…med the file to be more appropriate


          python(fix): scoped the integration export jobs to 10s

63acd04


          python(fix): add sync wrapper tests for run_in_executer loop scenarios

7e0fbba


          python(refactor): add a run_sync_function util for running blocking c…

058fcaa

…alls off the event loop


          python(refactor): move wait_and_download to JobsAPI and add the metho…

7bbbda0

…d on the job object itself


          python(fix): scoped the export_by_asset test to use one channel

80cac44


          python(fix): updated export_by_asset integration tests with ingested …

25ae348

…test data


          python(refactor): removed redundant tests, following existing test st…

eb9f9c5

…ructure


          python(refactor): renamed arguments to be more descriptive, added try…

1a26784

…/catch for zip


          python(refactor): use run_sync_function util instead of reimplementin…

eee4b52

…g in test_sync_wrapper


          python(refactor): allow wait_and_download to handle non-zip files

9a17655

wei-qlu force-pushed the python/data-export-high-level-interface branch from 34722e0 to 9a17655 Compare

March 23, 2026 22:23

wei-qlu added 3 commits

March 23, 2026 15:29


          python(refactor): moved job parameter to a positional argument

040b5c8


          python(refactor): relocate tests to proper scoping

e971623


          python(refactor): use ingestionAPI instead of raw HTTP in export test…

3c92f8e

… fixture

alexluck-sift self-requested a review

March 24, 2026 00:06

alexluck-sift approved these changes

View reviewed changes

alexluck-sift left a comment

Collaborator

lgtm! Nice work!

wei-qlu merged commit c47512b into main

22 of 24 checks passed

wei-qlu deleted the python/data-export-high-level-interface branch

March 24, 2026 00:13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet