Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 57 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,47 @@ on:
branches: [main, develop]
pull_request:
branches: [main, develop]
workflow_call:
inputs:
prepare-publish:
description: Require main and an unpublished version before building
type: boolean
default: false

permissions:
contents: read

jobs:
build:
name: Build distributions
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Python 3.14
uses: actions/setup-python@v5
with:
python-version: '3.14'
- name: Check publish ref and PyPI version
if: inputs.prepare-publish
run: python scripts/publish_check.py
- name: Install UV CLI
run: python -m pip install uv==0.12.19
- name: Build wheel and source archive
run: uv build
- name: Validate metadata and README rendering
run: uvx twine check --strict dist/*
- name: Upload distributions for testing and publishing
uses: actions/upload-artifact@v4
with:
name: distributions
path: dist/
if-no-files-found: error

base-install:
name: Base wheel smoke (${{ matrix.os }}, Python 3.14)
runs-on: ${{ matrix.os }}
needs: build
strategy:
fail-fast: false
matrix:
Expand All @@ -26,8 +59,11 @@ jobs:
python-version: '3.14'
- name: Install UV CLI
run: python -m pip install uv==0.12.19
- name: Build wheel and source archive
run: uv build
- name: Download built distributions
uses: actions/download-artifact@v4
with:
name: distributions
path: dist
- name: Install fixture producer from lockfile
run: uv sync --locked --group dev --all-extras
- name: Produce bundle and check artifact contents
Expand Down Expand Up @@ -86,6 +122,7 @@ jobs:
tests:
name: Tests (${{ matrix.os }}, Python 3.14)
runs-on: ${{ matrix.os }}
needs: build
strategy:
fail-fast: false
matrix:
Expand All @@ -101,8 +138,25 @@ jobs:
run: python -m pip install uv==0.12.19
- name: Install development dependencies from lockfile
run: uv sync --locked --group dev --all-extras
- name: Download built distributions
uses: actions/download-artifact@v4
with:
name: distributions
path: dist
- name: Install the wheel that will be published
shell: bash
run: |
if [[ "$RUNNER_OS" == "Windows" ]]; then
test_python="$GITHUB_WORKSPACE/.venv/Scripts/python.exe"
else
test_python="$GITHUB_WORKSPACE/.venv/bin/python"
fi
uv pip install --python "$test_python" --no-deps --reinstall dist/*.whl
"$test_python" -I -c 'import mlbundle; from pathlib import Path; assert "site-packages" in Path(mlbundle.__file__).parts; print(mlbundle.__version__, mlbundle.__file__)'
- name: Run tests with coverage
run: uv run pytest
shell: bash
working-directory: ${{ runner.temp }}
run: uv run --project "$GITHUB_WORKSPACE" --no-sync pytest "$GITHUB_WORKSPACE/tests" -c "$GITHUB_WORKSPACE/pyproject.toml" --cov-config="$GITHUB_WORKSPACE/pyproject.toml" --cov-report="xml:$GITHUB_WORKSPACE/coverage.xml"
- name: Upload coverage report
uses: actions/upload-artifact@v4
with:
Expand Down
36 changes: 36 additions & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Publish to PyPI

on:
workflow_dispatch:

permissions:
contents: read

concurrency:
group: mlbundle-pypi-publish
cancel-in-progress: false

jobs:
build:
uses: ./.github/workflows/ci.yml
with:
prepare-publish: true

publish:
needs: build
runs-on: ubuntu-latest
environment:
name: pypi_env
url: https://pypi.org/p/mlbundle
permissions:
id-token: write
steps:
- name: Download CI-validated distributions
uses: actions/download-artifact@v4
with:
name: distributions
path: dist
- name: Publish to PyPI with Trusted Publishing
uses: pypa/gh-action-pypi-publish@release/v1
with:
packages-dir: dist/
16 changes: 8 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,8 @@ still preserve validation, security and reproducibility.
or PR text.
- `C:\software\mlsuite` and `C:\software\mlschema` are read-only references.
Never modify, commit or run write commands inside either checkout.
- Do not publish to PyPI, create releases or version tags, or add a publish job.
"Tags" in this project means GitHub repository topics.
- Publish only via manual `publish.yml` from `main`, through `pypi_env` with owner approval, on the owner's explicit request for that specific version.
Agents never run or approve it on their own initiative or add publish paths/tokens/secrets. No GitHub releases/version tags; "tags" means GitHub repository topics.
- sklearn-to-ONNX conversion, per-model environments, bundle signing and additional
libraries (including LightGBM and PyTorch) are outside the project scope.
- Never claim a verification that was not run; report its real output or blocker.
Expand All @@ -154,8 +154,8 @@ uv run pre-commit install
```

After changing dependency metadata, use `uv sync` to regenerate `uv.lock` and
commit the lockfile. CI uses `uv sync --locked --group dev --all-extras` on Python 3.14.
The default dev group installs `mlbundle[all]`; base users can omit the dev group.
commit the lockfile. CI and manual `publish.yml` use locked dev/all extras on 3.14.
Default dev installs `mlbundle[all]`; base users can omit it. Publishing reuses CI.
Ruff handles lint and formatting; Pyright uses the family's basic mode.

## Verifying
Expand All @@ -174,8 +174,8 @@ The Ruff commands also work without the final `.` because the default is the
current directory: `uv run ruff check` and `uv run ruff format --check`.
`uv run pytest` runs `tests/`, measures branch coverage and enforces the family's
80% minimum, writing terminal and XML reports. Tests use the installed package;
its version comes from package metadata. CI checks lint and types and runs tests
and a clean base-wheel smoke check on Ubuntu and Windows. The local build must produce a wheel and source archive.
its version comes from package metadata. CI checks lint/types and the built wheel
on Ubuntu/Windows; manual `publish.yml` requires that CI. Build a wheel and sdist.

Start with the smallest proof of a behavior change, then broaden when needed.
If a check cannot run, report the exact command and blocker. Never claim a test,
Expand Down Expand Up @@ -285,9 +285,9 @@ its kind/count inference remains ONNX-owned. Keep native behavior in adapters; f
- `src/mlbundle/inspection.py`, `results.py` — metadata-only inspection and result records without adapter resolution.
- `docs/format.md` — version 1 wire specification; linked container, adapter/structure, ONNX and schema/sample sections are normative.
- `tests/` — one file per feature, with success and error cases together.
- `scripts/package_smoke.py` — fixture, isolated install and artifact-content checks.
- `scripts/package_smoke.py`, `publish_check.py` — artifact checks and publish preflight.
- `pyproject.toml` and `uv.lock` — package metadata, tool configuration and lock.
- `.github/workflows/ci.yml` — lint, types and cross-platform tests.
- `.github/workflows/ci.yml`, `publish.yml` — shared CI and owner-approved publishing.
- `.pre-commit-config.yaml` — Ruff, Pyright and basic file hygiene.
- `README.md`, `SECURITY.md` and this file — project and contributor context.

Expand Down
46 changes: 24 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,15 @@ mlbundle replaces three gaps in separate model/DataFrame uploads:
- **Training data leaving the machine:** schema inference runs locally;
training rows are included only when explicitly requested as samples.

**Pre-release v0.1.0; format version 1 is not yet frozen.** Nothing is published
on PyPI. Requires **Python >=3.14**. The tested baseline is CPython 3.14 on
Windows and Linux; the schema extra currently works only on Python 3.14.
**Pre-alpha v0.1.0a1, published to reserve the PyPI name.** Format version 1 is
not frozen; bundles may need re-export. The security design has not had an
independent review: do not accept bundles from untrusted parties.
Requires **Python >=3.14**; tested on Windows/Linux. Schema requires Python 3.14.

## Install and try scikit-learn

Install from Git with
`python -m pip install "mlbundle[sklearn] @ git+https://github.com/UlloaSP/mlbundle.git@develop"`.
Install the pre-release with
`python -m pip install --pre "mlbundle[sklearn]"`.
Replace `[sklearn]` with `[xgboost]`, `[catboost]`, `[onnx]`, `[schema]` or `[all]` for the
corresponding dependencies. Combine extras as `[sklearn,schema]` for local
DataFrame inference. Omit the brackets for a base install that can inspect
Expand Down Expand Up @@ -99,7 +100,7 @@ assert bundle.predict_proba(rows) == model.predict_proba(rows).tolist()
Depthwise/Lossguide, custom Python objectives/metrics and other losses are rejected.
Classifier losses: Logloss, CrossEntropy, MultiClass. Regressors: RMSE, MAE,
Quantile, Huber, Poisson, MAPE. Complete trees/borders/NaN handling/importances
are verified on load. [The CatBoost contract](docs/format-catboost.md) details
are verified on load. [The CatBoost contract](https://github.com/UlloaSP/mlbundle/blob/main/docs/format-catboost.md) details
wrapper recovery, native-parser limits and the private in-memory serializer.

## Composite pipelines
Expand Down Expand Up @@ -132,7 +133,7 @@ Each needs a fitted, data-free width proof. Unknown widths, FeatureUnion/KNNImpu
slice/callable/scalar selectors, caching, metadata routing and other native placements
remain rejected. Step names/outer options are declarations; trees address transformed
features, without original-feature attribution. Schema/dtypes/samples describe original
inputs; six members permit schema plus explicit samples. See [the pipeline contract](docs/format-pipeline.md).
inputs; six members permit schema plus explicit samples. See [the pipeline contract](https://github.com/UlloaSP/mlbundle/blob/main/docs/format-pipeline.md).

## Existing ONNX files

Expand All @@ -157,7 +158,7 @@ Paths and bytes are preserved unchanged; a ModelProto is serialized once.
There is no conversion. `native_model` contains the original `data` and CPU
`session`. Native graph labels are verified; supplied labels must agree with
them. Without native/supplied labels, encoded indices are generated. ONNX has
no visualization structure. See [the exact ONNX contract](docs/format-onnx.md).
no visualization structure. See [the exact ONNX contract](https://github.com/UlloaSP/mlbundle/blob/main/docs/format-onnx.md).

## Local schema and explicit samples

Expand Down Expand Up @@ -199,7 +200,7 @@ steps, categories (including unused), one-hot suffixes, pair-axis metadata and
class labels. No inferred ranges, means, counts or arbitrary text/example rows.
Names/categories and learned models/structure can reveal training information.
Explicit `sample_rows=N` includes the first N feature rows, without target/index;
N must be positive and within the frame. See [schema and samples](docs/format-schema.md).
N must be positive and within the frame. See [schema and samples](https://github.com/UlloaSP/mlbundle/blob/main/docs/format-schema.md).

## Inspect, load and the CLI

Expand Down Expand Up @@ -229,9 +230,9 @@ dependency rejection (`MLBundleError`), **2** usage/I/O failure.
dense brute-force neighbors, NearestCentroid, MLP and MultiOutput/RegressorChain.
SimpleImputer/KNNImputer, scalers, KBinsDiscretizer, categorical encoders,
univariate selectors and closed NumPy FunctionTransformer functions round-trip,
including nested ColumnTransformer pipelines; see [the exact lists](docs/format-state.md).
including nested ColumnTransformer pipelines; see [the exact lists](https://github.com/UlloaSP/mlbundle/blob/main/docs/format-state.md).
All-sklearn Pipeline structure stays null; composite trees expose the inner boundary. See
[tested configurations](tests/test_sklearn_families.py) and [core round trips/rejections](tests/test_sklearn.py).
[tested configurations](https://github.com/UlloaSP/mlbundle/blob/main/tests/test_sklearn_families.py) and [core round trips/rejections](https://github.com/UlloaSP/mlbundle/blob/main/tests/test_sklearn.py).
- **Sklearn rejections:** custom estimators/subclasses, functions/types outside the
closed lists (including lambdas and user functions), compound fitted dtypes, classifiers without
probabilities, multilabel/ClassifierChain layouts, unsupported native placements,
Expand All @@ -240,22 +241,22 @@ dependency rejection (`MLBundleError`), **2** usage/I/O failure.
- **Sklearn versions:** the closed class/attribute catalog is audited/tested
against **1.9.1 with skops 0.16.0**, not every version allowed by `>=1.6`.
Unknown state fails; export/load sklearn versions must match. Configurations
must satisfy the catalog and round trip; see [state ownership](docs/format-state.md).
must satisfy the catalog and round trip; see [state ownership](https://github.com/UlloaSP/mlbundle/blob/main/docs/format-state.md).
- **XGBoost:** exact XGBClassifier/XGBRegressor, one target, native gbtree
(including XGBoost 3.4.1 dart normalized to gbtree). Closed probability/regression
objectives; raw boosters, RF wrappers, subclasses, ranking/survival/custom
objectives and unsupported storage are rejected. At most 100,000 trees/depth 14,
with complete state/dumps; see [the mapping](docs/format-adapters.md#implemented-xgboost-mapping).
with complete state/dumps; see [the mapping](https://github.com/UlloaSP/mlbundle/blob/main/docs/format-adapters.md#implemented-xgboost-mapping).
- **ONNX:** fixed-width rank-two numeric inputs; one regression/probability target,
optional native labels and dynamic/one-row batches. Unsupported layouts,
external data/custom domains/functions and runtime/training configuration fail
before CPU sessions. Tested with ONNX 1.23.0/Runtime 1.30.0; see [ONNX bounds](docs/format-onnx.md).
before CPU sessions. Tested with ONNX 1.23.0/Runtime 1.30.0; see [ONNX bounds](https://github.com/UlloaSP/mlbundle/blob/main/docs/format-onnx.md).

Archive ceilings: **6 members**, **512 MiB** archive/compressed/member bytes,
**1 GiB** aggregate uncompressed bytes, ratio **200**, JSON depth **32**.
Manifest: **8 MiB**; structure: **256 MiB**; schema/samples: **16 MiB each**.
Features/classes: at most **100,000**; outputs: **1,024** (XGBoost/CatBoost/ONNX: one).
Callers may only lower `ArchiveLimits`. Native [state](docs/format-state.md)/[ONNX](docs/format-onnx.md)
Callers may only lower `ArchiveLimits`. Native [state](https://github.com/UlloaSP/mlbundle/blob/main/docs/format-state.md)/[ONNX](https://github.com/UlloaSP/mlbundle/blob/main/docs/format-onnx.md)
bounds do not cap process memory, prediction rows or CPU.

## Bundle contents and security
Expand All @@ -267,15 +268,15 @@ Every bundle contains `manifest.json` and one native member, or the declared
The manifest records identity/time/writer, model parameters/counts/state,
ordered features/dtypes/origin, target labels/name, environment, member
SHA-256/sizes, references and free-form user metadata. The normative
[format specification](docs/format.md) owns these records and verification rules.
[format specification](https://github.com/UlloaSP/mlbundle/blob/main/docs/format.md) owns these records and verification rules.

Treat bundles as untrusted. The reader validates ZIP layout, allowed members,
limits, strict JSON/manifest and hashes; adapters use closed trust/loading paths
and recompute claims. **Hashes detect corruption, not deliberate tampering**;
there is no signing or proof of authorship, training history, quality or fairness.
Schema, samples and training-environment declarations are not authenticated.
Native parsers/inference require process isolation, memory/CPU/row limits and
maintained dependencies. See [SECURITY.md](SECURITY.md) for guarantees and reporting.
maintained dependencies. See [SECURITY.md](https://github.com/UlloaSP/mlbundle/blob/main/SECURITY.md) for guarantees and reporting.

## Development

Expand All @@ -288,8 +289,9 @@ versions; dependency edits require `uv sync` and committing the updated lockfile
Run `uv run ruff check .`, `uv run ruff format --check .`, `uv run pyright`,
`uv run pytest` and `uv build`. Tests execute every fenced README example and
write terminal/XML coverage with an 80% branch-inclusive minimum.
CI preserves lint/types and Ubuntu/Windows tests and checks a clean base wheel
installation on both OSes. `uv build` creates a local wheel and source archive.
Read [AGENTS.md](AGENTS.md) before contributing. Publishing, releases, version
tags, conversions, per-model environments, signing and additional model libraries
remain outside scope. The project is MIT licensed.
CI checks lint/types and tests the same built wheel on Ubuntu/Windows, including
clean base installs. The owner, or an agent on the owner's explicit request for
that version, runs manual [publishing](https://github.com/UlloaSP/mlbundle/blob/main/docs/publishing.md) from `main` with `pypi_env` approval. Read [AGENTS.md](https://github.com/UlloaSP/mlbundle/blob/main/AGENTS.md).
Agents never run or approve it on their own initiative or add publish paths, tokens or secrets.
Releases, version tags, conversions, per-model environments, signing and additional
model libraries remain outside scope. The project is MIT licensed.
5 changes: 3 additions & 2 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# Security policy

mlbundle v0.1 is pre-release; format version 1 is not yet frozen. It implements
strict manifest validation, secure container reading/writing, metadata-only
The published v0.1.0a1 pre-alpha has not had an independent security review;
do not accept bundles from untrusted parties. Format version 1 is not frozen.
It implements strict manifest validation, secure container reading/writing, metadata-only
`inspect`, export/load, sklearn, XGBoost, CatBoost and ONNX adapters, local schema inference,
explicit samples, prediction-state verification and the inspect/verify CLI.
The container validates ZIP/ZIP64 layout, bounds sources and actual decompression,
Expand Down
Loading
Loading