Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,11 +218,11 @@ claims. `native_model` is an OnnxModel with original bytes and a CPU session.
Kind is inferred from supported outputs; native labels are verified, absent
labels use count-checked declarations. Structure is null. See the normative
ONNX section linked from `docs/format.md` for exact layouts and resource bounds.
CatBoost supports exact numeric single-target symmetric classifiers/regressors, with native state commitments and in-memory loading. Sklearn pipelines containing XGBoost/CatBoost estimators are rejected; export the estimator directly.
The closed member list is: required `manifest.json`, exactly
one of `model.skops`, `model.ubj`, `model.cbm` or `model.onnx`, optional `schema.json`, and
CatBoost supports exact numeric single-target symmetric classifiers/regressors, with native state commitments and in-memory loading. Exact outer sklearn Pipelines ending in XGBoost/CatBoost use separately checked skops/native components; other native placements remain rejected.
The closed member list is: required `manifest.json`, one
native member, or declared `model.skops` + `model.ubj`/`model.cbm` pair, optional `schema.json`, and
`structure.json` when the supported model has extractable structure. Optional
`samples.json` requires explicit user intent. No other members are allowed.
`samples.json` requires explicit user intent. The ceiling is six members; no others are allowed.

The manifest must record format name/version, bundle id, UTC creation time and
writer version; model library, estimator class, classifier/regressor kind,
Expand Down Expand Up @@ -261,11 +261,12 @@ its kind/count inference remains ONNX-owned. Keep native behavior in adapters; f
- `src/mlbundle/contract.py` — closed identifiers and native member/format pairing.
- `src/mlbundle/limits.py` — shared archive ceilings and specific member caps.
- `src/mlbundle/canonical.py` and `_canonical_work.py` — deterministic JSON and bounded decoding/expansion.
- `src/mlbundle/model_io.py` — library-independent public export/load, manifest construction and round-trip verification.
- `src/mlbundle/model_io.py`, `_native_io.py`, `_export_manifest.py` — public I/O, native round trips and manifest construction.
- `src/mlbundle/_composite/`, `_pipeline_contract.py` — strict recipe, separate I/O, closed reconstruction/claims/prediction and input/inner boundaries. Read `docs/format-pipeline.md` when changing composites.
- `src/mlbundle/training.py` — lazy local DataFrame alignment and schema inference.
- `src/mlbundle/schema.py` and `_schema_fields.py` — closed schema/report declarations and reader cross-checks without mlschema.
- `src/mlbundle/samples.py` — explicit feature-row selection and bounded cell conversion.
- `src/mlbundle/_sklearn/` — closed dtype/type/function trust, skops loading, MLP state, parameter/claim extraction, native structure, nested payload bounds and fitted categorical prediction.
- `src/mlbundle/_sklearn/` — owned trust/skops/state/parameters/structure, transformer route and closed fitted-width/name proofs. Composite commitments/native structure bind the rebuilt Pipeline; all-sklearn structure stays null.
- `src/mlbundle/_xgboost/` — closed wrappers/objectives, native UBJSON preflight/loading, label binding, recoverable config, complete tree structure and validated prediction.
- `src/mlbundle/_catboost/` — closed numeric symmetric wrappers/losses, CBM preflight/in-memory loading, complete native state/structure and validated prediction.
- `src/mlbundle/_onnx/` — bounded protobuf/graph validation, unchanged bytes, native signatures/labels, CPU sessions and tabular prediction.
Expand All @@ -276,13 +277,12 @@ its kind/count inference remains ONNX-owned. Keep native behavior in adapters; f
- `src/mlbundle/environment.py` — training declarations and distribution metadata capture.
- `src/mlbundle/adapters.py` and `_prediction_input.py` — adapter/predictor protocols, output types and shared input checks.
- `src/mlbundle/registry.py` — closed lazy resolution of `_sklearn`, `_xgboost`, `_catboost` and `_onnx` adapters.
- `src/mlbundle/results.py` — metadata-only and loaded reader result types.
- `src/mlbundle/_container_rules.py` — shared source type, closed member sets/order and ceilings.
- `src/mlbundle/_zip_directory.py` — bounded ZIP/ZIP64 directory and extra-field validation.
- `src/mlbundle/_zip_members.py` — local headers/descriptors and bounded decompression with CRC.
- `src/mlbundle/container.py` — source bounds, validated in-memory reading and integrity checks.
- `src/mlbundle/writing.py` — integrity ownership, deterministic ZIP and atomic publication.
- `src/mlbundle/inspection.py` — public metadata-only inspection without adapter resolution.
- `src/mlbundle/inspection.py`, `results.py` — metadata-only inspection and result records without adapter resolution.
- `docs/format.md` — version 1 wire specification; linked container, adapter/structure, ONNX and schema/sample sections are normative.
- `tests/` — one file per feature, with success and error cases together.
- `scripts/package_smoke.py` — fixture, isolated install and artifact-content checks.
Expand Down
127 changes: 61 additions & 66 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,8 @@ from sklearn.tree import DecisionTreeClassifier
from mlbundle import export, inspect, load

rows = [[0.0, 1.0], [1.0, 0.0], [0.1, 0.9], [0.9, 0.1]]
model = DecisionTreeClassifier(random_state=7).fit(
rows, ["cold", "warm", "cold", "warm"]
)
model = DecisionTreeClassifier(random_state=7).fit(rows, ["cold", "warm"] * 2)
export(model, "trained.mlbundle", feature_names=["temperature", "humidity"])

bundle = load("trained.mlbundle")
assert isinstance(bundle.native_model, DecisionTreeClassifier)
assert bundle.native_model.predict(rows).tolist() == model.predict(rows).tolist()
Expand Down Expand Up @@ -63,23 +60,18 @@ from mlbundle import export, load

rows = [[0.0, 1.0], [1.0, 0.0], [0.1, 0.9], [0.9, 0.1]]
model = XGBClassifier(n_estimators=3, max_depth=2, n_jobs=1).fit(rows, [0, 1, 0, 1])
export(
model,
"xgboost.mlbundle",
feature_names=["temperature", "humidity"],
class_labels=[["cold", "warm"]],
)
export(model, "xgboost.mlbundle", class_labels=[["cold", "warm"]])
bundle = load("xgboost.mlbundle")
assert isinstance(bundle.native_model, XGBClassifier)
assert bundle.native_model.predict(rows).tolist() == model.predict(rows).tolist()
assert bundle.predict_proba(rows) == model.predict_proba(rows).tolist()
assert bundle.class_labels == [["cold", "warm"]]
```

The adapter binds labels/settings to a native **copy**, leaving your model
unchanged. Load verifies that binding, not historical truth. Omitted labels use
encoded indices or an existing binding; native `predict` still returns indices.
Sklearn, CatBoost and regressors reject `class_labels=`.
Labels/settings bind to a native **copy**. Load verifies consistency, not history.
Omitted labels use encoded indices/an existing binding; native predict stays encoded.
Standalone sklearn, CatBoost and regressors reject `class_labels=`; composites
delegate labels to the final native adapter.

Only recoverable native configuration is recorded; learning rate, max depth,
seed and thread count do not survive native XGBoost I/O and are not reconstructed.
Expand All @@ -103,14 +95,45 @@ assert bundle.class_labels == [["cold", "warm"]]
assert bundle.predict_proba(rows) == model.predict_proba(rows).tolist()
```

Export a pipeline's estimator as `export(pipeline.named_steps["est"], ...)`.
CatBoost pipelines, `class_labels=`, categorical/text/embedding features,
`class_labels=`, categorical/text/embedding features,
Depthwise/Lossguide, custom Python objectives/metrics and other losses are rejected.
Classifier losses: Logloss, CrossEntropy, MultiClass. Regressors: RMSE, MAE,
Quantile, Huber, Poisson, MAPE. Complete trees/borders/NaN handling/importances
are verified on load. [The CatBoost contract](docs/format-catboost.md) details
wrapper recovery, native-parser limits and the private in-memory serializer.

## Composite pipelines

Install `[sklearn,xgboost]` or `[sklearn,catboost]`. Export the fitted Pipeline
as it is; transformers use skops and the final estimator uses its native adapter:

```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier
from mlbundle import export, load

rows = [[0.0, 1.0], [1.0, 0.0], [0.1, 0.9], [0.9, 0.1]]
model = make_pipeline(StandardScaler(), XGBClassifier(n_estimators=3, n_jobs=1))
model.fit(rows, [0, 1, 0, 1])
export(model, "pipeline.mlbundle", class_labels=[["cold", "warm"]])
bundle = load("pipeline.mlbundle")
assert bundle.predict_proba(rows) == model.predict_proba(rows).tolist()
assert bundle.native_model.predict(rows).tolist() == model.predict(rows).tolist()
assert bundle.class_labels == [["cold", "warm"]]
```

Load rebuilds a real Pipeline after verifying both parts. Bundle probability labels
use XGBoost's bound originals; native classes/predict stay encoded. CatBoost keeps
native labels. Prefixes include scalers, PCA, SimpleImputer (indicators/empty columns),
KBinsDiscretizer, PolynomialFeatures, univariate selectors, OneHotEncoder/OrdinalEncoder,
identity/closed NumPy FunctionTransformer, nested native-free Pipeline and ColumnTransformer.
Each needs a fitted, data-free width proof. Unknown widths, FeatureUnion/KNNImputer,
slice/callable/scalar selectors, caching, metadata routing and other native placements
remain rejected. Step names/outer options are declarations; trees address transformed
features, without original-feature attribution. Schema/dtypes/samples describe original
inputs; six members permit schema plus explicit samples. See [the pipeline contract](docs/format-pipeline.md).

## Existing ONNX files

Install the `onnx` extra. This example assumes your existing `trained.onnx` is
Expand Down Expand Up @@ -148,11 +171,8 @@ from sklearn.tree import DecisionTreeClassifier
from mlbundle import export, inspect, load

training = pd.DataFrame(
{
"temperature": [0.0, 1.0, 0.1, 0.9],
"humidity": [1.0, 0.0, 0.9, 0.1],
"target": ["cold", "warm", "cold", "warm"],
}
[[0.0, 1.0, "cold"], [1.0, 0.0, "warm"]] * 2,
columns=["temperature", "humidity", "target"],
)
features = training.drop(columns="target")
model = DecisionTreeClassifier(random_state=7).fit(features, training["target"])
Expand All @@ -162,15 +182,8 @@ assert view.manifest.features.dtypes == ["float64", "float64"]
assert view.schema["reports"][0]["labels"] == ["cold", "warm"]
assert view.samples is None
assert load("schema.mlbundle").schema == view.schema

# Samples are real data: explicitly include the first two feature rows.
export(
model,
"sampled.mlbundle",
training_data=training,
target_name="target",
sample_rows=2,
)
export(model, "sampled.mlbundle", training_data=features, sample_rows=2)
assert inspect("sampled.mlbundle").samples == features.head(2).to_dict("records")
```

Expand All @@ -196,8 +209,6 @@ metadata or structure matches the model. `load` additionally deserializes throug
the closed adapter and recomputes native claims and structure before returning
the model and predictor. Export checks the same native round trip before writing.

After the sklearn example above, these commands work:

```shell
mlbundle inspect trained.mlbundle
mlbundle inspect trained.mlbundle --json
Expand All @@ -212,61 +223,45 @@ dependency rejection (`MLBundleError`), **2** usage/I/O failure.

## Supported models and limits

Support requires a faithful closed-adapter round trip; a catalog entry does not guarantee every fitted configuration exports.

- **Sklearn:** fitted classifiers exposing `predict_proba` and regressors with
recoverable feature/output counts. Tested families include trees/ensembles,
linear/regularized/robust/GLM models, SVMs, PLSRegression, naive Bayes, LDA/QDA,
dense brute-force neighbors, NearestCentroid, MLP and MultiOutput/RegressorChain.
MLPClassifier/Regressor (adam, sgd, lbfgs) retain optimizer state, alone/in pipelines, with null structure and complete commitments; see [MLP tests](tests/test_sklearn_mlp.py).
SimpleImputer/KNNImputer, scalers, KBinsDiscretizer, categorical encoders,
univariate selectors and closed NumPy FunctionTransformer functions round-trip,
including nested ColumnTransformer pipelines; see [the exact lists](docs/format-state.md).
Pipeline structure stays null because format 1 does not map transformed features. See
All-sklearn Pipeline structure stays null; composite trees expose the inner boundary. See
[tested configurations](tests/test_sklearn_families.py) and [core round trips/rejections](tests/test_sklearn.py).
- **Sklearn rejections:** custom estimators/subclasses, functions/types outside the
closed lists (including lambdas and user functions), compound fitted dtypes, classifiers without
probabilities, multilabel/ClassifierChain layouts, XGBoost/CatBoost pipeline steps,
probabilities, multilabel/ClassifierChain layouts, unsupported native placements,
and untrusted nested storage such as fitted sparse neighbor arrays.
HistGradientBoosting uses untrusted `TreePredictor` storage and is rejected.
- **Sklearn versions:** the closed class/attribute catalog is audited/tested
against **1.9.1 with skops 0.16.0**, not every version allowed by `>=1.6`.
Unknown classes/attributes fail export/load. The committed sklearn version
must match at export/load; other matching-version configurations can work
when they satisfy the catalog. No permissive fallback or advertised broader
range. See [state ownership](docs/format-state.md).
- **XGBoost:** exact XGBClassifier/XGBRegressor, one target, native `gbtree`
storage (including XGBoost 3.4.1 `dart` dropout normalized to gbtree).
Classifiers: `binary:logistic`, `multi:softprob`. Regressors:
`reg:squarederror`, `reg:absoluteerror`, `reg:pseudohubererror`,
`reg:squaredlogerror`, `reg:logistic`, `count:poisson`, `reg:gamma`,
`reg:tweedie`. Raw boosters, RF wrappers, subclasses, gblinear, legacy dart
storage, ranking/survival/custom objectives and multiple targets are rejected.
At most 100,000 trees and depth 14; exact native state and complete dumps are
required. See [the XGBoost mapping](docs/format-adapters.md#implemented-xgboost-mapping).
- **ONNX:** rank-two numeric matrix or width-one inputs, fixed width,
float32/float64/int32/int64; dynamic or one-row batches. One target: float
regression `[batch]`/`[batch,1]` or tensor/ZipMap probabilities, optional labels.
Classification requires native graph evidence. Ambiguous/image/string layouts,
larger fixed batches, multiple targets, external tensors, custom domains/local
functions and training/runtime configuration fail before CPU sessions.
Tested with ONNX 1.23.0/Runtime 1.30.0; operators/opsets must pass both.
Large unpacked ensembles/ZipMap and multiline producer versions work. One wire
validation per export/load; field/scalar ceilings follow 512 MiB, producer strings
allow 65,536 characters. See [ONNX bounds](docs/format-onnx.md).

Archive ceilings: **5 members**, **512 MiB** archive/compressed/member bytes,
Unknown state fails; export/load sklearn versions must match. Configurations
must satisfy the catalog and round trip; see [state ownership](docs/format-state.md).
- **XGBoost:** exact XGBClassifier/XGBRegressor, one target, native gbtree
(including XGBoost 3.4.1 dart normalized to gbtree). Closed probability/regression
objectives; raw boosters, RF wrappers, subclasses, ranking/survival/custom
objectives and unsupported storage are rejected. At most 100,000 trees/depth 14,
with complete state/dumps; see [the mapping](docs/format-adapters.md#implemented-xgboost-mapping).
- **ONNX:** fixed-width rank-two numeric inputs; one regression/probability target,
optional native labels and dynamic/one-row batches. Unsupported layouts,
external data/custom domains/functions and runtime/training configuration fail
before CPU sessions. Tested with ONNX 1.23.0/Runtime 1.30.0; see [ONNX bounds](docs/format-onnx.md).

Archive ceilings: **6 members**, **512 MiB** archive/compressed/member bytes,
**1 GiB** aggregate uncompressed bytes, ratio **200**, JSON depth **32**.
Manifest: **8 MiB**; structure: **256 MiB**; schema/samples: **16 MiB each**.
Features/classes: at most **100,000**; outputs: **1,024** (XGBoost/CatBoost/ONNX: one).
Callers may lower `ArchiveLimits`, never raise these ceilings. Native work has
additional [state](docs/format-state.md) and [ONNX](docs/format-onnx.md) bounds;
these byte budgets do not cap total process memory, prediction rows or CPU.
Callers may only lower `ArchiveLimits`. Native [state](docs/format-state.md)/[ONNX](docs/format-onnx.md)
bounds do not cap process memory, prediction rows or CPU.

## Bundle contents and security

Every bundle contains `manifest.json` and exactly one of `model.skops`,
`model.ubj`, `model.cbm`, `model.onnx`. Optional members are `schema.json`, `structure.json`
Every bundle contains `manifest.json` and one native member, or the declared
`model.skops` + `model.ubj`/`model.cbm` composite pair. Single-step composites use only the leaf. Optional members are `schema.json`, `structure.json`
(required when the adapter extracts structure) and explicitly requested
`samples.json`. No training DataFrame or arbitrary files are included.
The manifest records identity/time/writer, model parameters/counts/state,
Expand Down
17 changes: 12 additions & 5 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,18 +101,25 @@ make the native parser a sandbox.
The installed skops load path builds nodes, audits them and only then constructs
objects; NumPy buffers use `allow_pickle=False`. No pickle/joblib loading fallback
is used. Skops imports `pickle` only to recognize an unrelated save-side exception;
this does not deserialize pickle bytes. Custom estimators/functions and XGBoost/CatBoost
estimators inside sklearn pipelines are rejected. CatBoost rejections name the unsupported
type and advise exporting the estimator directly; neither adapter unwraps pipelines. HistGradientBoosting's separate
this does not deserialize pickle bytes. Custom estimators/functions and unsupported
native placements are rejected. Exact outer Pipelines ending in an admitted native
wrapper split transformer skops bytes from UBJSON/CBM, retaining each adapter's
preflight, trust, state/claims and structure checks; no booster type enters skops.
Closed fitted-state width proofs never use data. Shared bounded names/parameters/
commitments and inner tree coordinates are recomputed before closed Pipeline
reconstruction is returned. Cache/metadata routing/unknown outer state fail;
step names and outer options are declarations, not authenticated training history.
See [the composite contract](docs/format-pipeline.md).
HistGradientBoosting's separate
`TreePredictor` storage is also excluded. Broadening the supplemental list requires
source review and hostile-payload tests, never author approval or bundle metadata.

Both export and load recompute the same model claims and structures and compare
canonical bytes, preserving class-label types and order. Export additionally
compares the original and reloaded native models before publication. Loaded results
contain the native model for runtime explanations; mutating it invalidates the
relationship to the verified metadata. Pipeline structure is null because v1
does not describe its input transformations.
relationship to the verified metadata. All-sklearn Pipeline structure stays null;
composites expose checked native trees in transformed feature coordinates.

The closed sklearn class/attribute catalog is audited against 1.9.1 with skops
0.16.0. Other versions/configurations can be rejected for unknown state. The
Expand Down
Loading
Loading