feat: New variational losses for Lp calibration errors

.github/workflows/testing.yml

@@ -30,11 +30,11 @@ jobs:
           # Install a specific version of uv.
           version: "0.5.4"
       - name: Install hatch
-        run: uv pip install --system hatch
+        run: uv pip install --system hatch "virtualenv<21"
       - name: Install swig
         run: uv pip install --system swig
       - name: Run tests
-        run: hatch test
+        run: hatch run pytest
       - name: Upload coverage reports to Codecov
         uses: codecov/codecov-action@v4.5.0
         with:

README.md

@@ -14,7 +14,8 @@ metrics for assessing the quality of probabilistic predictions, and
     logistic-based calibration functions.
 
 It accompanies our papers
-[Rethinking Early Stopping: Refine, Then Calibrate](https://arxiv.org/abs/2501.19195) and [Structured Matrix Scaling for Multi-Class Calibration](https://arxiv.org/abs/2511.03685).
+[Rethinking Early Stopping: Refine, Then Calibrate](https://arxiv.org/abs/2501.19195) and [Structured Matrix Scaling for Multi-Class Calibration](https://arxiv.org/abs/2511.03685) 
+and [A Variational Estimator for Lp Calibration Errors](https://arxiv.org/abs/2602.24230).
 Please cite us if you use this repository for research purposes.
 The experiments from the papers can be found here: 
 - Rethinking Early Stopping:
@@ -23,6 +24,8 @@ The experiments from the papers can be found here:
   - [theory](https://github.com/eugeneberta/RefineThenCalibrate-Theory).
 - Structured Matrix Scaling: 
   [all experiments](https://github.com/eugeneberta/LogisticCalibrationBenchmark).
+- A Variational Estimator for Lp Calibration Errors: 
+  [all experiments](https://github.com/ElSacho/Evaluating_Lp_Calibration_Errors).
 
 ## Installation
 
@@ -31,10 +34,11 @@ Probmetrics is available via
 pip install probmetrics
 ```
 To obtain all functionality, install `probmetrics[extra,dev,dirichletcal]`.
-- extra installs more packages for smooth ECE, 
+- extra installs more packages for our CatBoost/LightGBM-based $L_p$ calibration 
+  error metrics, smooth ECE (only works with scikit-learn versions <= 1.6), 
   Venn-Abers calibration, 
   centered isotonic regression, 
-  the temperature scaling implementation in NetCal.
+  and the temperature scaling implementation in NetCal.
 - dev installs more packages for development (esp. documentation)
 - dirichletcal installs Dirichlet calibration, 
   which however only works for Python 3.12 upwards.
@@ -156,7 +160,10 @@ metrics = Metrics.from_names([
     'refinement_logloss_ts-mix_all', 
     'calib-err_logloss_ts-mix_all',
     'refinement_brier_ts-mix_all', 
-    'calib-err_brier_ts-mix_all'
+    'calib-err_brier_ts-mix_all',
+    'calib-err_proper-L1-binary-as-1d_WS_CatboostClassifier_all',
+    'calib-err_proper-L2-binary-as-1d_WS_CatboostClassifier_all',
+    'calib-err_proper-Linf-binary-as-1d_WS_CatboostClassifier_all',
 ])
 ```
 
@@ -168,12 +175,137 @@ Metrics.get_available_names(metric_type=MetricType.CLASS)
 
 While there are some classes for regression metrics, they are not implemented.
 
+## Advanced calibration, confidence, and top-class metrics
+
+Beyond standard metrics, you can evaluate proper Lp calibration errors for 
+any p, as well as isolate specific types of errors like over-confidence, 
+under-confidence, and top-class errors. 
+
+**Note:** Over- and under-confidence metrics are designed for binary classification.
+To use those for multi-class, please use `TopClassLoss(OverConfidenceLoss(your_metric))`.
+
+```python
+from probmetrics.metrics import (
+  ProperLpLoss,
+  BrierLoss,
+  OverConfidenceLoss,
+  UnderConfidenceLoss,
+  TopClassLoss
+)
+
+# Evaluate proper Lp calibration errors for any p
+lp_loss_l1 = ProperLpLoss(p=1)  # Evaluate E[ \| Y - E[Y|f(X)] \|_1 ] 
+lp_loss_l2 = ProperLpLoss(p=2)  # Evaluate E[ \| Y - E[Y|f(X)] \|_2 ] 
+
+# Evaluate over-confidence and under-confidence 
+# (Initialize via string name or by passing a metric object)
+over_brier = OverConfidenceLoss.from_name("brier")
+under_L1 = UnderConfidenceLoss.from_name("proper-L1")
+
+# Evaluate top-class error with any accompanying loss
+topclass_brier = TopClassLoss(BrierLoss(binary_as_multiclass=False))
+topclass_L1 = TopClassLoss.from_name("proper-L1")
+
+# Compose wrappers (e.g., top-class with underconfidence for proper-L1)
+under_topclass_l1 = TopClassLoss(UnderConfidenceLoss.from_name("proper-L1"))
+over_topclass_brier = TopClassLoss(OverConfidenceLoss(BrierLoss()))
+
+# Some metrics are listed by default, here are some of them
+metrics = metrics = Metrics.from_names([
+    'proper-L1-binary-as-1d', # use to estimate  E[ \| Y - E[Y|f(X)] \|_1 ] and treat binary 
+                              # predictions as scalars with shapes (n,1) )
+    'proper-L2', # use to estimate  E[ \| Y - E[Y|f(X)] \|_2 ] (and treat binary predictions 
+                 # as vector with shapes (n,2) )
+    "topclass-proper-L1-binary-as-1d", # Estimate L1 calibration error of top class 
+    "topclass-under-proper-L1-binary-as-1d", # Estimate L1-overconfidence of top class 
+    "topclass-over-proper-L1-binary-as-1d", # Estimate L1-underconfidence of top class
+])
+
+```
+
+Once those losses are defined, you can evaluate the calibration error by doing:
+
+```python
+from probmetrics.metrics import MetricsWithCalibration, CombinedMetrics
+from probmetrics.classifiers import WS_CatboostClassifier, WS_LGBMClassifier
+from probmetrics.splitters import CVSplitter
+
+loss = ProperLpLoss(p=2) 
+
+metrics = MetricsWithCalibration(loss,
+                            calibrator=WS_CatboostClassifier(), # The classifier used to recalibrate the predictions
+                            val_splitter=CVSplitter(n_cv=5) # cross-validation splitter
+                            )
+
+# or use combined metrics
+combined_losses = CombinedMetrics( 
+                                    [
+                                    ProperLpLoss(p=1), 
+                                    OverConfidenceLoss.from_name("brier"), 
+                                    OverConfidenceLoss.from_name("proper-L1") , 
+                                    UnderConfidenceLoss.from_name("proper-L1" ), 
+                                    UnderConfidenceLoss( BrierLoss() ),
+                                    BrierLoss()
+                                    ]
+                                  )
+
+metrics = MetricsWithCalibration(combined_losses,
+                            calibrator=WS_LGBMClassifier(), 
+                            val_splitter=CVSplitter(n_cv=5)
+                            )
+
+y_true = torch.tensor(...)
+y_prob = torch.tensor(...)
+results = metrics.compute_all_from_labels_probs(y_true, y_prob)
+```
+
+The `calibrator` argument is a class used to recalibrate the original predictions. 
+Any estimator that inherits from sklearn.base.ClassifierMixin (i.e., follows the 
+scikit-learn classifier API) can be used.
+We recommend using `WS_CatboostClassifier` with default parameters. 
+The "WS" stands for "Warm Start", as predictions are initialized at the 
+original predicted $f(x)$ values, (see the paper A Variational Estimator for Lp 
+Calibration Errors for additional information). 
+
+
+### Binary vs. multiclass formatting
+
+The library expects predictions in a multiclass format with shape `(n_samples, n_classes)`.
+For binary classification, you can control whether to treat the output as a two-column distribution 
+or a single-column probability using the `binary_as_multiclass` parameter.
+
+Setting `binary_as_multiclass=False` tells the loss function to treat 
+`(n_samples, 2)` predictions as a single-column `(n_samples, 1)` probability.
+ It automatically transforms binary labels $Y \in {0, 1}$ and the probability
+  column $f(X) \in [0, 1]$ for the calculation.
+
+Those features are also valid with the `TopClassLoss`.
+The `TopClassLoss` wrapper focuses the loss calculation on the class with 
+the highest predicted probability. The behavior changes based on your binary setting,
+for instance:
+
+| Configuration                                                  | Estimate | Description |
+|:---------------------------------------------------------------| :--- | :--- |
+| `TopClassLoss( ProperLpLoss(p=1) )`                            | $\mathbb{E}[ \lvert Z - \mathbb{E}[Z \mid topf(X)] \rvert ]$ | Scalar probability: $topf(X)$ is the scalar probability of the top class of $f(X)$; $Z \in \{0, 1\}$ equals $1$ if the label is what the top-class predicted and $0$ otherwise. Evaluates the absolute error of the top-class prediction. |
+| `TopClassLoss( ProperLpLoss(p=1, binary_as_multiclass=True) )` | $\mathbb{E}[ \Vert \mathbf{Z} - \mathbb{E}[\mathbf{Z} \mid topf(X)] \Vert_1 ]$ | Vectorized: $\mathbf{Z}$ is a one-hot vector. Calculates the $L_1$ norm of the error vector. |
+
+
 ## Contributors
 - David Holzmüller
 - Eugène Berta
+- Sacha Braun
 
 ## Releases
 
+- v1.2.0 by [@elsacho](https://github.com/elsacho): Added new proper loss functions:
+  - ProperLpLoss(p=p): Metrics to evaluate $E[ \Vert f(X) - E[Y|f(X)] \Vert_p ]$ where $f(X)$ are the 
+    predictions of the classifier, $p >= 1$, including `p=float("inf")`
+  - TopClassLoss: A wrapper to variationally evaluate top-class errors.
+  - OverConfidenceLoss & UnderConfidenceLoss: Wrappers to variationally evaluate 
+    over/under-confidence in binary predictors.
+  - New classifiers:  Added `WS_CatboostClassifier` and `WS_LGBMClassifier` for 
+    evaluating calibration errors.
+  - removed sklearn < 1.7 constraint.
 - v1.1.0 by [@eugeneberta](https://github.com/eugeneberta): Improvements to the SVS and SMS calibrators:
   - logit pre-processing with `'ts-mix'` is now automatic, 
     and the global scaling parameter $\alpha$ is fixed to 1. This yields:
@@ -197,4 +329,6 @@ While there are some classes for regression metrics, they are not implemented.
   - add TorchCal temperature scaling
   - minor fixes in AutoGluon temperature scaling 
     that shouldn't affect the performance in practice
-- v0.0.1 by [@dholzmueller](https://github.com/dholzmueller): Initial release
+- v0.0.1 by [@dholzmueller](https://github.com/dholzmueller):
+  Initial release with classification metrics, 
+  calibration/refinement metrics, and some post-hoc calibration methods.

probmetrics/__about__.py

@@ -1,2 +1,2 @@
 # SPDX-License-Identifier: Apache-2.0
-__version__ = "1.1.0"
+__version__ = "1.2.0"