diff --git a/README.md b/README.md index e58c6413..db87c950 100644 --- a/README.md +++ b/README.md @@ -1,67 +1,217 @@ # FroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learning -This is the official PyTorch implementation of the [FroSSL paper](https://arxiv.org/pdf/2310.02903): +[![Paper](https://img.shields.io/badge/Paper-ECCV%202024-blue)](https://arxiv.org/abs/2310.02903) +[![arXiv](https://img.shields.io/badge/arXiv-2310.02903-b31b1b.svg)](https://arxiv.org/abs/2310.02903) +[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT) +[![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg)](https://www.python.org/downloads/) +[![PyTorch 2.0](https://img.shields.io/badge/PyTorch-2.0.1-ee4c2c.svg)](https://pytorch.org/) -``` -@inproceedings{skean2024frossl, - title={FroSSL: Frobenius Norm Minimization for Self-Supervised Learning}, - author={Skean, Oscar and Dhakal, Aayush and Jacobs, Nathan and Giraldo, Luis Gonzalo Sanchez}, - booktitle={European Conference on Computer Vision}, - year={2024} -} -``` -This implementation started as a fork of the fantastic [solo-learn](https://github.com/vturrisi/solo-learn.git) library. We are currently working on a pull request to merge our contributions into the library. +**FroSSL is a self-supervised learning (SSL) objective for efficient multiview representation learning.** It combines covariance regularization with multiview training while avoiding expensive eigendecomposition, letting you reach strong representations *faster* without sacrificing quality. The objective is simple: mean-squared error for augmentation invariance plus a Frobenius-norm covariance term to prevent collapse. + +If you are looking for an SSL baseline that is easy to understand, cheap to train, and scales naturally to many augmented views, FroSSL is designed for you. + +## Why use FroSSL? + +- **Computationally efficient covariance regularization** — no eigendecomposition, so per-step cost stays low even as you add views. See [Faster convergence](#faster-convergence). +- **Naturally supports multiview SSL** — designed from the start for more than two views. See [Why multiview matters](#why-multiview-matters). +- **Faster convergence than competing SSL objectives** — reaches a target accuracy in fewer epochs and less wall-clock time. See [Faster convergence](#faster-convergence). +- **Simple objective** — just MSE invariance + Frobenius covariance regularization ([~30 lines of code](solo/losses/frossl.py)). See [Method overview](#method-overview). +- **Competitive linear-probe accuracy** across standard benchmarks (STL-10, Tiny ImageNet, ImageNet-100). See [Results](#results). + +### How does it compare? + +The goal of this table is not to claim FroSSL wins everywhere — it is to help you see where FroSSL fits among common baselines. + +| Method | Family | Negative pairs | Momentum encoder | Eigendecomposition | Multiview-friendly | +|--------------|---------------------|:--------------:|:----------------:|:------------------:|:------------------:| +| SimCLR | Sample-contrastive | Yes | No | No | Limited | +| BYOL | Asymmetric | No | Yes | No | Moderate | +| Barlow Twins | Dimension-contrastive | No | No | No | Moderate | +| VICReg | Dimension-contrastive | No | No | No | Moderate | +| W-MSE | Dimension-contrastive | No | No | Yes (whitening) | Limited | +| **FroSSL** | Dimension-contrastive | **No** | **No** | **No** | **Yes** | + +Want the details? See [`docs/WHY_FROSSL.md`](docs/WHY_FROSSL.md) for a deeper comparison of the math, tradeoffs, and evidence against SimCLR, BYOL, VICReg, and Barlow Twins. + +## Faster convergence -## Preparation -### Installing Requirements -We provide instructions for how to setup a conda environment and install the necessary dependencies: +FroSSL's central contribution is **optimization efficiency**: it is designed to reach strong representations in *fewer training epochs*, reducing overall training cost. The figure below reports the number of epochs (and wall-clock time) each method needs to reach a target STL-10 accuracy, along with per-step memory and batch time. +![Epochs and wall-clock time to reach a target accuracy for common SSL methods](experiments/time_scaling/ssl_method_triangles.png) + +FroSSL is efficient with two views and gets *even more efficient with more views* — the 8-view configuration reaches the target in the fewest epochs and least wall-clock time of any method compared. Because the objective avoids eigendecomposition, adding views stays cheap. The paper provides theoretical and empirical support that this faster convergence comes from how FroSSL shapes the eigenvalues of the embedding covariance matrices — without ever computing them. + +## Why multiview matters + +Increasing the number of augmented views often improves SSL optimization, but many covariance-based objectives become expensive in this setting because their regularizers rely on eigendecomposition, which scales poorly with the number of views. FroSSL was designed specifically to make **multiview covariance regularization practical**: the Frobenius-norm term is cheap to compute per view, so training on 4 or 8 views is a straightforward configuration change rather than a computational burden. + +## Available in popular SSL libraries + +To make FroSSL easy to use with a familiar API, we are integrating it into widely used SSL libraries: + +- **lightly** — a `FroSSLLoss` module with PyTorch and PyTorch-Lightning examples and a CIFAR-10 benchmark entry. **Merged** ([pull request](https://github.com/lightly-ai/lightly/pull/1962)) — available now on lightly's `master` branch, not yet in a tagged PyPI release; install from source (`pip install git+https://github.com/lightly-ai/lightly.git`) to use it today. +- **solo-learn** — the FroSSL method and loss with training configs, tests, and docs ([pull request](https://github.com/vturrisi/solo-learn/pull/398), in progress). + +You can also use this repository directly (see [Quick Start](#quick-start)). + +## Quick Start + +```bash +# 1. Clone +git clone https://github.com/OFSkean/FroSSL.git +cd FroSSL + +# 2. Install (Python 3.10 recommended) +conda create -n frossl python=3.10 && conda activate frossl +pip install -r requirements.txt + +# 3. Train FroSSL on CIFAR-10 (downloads data automatically) +bash run_cifar10.sh ``` -# clone the repository -git clone git@github.com:OFSkean/frossl.git -cd ./frossl -# create env +That's it — `run_cifar10.sh` pretrains a ResNet-18 with FroSSL and then trains a linear probe. Metrics are logged to [Weights & Biases](https://wandb.ai/) (make sure you are logged in via `wandb login`). + +## Installation + +```bash +git clone https://github.com/OFSkean/FroSSL.git +cd FroSSL + conda create -n frossl python=3.10 conda activate frossl -# install dependencies pip install -r requirements.txt ``` +FroSSL builds on the excellent [solo-learn](https://github.com/vturrisi/solo-learn) library. FroSSL is also available in [lightly](https://github.com/lightly-ai/lightly) (merged, see [Available in popular SSL libraries](#available-in-popular-ssl-libraries)); a pull request to add it to solo-learn is in progress. + ### Datasets -The datasets CIFAR-10, CIFAR-100, and STL-10 are able to be downloaded automatically by Pytorch. If you want to run on other datasets like tiny-imagenet, some preparation will be required. By default, the data is assumed to live in `./datasets/{dataset-name}`. To change this, you have to adjust the configuration files which can be found in `./scripts/pretrain/*`. -#### Tiny ImageNet -We have provided an installation script for Tiny ImageNet that can be used like: +CIFAR-10, CIFAR-100, and STL-10 download automatically via PyTorch. Other datasets need a little setup. By default data lives in `./datasets/{dataset-name}`; change this in the config files under `scripts/pretrain/*`. -``` -cd scripts/utils/tiny-imagenet -./downloader.sh +- **Tiny ImageNet**: `cd scripts/utils/tiny-imagenet && ./downloader.sh` +- **ImageNet**: follow [this guide](https://cloud.google.com/tpu/docs/imagenet-setup#download-dataset). Required to build ImageNet-100. +- **ImageNet-100**: `python make_imagenet100.py full/imagenet/path desired/imagenet100/path` + +## Training examples + +Each dataset and method has its own config under `scripts/pretrain/`. By default these match the paper's settings. Training logs to Weights & Biases, so run `wandb login` first. + +```bash +# Pretrain + linear probe in one shot: +./pretrain_then_linear.sh frossl-cifar10 cifar10 frossl 2 +./pretrain_then_linear.sh frossl-imagenet100 imagenet100 frossl 2 +./pretrain_then_linear.sh frossl-tiny-imagenet tiny-imagenet frossl 2 ``` -#### ImageNet -If you don't already have ImageNet downloaded, we recommend following [this guide](https://cloud.google.com/tpu/docs/imagenet-setup#download-dataset). Note that ImageNet **must be downloaded** to prepare the ImageNet-100 dataset. +Convenience wrappers are also provided: `bash run_cifar10.sh`, `bash run_imagenet100.sh`, `bash run_tiny.sh`. + +To tweak hyperparameters, augmentations, or the loss, edit the relevant YAML, e.g. `scripts/pretrain/cifar/frossl.yaml`. + +### Using FroSSL on your own dataset + +FroSSL works on any image dataset with no code changes — just point a config at your data. Copy an existing config and edit the `data` block: + +```yaml +# scripts/pretrain/custom/frossl.yaml +method: "frossl" +backbone: + name: "resnet18" +method_kwargs: + proj_hidden_dim: 2048 + proj_output_dim: 1024 + kernel_type: "linear" + invariance_weight: 1.0 +data: + dataset: "custom" + train_path: "PATH_TO_TRAIN_DIR" # ImageFolder layout: train//.jpg + val_path: "PATH_TO_VAL_DIR" # remove if you have no validation split + format: "image_folder" + no_labels: True # set True if images are not in per-class subfolders +``` -#### ImageNet-100 -Once you have the ImageNet dataset downloaded, you can create the ImageNet-100 dataset with: +Then launch: +```bash +python3 -u main_pretrain.py --config-path scripts/pretrain/custom --config-name frossl.yaml ``` -python make_imagenet100.py full/imagenet/path desired/imagenet100/path + +To use more views, increase the number of crops in the augmentation config (`scripts/pretrain/custom/augmentations/`). This is where FroSSL's efficiency advantage grows. + +## Results + +**Takeaway:** FroSSL learns representations of competitive quality while consistently reaching a target accuracy in *fewer training epochs* than other SSL objectives, and its advantage widens as more views are added. In the paper, FroSSL trains a ResNet-18 to competitive linear-probe accuracy on STL-10, Tiny ImageNet, and ImageNet-100. + +See the [FroSSL paper](https://arxiv.org/abs/2310.02903) for the full linear-probe tables, convergence curves, and ablations. + +### Reproducible CIFAR-10 comparison + +As an independent check while adding FroSSL to [lightly](https://github.com/lightly-ai/lightly/pull/1962) (merged — see [Available in popular SSL libraries](#available-in-popular-ssl-libraries)), we ran it through lightly's CIFAR-10 kNN benchmark against seven common SSL methods **under an identical protocol on the same GPU** (ResNet-18, batch size 512, 200 epochs, kNN with k=200). This is a two-view configuration — FroSSL's advantage grows further with more views (see [Faster convergence](#faster-convergence)). + +| Method | kNN Top-1 | Runtime | Peak GPU | +|--------------|:---------:|:-------:|:--------:| +| **FroSSL** | **86.9%** | 52.1 min | 4.83 GB | +| BYOL | 86.5% | 62.5 min | 5.43 GB | +| DCL | 85.0% | 51.9 min | 4.85 GB | +| SimCLR | 84.8% | 51.6 min | 4.85 GB | +| MoCo | 84.8% | 64.9 min | 5.53 GB | +| NNCLR | 83.8% | 52.8 min | 4.96 GB | +| Barlow Twins | 83.5% | 51.8 min | 4.96 GB | +| SimSiam | 82.0% | 52.1 min | 4.97 GB | + +FroSSL reaches the **highest kNN accuracy**, at the **lowest peak memory** and in the fastest tier of runtimes (the momentum-encoder methods BYOL and MoCo are noticeably slower). It is also the fastest to converge: it matches every other method's *final* 200-epoch accuracy in fewer epochs — e.g. Barlow Twins' best by epoch 138 and SimCLR's by epoch 151 — then continues to improve. + +![CIFAR-10 kNN accuracy vs epoch for FroSSL and seven common SSL methods, same GPU and protocol](experiments/lightly_cifar10/cifar10_knn_curves.png) + +Single seed. kNN accuracy is hardware-independent; runtime and peak memory were measured on one NVIDIA RTX 4090 and are comparable within this run. Per-epoch data: [`experiments/lightly_cifar10/cifar10_knn_curves.csv`](experiments/lightly_cifar10/cifar10_knn_curves.csv). + +## Method overview + +FroSSL trains an encoder so that different augmented views of the same image map to similar embeddings (invariance), while keeping each view's embedding dimensions decorrelated and informative (regularization) to avoid collapse. + +```mermaid +flowchart TD + X["Input image"] --> A1["Augmented view 1"] + X --> A2["Augmented view 2"] + X --> AN["Augmented view V"] + A1 --> E["Shared encoder"] + A2 --> E + AN --> E + E --> P["Projection head"] + P --> Z1["Embedding z1"] + P --> Z2["Embedding z2"] + P --> ZN["Embedding zV"] + Z1 --> INV["MSE invariance loss (views agree)"] + Z2 --> INV + ZN --> INV + Z1 --> REG["Frobenius covariance regularization (no collapse)"] + Z2 --> REG + ZN --> REG ``` -## Training and Evaluating a Model -1. Make sure you have a wandb account and are logged in via the CLI. All hyperparameters, losses, system details, etc. will get logged to wandb. +The full objective is `L = MSE(views) + Frobenius-norm covariance regularization`. Concretely, for each view the covariance (or Gram) matrix is formed and its (negative log) Frobenius norm is minimized — an eigendecomposition-free way to encourage a well-spread eigenvalue spectrum. See equations (3) and (6) in the paper. + +- Loss implementation: [`solo/losses/frossl.py`](solo/losses/frossl.py) +- Method / training step: [`solo/methods/frossl.py`](solo/methods/frossl.py) +- Deeper comparison and intuition: [`docs/WHY_FROSSL.md`](docs/WHY_FROSSL.md) -2. Check out the .yaml configuration at `./scripts/pretrain/stl10/frossl.yaml`. This file is where you can tweak hyperparameters for the training procedure, augmentations, and loss. Every dataset and method has its own configuration file. By default, these configurations are setup to match what we used for the paper. +## Citation -3. We have provided three scripts to serve as examples for training a model: **run_cifar10.sh**, **run_imagenet100.sh**, **run_tiny.sh**. These are currently configured to run FroSSL on a specific dataset. +If you find FroSSL useful, please cite: + +```bibtex +@inproceedings{skean2024frossl, + title={FroSSL: Frobenius Norm Minimization for Self-Supervised Learning}, + author={Skean, Oscar and Dhakal, Aayush and Jacobs, Nathan and Giraldo, Luis Gonzalo Sanchez}, + booktitle={European Conference on Computer Vision (ECCV)}, + year={2024} +} +``` -4. Run an above script (or modify it) like `bash ./run_cifar10.sh` +## Let us know +Please [open an issue](https://github.com/OFSkean/FroSSL/issues) if you hit any errors or difficulties. We are happy to help resolve issues or answer questions. -## Objective Function -The FroSSL objective function is [implemented here](https://github.com/OFSkean/FroSSL/blob/main/solo/losses/frossl.py) and [used here](https://github.com/OFSkean/FroSSL/blob/main/solo/methods/frossl.py). +## Acknowledgements -## Let Us Know -Please open an issue on Github if you encounter any errors or difficulties using this implementation. We are happy to help resolve issues or answer any questions you may have! +This implementation started as a fork of the fantastic [solo-learn](https://github.com/vturrisi/solo-learn) library, and is distributed under the MIT license. diff --git a/docs/WHY_FROSSL.md b/docs/WHY_FROSSL.md new file mode 100644 index 00000000..d75f1e53 --- /dev/null +++ b/docs/WHY_FROSSL.md @@ -0,0 +1,138 @@ +# Why FroSSL? A deeper comparison + +This document expands on the short comparison in the [README](../README.md). It explains the +mathematical intuition behind FroSSL, how it relates to popular self-supervised learning (SSL) +objectives, and the empirical evidence for its efficiency. The README is meant to be concise; this +page is the place to link from issues, discussions, and framework integrations when someone asks +"how is FroSSL different from *X*?" + +For the authoritative treatment, see the [FroSSL paper](https://arxiv.org/abs/2310.02903). + +## The problem FroSSL addresses + +Multiview SSL methods learn representations by making augmented views of the same image agree +(*invariance*) while preventing the trivial solution where every embedding collapses to a constant +(*collapse avoidance*). Recent methods fall into three families: + +- **Sample-contrastive** (e.g. SimCLR, MoCo): pull positives together, push negatives apart. +- **Asymmetric / distillation** (e.g. BYOL, SimSiam): a predictor and stop-gradient (often with a + momentum encoder) prevent collapse without negatives. +- **Dimension-contrastive** (e.g. Barlow Twins, VICReg, W-MSE): regularize the embedding covariance + so that feature dimensions stay decorrelated and informative. + +These families tend to converge to solutions of similar final quality, but they are **not equally +efficient**: some need many more epochs to reach a target accuracy. Two known levers for improving +efficiency are (1) regularizing the covariance eigenvalues and (2) using more augmented views. +Unfortunately, these two levers are hard to combine: explicit eigenvalue regularization requires an +**eigendecomposition**, whose cost grows quickly as more views are added. + +**FroSSL reconciles both levers while avoiding eigendecomposition entirely.** + +## The FroSSL objective + +FroSSL is a dimension-contrastive method with two terms: + +1. **Invariance** — mean-squared error (MSE) between the (normalized) embeddings of different views, + pulling views of the same image together. +2. **Regularization** — the negative log Frobenius norm of each view's (trace-normalized) + covariance/Gram matrix, which discourages collapse. + +Written per view `v`: + +``` +L = Σ_v [ MSE(z_v, z̄) − log ‖ C_v ‖²_F ] +``` + +where `z̄` is the mean embedding across views and `C_v` is the trace-normalized covariance (or Gram) +matrix of view `v`. + +### Why the Frobenius norm? + +Minimizing the Frobenius norm of a trace-normalized covariance matrix is equivalent to encouraging +its eigenvalues to be **uniform** (maximally spread out), which is exactly what collapse-avoidance +needs — but the Frobenius norm can be computed directly from matrix entries via +`‖C‖²_F = Σ_ij C_ij²`, with **no eigendecomposition**. This is the key trick: FroSSL gets the +spectrum-shaping benefits of eigenvalue regularization at the cost of a cheap matrix norm. + +The paper connects this to a matrix-based Rényi entropy view: minimizing `−log ‖C‖²_F` maximizes a +second-order entropy of the embeddings, which spreads information across dimensions. + +## Head-to-head comparison + +| Property | SimCLR | BYOL | Barlow Twins | VICReg | **FroSSL** | +|---|---|---|---|---|---| +| Family | Sample-contrastive | Asymmetric | Dimension-contrastive | Dimension-contrastive | Dimension-contrastive | +| Needs negative pairs | Yes | No | No | No | **No** | +| Needs momentum encoder / predictor | No | Yes | No | No | **No** | +| Large-batch sensitivity | High | Moderate | Moderate | Low | **Low** | +| Uses eigendecomposition | No | No | No | No | **No** | +| Explicit spectrum shaping | No | No | Partial (cross-corr.) | Partial (variance/cov.) | **Yes (Frobenius)** | +| Scales cheaply to many views | Limited | Moderate | Moderate | Moderate | **Yes** | +| Epoch efficiency (to target acc.) | Moderate | Moderate | Moderate | Moderate | **High** | + +### vs. SimCLR + +SimCLR relies on many negative samples and therefore large batches, and its InfoNCE loss does not +extend naturally to more than two views. FroSSL needs no negatives and no large batches, and adding +views is a first-class configuration. + +### vs. BYOL + +BYOL avoids collapse with an asymmetric predictor, stop-gradient, and a momentum (EMA) encoder — +extra moving parts and memory. FroSSL avoids collapse with an explicit, symmetric covariance term: +no predictor, no momentum encoder, and a loss that is easy to reason about. + +### vs. Barlow Twins + +Barlow Twins drives the cross-correlation matrix between two views toward the identity. This +implicitly decorrelates dimensions but is formulated for two views and does not directly optimize +the covariance spectrum. FroSSL regularizes each view's own covariance via the Frobenius norm, which +generalizes cleanly to `V` views and empirically converges faster. + +### vs. VICReg + +VICReg is the closest relative: it also regularizes variance and covariance without negatives or a +momentum encoder. The differences are (1) VICReg uses separate variance + covariance hinge terms +with three loss weights to tune, whereas FroSSL uses a single Frobenius term with essentially one +invariance weight, and (2) FroSSL is designed so that the regularizer stays cheap as views increase, +making multiview training practical. In the paper's experiments FroSSL reaches target accuracies in +fewer epochs. + +### vs. W-MSE and other whitening / eigen-based methods + +Whitening-based methods (e.g. W-MSE) and log-determinant / spectral regularizers achieve strong +decorrelation but require matrix inversions or eigendecompositions, which become expensive with more +views and can be numerically delicate. FroSSL targets the same spectrum-shaping goal with only a +Frobenius norm. + +## Empirical evidence + +- **Faster convergence.** FroSSL reaches a target STL-10 accuracy in fewer epochs and less + wall-clock time than the compared methods; the multiview (4- and 8-view) configurations are the + most efficient of all. See the efficiency figure in the [README](../README.md#faster-convergence). +- **Competitive quality.** On linear-probe evaluation with a ResNet-18, FroSSL is competitive with + strong baselines on STL-10, Tiny ImageNet, and ImageNet-100. +- **Why it converges faster.** The paper gives theoretical and empirical evidence that FroSSL's + speedup comes from how it shapes the eigenvalue spectrum of the embedding covariance — achieved + without ever computing eigenvalues. + +## When should I use FroSSL? + +Use FroSSL when you want a **simple, negative-free, momentum-free** SSL objective that is **cheap to +train**, **scales to many views**, and **reaches good representations quickly** — for example as a +strong, fast-to-run baseline in SSL research or when compute budget is limited. + +If you specifically need a method with published large-scale ImageNet-1k checkpoints today, or a +particular downstream ecosystem, check the relevant model zoos first — FroSSL checkpoints and +framework integrations (solo-learn, lightly) are in progress. + +## Citation + +```bibtex +@inproceedings{skean2024frossl, + title={FroSSL: Frobenius Norm Minimization for Self-Supervised Learning}, + author={Skean, Oscar and Dhakal, Aayush and Jacobs, Nathan and Giraldo, Luis Gonzalo Sanchez}, + booktitle={European Conference on Computer Vision (ECCV)}, + year={2024} +} +``` diff --git a/experiments/lightly_cifar10/cifar10_knn_curves.csv b/experiments/lightly_cifar10/cifar10_knn_curves.csv new file mode 100644 index 00000000..8bbb2b65 --- /dev/null +++ b/experiments/lightly_cifar10/cifar10_knn_curves.csv @@ -0,0 +1,201 @@ +epoch,FroSSL,BYOL,SimCLR,DCL,MoCo,NNCLR,BarlowTwins,SimSiam +1,31.8000,35.9300,42.5300,33.3800,32.8700,35.8100,27.3900,34.6100 +2,34.6800,37.8800,44.5100,38.1400,33.2700,38.8000,28.2000,35.8900 +3,36.6000,38.7300,47.4000,41.4300,35.6900,42.0300,28.4400,36.6000 +4,40.1300,39.6400,50.6600,42.8400,39.2800,43.1400,27.9100,36.5100 +5,41.5300,40.6300,52.9500,45.2100,41.4000,45.2400,27.7900,36.6400 +6,42.3700,42.3600,54.5200,47.2300,43.1200,48.0300,28.7800,37.8600 +7,44.6600,44.4100,56.4400,49.3800,44.9300,50.0600,30.6400,38.9600 +8,46.3800,45.8300,57.5300,52.8600,47.1400,52.0200,31.0600,39.6600 +9,49.8500,46.5500,58.5500,55.1300,48.5300,53.1400,33.2000,40.8200 +10,52.0400,47.5300,60.6100,57.3000,49.5900,53.7900,35.9000,40.9300 +11,53.9800,48.7600,61.1600,58.6500,51.0900,55.1700,37.5300,40.5100 +12,56.4100,43.9500,61.6400,60.9100,52.4700,55.1900,39.5000,38.9100 +13,58.0500,50.6000,63.4300,62.5200,53.6200,56.6600,41.1500,39.7500 +14,60.4100,51.1700,63.1400,62.9100,55.0800,57.6800,44.2200,43.7500 +15,61.9900,53.1300,64.7200,64.5400,55.5400,58.7800,46.1900,43.8100 +16,62.8800,54.1800,64.6800,64.5000,56.1400,59.5500,47.3300,44.6300 +17,64.2200,54.2100,64.6500,64.0200,58.1600,59.2500,48.2200,46.1200 +18,64.5600,56.0400,66.4400,65.3400,58.8900,60.0400,49.0500,47.2800 +19,66.1500,57.0800,65.4800,65.7500,60.2900,61.9800,50.4600,48.5500 +20,65.8500,57.3500,67.3500,67.7300,60.8800,61.5700,51.1600,49.8500 +21,66.5700,59.1000,66.6600,66.1700,61.6600,61.8300,52.4000,50.1900 +22,67.1500,59.9400,67.0800,68.4900,62.2600,61.3100,52.2600,51.8900 +23,68.5400,60.8400,67.6000,68.3800,62.6900,63.6800,53.6400,52.3600 +24,68.2700,61.9200,67.3400,68.7400,62.5500,62.0900,54.9200,53.0900 +25,68.8100,63.4400,68.3800,68.7800,64.0300,62.7800,55.0100,55.4200 +26,70.1300,63.7700,68.5400,67.2700,65.0500,64.6900,57.3300,56.7900 +27,70.4700,63.5800,68.0300,69.0600,65.3500,63.4400,57.8000,57.3600 +28,70.4700,64.7300,69.6700,69.5500,65.5400,63.7700,59.0200,58.5000 +29,70.6000,65.0700,70.7000,69.7000,66.7900,64.3900,60.2800,59.6300 +30,71.0800,66.1900,69.5100,70.0100,66.4700,64.7600,60.4700,59.1000 +31,70.7700,66.3900,68.5600,70.1800,67.1900,65.8500,61.5500,60.4400 +32,71.9100,66.7200,69.0200,69.3100,67.7600,64.7400,62.0700,61.3800 +33,72.0300,68.1800,70.0700,70.4200,68.6300,65.0900,62.6800,62.1600 +34,72.5900,68.4700,70.3800,69.7800,69.0900,65.3600,64.4400,60.9100 +35,71.7900,69.1100,70.0200,69.3900,68.5900,65.2500,63.7200,61.7300 +36,72.5400,70.0300,70.5400,70.5600,68.6200,66.3200,64.8800,62.5000 +37,73.2400,70.2000,70.6900,70.7700,69.5600,66.6200,64.8400,62.8300 +38,73.5100,71.1000,70.5700,70.7700,69.9700,66.9600,66.0600,64.0100 +39,72.7200,70.5800,72.0200,71.7700,70.1200,66.0200,65.9500,63.2600 +40,72.8800,70.9300,70.3800,71.5900,70.3100,65.9000,65.8900,63.3500 +41,73.4900,70.9400,72.0800,72.2000,70.9100,66.5400,67.4800,63.7000 +42,74.2100,71.9600,71.2100,71.5600,70.8000,67.7200,67.5500,63.5700 +43,73.7200,71.4400,70.8500,72.0200,70.7000,67.9800,68.0100,64.6500 +44,75.1400,72.1100,70.7200,72.3000,71.4100,66.5300,67.4700,64.3600 +45,74.2200,72.2300,72.6000,72.6400,71.1600,66.7000,68.0700,64.6500 +46,74.5400,73.1800,71.1400,72.7000,71.2700,68.2900,68.5600,64.4600 +47,75.4900,73.0300,72.2000,72.1100,71.7200,67.9300,67.5500,63.8700 +48,75.3900,73.7000,72.4200,72.7600,71.5300,68.7700,68.9300,65.0000 +49,74.7700,73.3500,71.5300,73.2400,71.1300,69.1700,68.2300,64.5400 +50,75.4500,73.8100,71.7800,72.5700,72.4900,68.3300,68.9500,63.6700 +51,75.6000,74.3600,72.2600,73.1300,72.9400,69.5600,69.0100,64.0300 +52,75.5500,74.2800,72.6400,73.4400,71.5000,69.3000,69.7100,64.9100 +53,75.2900,74.7700,72.6300,72.8300,72.6500,69.3600,69.3700,60.6300 +54,75.7300,74.6000,71.8700,72.8200,73.1200,69.5400,70.1900,64.1400 +55,76.1600,75.1000,72.4400,73.6800,73.6200,68.6500,70.3700,64.7700 +56,76.7200,74.4200,72.5600,73.5800,73.1900,69.4900,69.8000,64.1400 +57,76.4100,75.3500,73.2400,73.4500,73.1700,68.6000,69.4100,64.8500 +58,76.6500,74.6600,72.7400,74.1800,73.4300,68.6600,69.8900,64.7900 +59,76.5500,75.1000,72.5300,73.7400,73.6800,69.8600,70.5800,65.4000 +60,75.6800,75.6900,73.3100,73.5900,73.0800,69.6200,71.6800,62.9200 +61,77.6600,76.2300,72.5200,73.4900,73.8300,70.6100,70.9600,63.9700 +62,76.7500,75.4100,73.1000,73.7200,73.5100,69.3800,70.5800,64.2900 +63,76.3000,75.7300,73.2100,74.0900,73.7400,69.7900,71.2100,65.7900 +64,76.8600,75.9400,73.6900,74.3200,74.0000,69.9400,71.1000,65.8000 +65,78.0000,76.5300,73.5200,73.8300,73.5900,69.2200,71.3500,65.5600 +66,77.3400,76.0700,73.5300,75.1400,74.7400,70.6900,72.2700,65.1800 +67,77.7500,76.3500,72.9900,73.7700,74.5400,69.7800,71.9200,66.6000 +68,76.9000,76.8200,72.9800,74.1900,75.1400,70.2800,72.6600,66.5100 +69,77.4900,76.4600,73.8200,74.6900,74.7900,70.3400,72.7800,66.7500 +70,77.8900,77.4000,73.2800,74.8200,74.7000,70.3200,73.2700,66.7700 +71,77.0600,77.0100,73.8300,75.0000,74.8700,70.8900,71.6000,66.3700 +72,78.4700,77.3100,74.1200,75.3100,75.6400,69.8000,73.0700,65.9600 +73,78.2300,78.2300,74.0200,74.8900,75.5100,71.1300,72.6200,66.0500 +74,78.5800,77.2300,74.0100,74.6400,75.1700,69.9400,73.0500,66.3700 +75,79.0000,77.2600,74.2000,74.8700,75.7000,71.2100,73.2600,66.7100 +76,78.5200,77.2400,73.7300,75.3500,75.6900,70.8400,73.0500,67.2700 +77,78.1600,77.8100,74.9800,75.5500,76.1800,71.4200,73.7000,67.5000 +78,78.0800,77.5900,74.3800,75.7200,75.8100,69.8500,73.4100,66.4700 +79,78.1000,78.5000,73.7800,75.7600,76.1500,70.5100,73.5300,67.5000 +80,78.1900,78.1100,74.2000,75.8300,76.4600,70.6900,73.3800,66.9700 +81,79.0300,77.7700,74.1400,75.7500,76.4600,71.5300,73.3800,67.2300 +82,78.8700,77.8900,74.7600,75.8300,76.4600,71.4800,73.2600,66.8400 +83,79.7800,78.3000,74.2900,75.6700,76.4500,71.6700,73.5000,67.4300 +84,79.4400,78.6800,73.7900,76.1200,76.9800,70.9900,74.7000,68.2700 +85,79.5600,78.2500,74.5700,76.0400,76.3300,72.4600,74.3400,67.8800 +86,79.1300,79.4600,74.3000,75.4800,77.2500,72.0300,74.4000,68.8700 +87,78.7100,78.4500,75.3900,76.0000,77.0400,71.7200,73.4000,68.9300 +88,79.2300,78.4500,75.0600,77.1700,76.9900,72.4400,74.0300,69.5300 +89,79.5400,78.3400,74.5800,76.5000,77.4700,71.1900,74.8700,69.3500 +90,78.7500,78.6700,75.2100,76.8000,76.8900,72.4600,75.6300,69.8000 +91,79.4700,78.6100,75.2900,76.5500,77.2000,72.3500,74.7800,70.0500 +92,79.5600,79.2200,74.9400,77.1200,77.4100,72.2900,73.7400,70.6300 +93,80.1900,78.9400,75.5400,75.8800,78.1200,72.6400,75.4600,70.5500 +94,79.9700,79.5000,74.9400,76.3000,77.9100,72.5200,74.6900,71.3600 +95,80.2400,79.7100,74.4000,76.1900,77.9100,73.5000,75.1300,71.4200 +96,79.8000,80.5700,76.0700,76.2400,77.8600,72.6500,76.0100,71.9400 +97,79.9500,79.8800,75.8000,76.6600,78.3300,72.2600,75.0900,71.5400 +98,80.4300,79.3800,75.5500,77.0800,77.6800,72.4700,75.0800,71.6600 +99,80.8100,79.3200,76.3500,77.5300,78.1700,73.2700,75.6500,72.1000 +100,79.6600,79.5800,75.5200,77.1100,77.5700,73.2700,75.7000,72.0800 +101,80.7500,79.8000,76.4600,77.7500,78.8800,73.9600,75.6500,72.2500 +102,80.7300,80.1900,76.5400,76.8900,78.8500,73.2500,76.1900,72.8700 +103,81.2300,80.3900,75.2800,77.2800,78.6900,73.9000,75.9000,73.1400 +104,80.9700,80.5900,76.6100,77.0600,78.0500,74.1600,76.0300,73.6600 +105,80.7600,80.7800,76.7900,77.3000,78.5700,74.4000,76.7500,73.5500 +106,81.3200,80.9800,76.8200,77.6000,78.5600,73.9500,76.1600,74.4100 +107,81.6400,81.1600,76.0400,77.7800,78.9300,73.9500,76.1400,73.6900 +108,81.5100,81.5100,76.6800,77.4700,79.3000,74.0300,76.8300,74.2100 +109,80.9600,80.8500,77.0000,78.1500,79.3000,74.0400,76.3700,72.9600 +110,81.2700,81.4200,76.6000,78.4200,78.5300,73.9100,77.4200,73.5200 +111,80.9400,81.2600,76.3800,77.7600,79.4700,74.2400,77.2500,74.0200 +112,81.2500,81.3800,77.1900,78.2400,79.5700,74.6700,77.2500,74.0900 +113,81.3300,81.6200,77.2900,77.8600,79.0600,75.3100,76.9900,74.5600 +114,81.7400,81.9400,78.1100,78.2300,79.7700,75.7100,76.7900,74.6100 +115,81.6500,81.4800,77.3300,78.7700,79.4900,75.5100,77.2900,75.2900 +116,82.0300,80.4600,77.3900,78.4900,79.7600,75.9400,78.2300,74.9400 +117,82.1600,81.3200,78.4900,78.2600,79.4700,75.2100,78.1700,75.1200 +118,81.9200,81.2100,77.5300,78.1700,79.4000,74.8700,77.7700,75.2600 +119,82.6400,81.7000,77.7200,78.7900,79.9600,75.6800,78.0300,75.5800 +120,81.6500,82.3100,78.0100,78.5900,79.7000,75.7400,77.3400,76.1100 +121,82.1900,81.9600,77.8200,79.7600,79.9100,75.8600,78.3800,76.1400 +122,81.6800,82.3900,77.8400,79.6600,80.0900,74.7900,77.7300,75.4900 +123,82.4700,81.6700,77.8600,79.3200,80.0400,76.3500,77.9300,76.0300 +124,82.6700,82.0600,78.4600,79.2500,80.1000,76.0900,78.8500,76.5100 +125,82.6900,82.3400,77.9400,79.3400,80.3900,75.7000,78.7000,76.5400 +126,82.2500,82.3700,78.8300,80.0100,80.6500,76.2900,78.7600,76.9200 +127,82.3500,82.2500,78.8200,79.3600,79.4700,76.6700,77.8100,76.0500 +128,82.8200,82.5100,78.4200,79.3400,80.6600,76.5300,78.7100,76.9400 +129,82.4000,82.3300,79.0100,79.3400,79.9300,76.9200,78.5100,76.9700 +130,82.6000,82.3600,78.8300,79.6800,80.7400,77.2000,78.6900,77.1800 +131,82.5700,82.7100,79.4800,80.4100,80.7400,76.2600,78.6400,77.2100 +132,83.2200,82.8900,79.3600,80.9300,80.7700,77.4600,78.9100,77.7100 +133,83.0100,82.8700,79.0000,79.7600,81.4400,77.4700,78.8000,77.3000 +134,82.5800,83.2400,78.0900,80.0900,81.2100,77.2300,78.5900,78.0200 +135,83.4800,84.0000,79.7600,80.1800,81.3100,77.9500,79.9200,77.6500 +136,83.2600,83.3900,79.4900,80.5600,81.2000,77.1700,79.7000,78.0200 +137,83.0500,83.4200,78.7000,80.2800,81.1800,77.6500,79.8300,78.0800 +138,83.5500,83.5600,78.9700,80.7400,81.1300,77.9000,79.3300,77.9700 +139,83.6300,83.8500,78.8700,80.5800,81.5300,77.7600,79.7600,77.8600 +140,83.4900,82.9400,79.2700,80.0500,81.6700,78.1200,79.7000,77.7700 +141,83.6000,83.6000,79.6000,81.2600,81.0800,77.9300,79.9500,78.2800 +142,83.1300,83.8900,79.9000,81.1800,81.4300,78.3200,79.8900,78.4700 +143,83.2500,84.0000,80.7300,81.0600,81.7100,78.4400,79.7800,78.8000 +144,83.8500,84.0100,80.3800,81.0700,81.9100,78.5400,80.2500,78.6400 +145,84.5700,84.0400,79.8900,81.5900,81.8700,78.8900,80.3300,79.0200 +146,84.1100,83.5600,80.7800,81.5300,82.2500,78.7900,80.4200,79.0900 +147,84.4900,83.9900,80.6200,81.2700,82.2400,78.7800,80.4600,79.0000 +148,84.0100,84.0900,80.3900,81.2300,82.3200,79.0300,80.7000,79.2100 +149,84.0200,84.6400,80.8100,81.0000,82.5300,79.2900,80.5600,79.1700 +150,84.6400,84.2400,80.7100,82.3000,82.3900,79.7200,81.1600,79.6400 +151,84.9000,84.2500,81.1000,82.1400,82.8800,79.7900,81.0400,79.6500 +152,84.7600,84.3800,81.2700,81.7500,82.0400,79.1800,81.3500,79.4400 +153,84.3800,84.3700,81.3100,81.7400,82.3400,79.4300,81.0600,79.6900 +154,84.4100,84.5500,80.9400,81.9700,82.6100,79.8100,81.4600,80.0100 +155,84.8800,85.0800,81.5500,82.7600,82.5300,80.1800,81.4200,79.6200 +156,84.6000,84.4900,80.9400,82.2700,82.3900,80.0500,81.0900,80.1300 +157,84.8400,85.1000,81.7100,82.2900,82.4100,80.5900,81.2100,80.2900 +158,85.1000,84.8300,81.7800,82.3500,82.4000,80.4000,81.4900,79.9500 +159,85.1100,84.9000,82.4700,82.6000,82.9000,80.3600,81.5200,80.6900 +160,84.9700,85.0900,82.0500,82.2600,83.2700,80.6200,81.3600,80.4000 +161,85.6100,84.9900,81.9900,82.8300,83.2800,81.2800,81.7300,80.3000 +162,85.4500,85.1800,82.3200,82.4800,83.0200,81.1500,82.0400,80.4000 +163,85.4900,85.4900,82.2000,82.4200,83.4300,80.5200,82.1300,80.4800 +164,85.5300,85.4900,82.3400,82.4700,83.6600,80.9900,81.8000,80.6400 +165,85.7300,85.4800,82.6100,82.3000,83.8200,81.3700,82.1600,80.7000 +166,85.7300,85.1600,82.8900,83.4600,83.7400,81.6400,82.2400,80.7900 +167,85.6500,85.3300,82.8400,83.1300,83.4100,81.5900,82.2700,81.1000 +168,85.8300,85.4400,82.7700,83.3800,83.5600,81.5600,82.6800,80.8400 +169,85.8600,85.1400,83.1200,83.2200,83.7200,81.9400,82.4600,80.5500 +170,86.0900,85.4400,83.1900,83.1000,83.5600,82.0100,82.6400,80.8900 +171,85.8200,85.6600,83.2100,83.4200,83.6800,82.0700,82.2900,80.7900 +172,86.1800,85.5500,83.0400,83.9500,84.3800,82.2300,82.7400,80.9800 +173,85.7900,85.6400,83.3800,83.6000,83.8200,82.5500,82.4400,80.9900 +174,85.9000,85.6900,83.3500,84.0000,84.3300,82.3500,82.5800,80.9700 +175,86.1200,85.9700,83.3500,83.9500,83.6900,82.8100,82.6600,80.9600 +176,86.4400,85.8000,84.0600,84.2200,84.1500,82.4800,82.6400,81.2800 +177,86.2900,85.7800,83.9300,83.8700,84.0800,82.3800,83.3200,81.4200 +178,86.6200,85.7800,83.8900,84.2900,84.2800,82.4900,82.8500,81.3200 +179,86.4200,86.1900,84.3700,84.4500,84.3300,82.9300,83.2100,81.5800 +180,86.6700,86.2700,84.2000,84.4600,84.2400,83.0100,82.9200,81.4900 +181,86.4300,86.1800,84.3400,84.3400,84.3000,82.7500,83.0200,81.6700 +182,86.3900,86.0800,84.2600,84.2900,84.3000,82.9500,83.0600,81.5800 +183,86.3600,86.0400,84.1800,84.4100,84.4300,82.9200,83.1800,81.7200 +184,86.4000,86.3300,84.2700,84.5700,84.4600,83.3400,83.3800,81.7600 +185,86.4700,86.4100,84.3100,84.4800,84.3100,83.1700,83.3600,81.8200 +186,86.6800,86.4000,84.4200,84.7600,84.4400,83.3200,83.5200,81.7100 +187,86.5500,86.3200,84.3300,84.7200,84.5800,83.4000,83.2500,81.7900 +188,86.7900,86.2700,84.1800,84.6900,84.4900,83.2900,83.5200,81.7900 +189,86.7300,86.3500,84.6000,84.9500,84.3800,83.3500,83.4100,81.9000 +190,86.7200,86.3200,84.6700,84.9100,84.6100,83.5600,83.4000,81.9200 +191,86.8700,86.3900,84.8100,84.7300,84.5400,83.4800,83.4400,81.9600 +192,86.8000,86.3600,84.8200,84.8000,84.7700,83.7800,83.5000,81.9400 +193,86.7700,86.4400,84.6600,84.8000,84.7200,83.6300,83.4300,81.9300 +194,86.6000,86.3400,84.6800,84.9500,84.6700,83.6000,83.4000,82.0200 +195,86.8000,86.4200,84.7900,84.8500,84.7200,83.7400,83.4700,81.9600 +196,86.8000,86.3800,84.7300,84.9400,84.7200,83.6400,83.5300,81.8700 +197,86.7900,86.3600,84.6800,84.8200,84.5400,83.8300,83.4900,81.9800 +198,86.8200,86.4500,84.7300,84.8800,84.6900,83.7800,83.4400,81.9900 +199,86.8500,86.4800,84.6600,84.8300,84.6000,83.7300,83.4400,81.9500 +200,86.8600,86.4600,84.7900,84.9500,84.6300,83.7500,83.3200,81.9300 diff --git a/experiments/lightly_cifar10/cifar10_knn_curves.png b/experiments/lightly_cifar10/cifar10_knn_curves.png new file mode 100644 index 00000000..ef5d6d52 Binary files /dev/null and b/experiments/lightly_cifar10/cifar10_knn_curves.png differ