Skip to content

Add AdamW–MuonClip nanoGPT WeightWatcher campaign #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
charlesmartin14 merged 1 commit into from
Aug 23, 2026
Merged

Add AdamW–MuonClip nanoGPT WeightWatcher campaign #115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 25 additions & 12 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,14 @@ python -m pip install -e './baseline[experiment]'
jupyter lab baseline/notebooks
```

Long-running outputs should live under `$HOME`, not `/tmp`:
Put generated data and runs beneath an explicit experiment root. The dated
one-head nanoGPT campaign enforces a `/tmp` root and redirects third-party
caches there as well:

```bash
export RG_BASELINE_DATA_DIR="$HOME/rg-optimizer-data"
export RG_BASELINE_RUN_ROOT="$HOME/rg-optimizer-runs"
export RG_BASELINE_EXPERIMENT_ROOT="/tmp/rg-optimizer-baselines"
export RG_BASELINE_DATA_DIR="$RG_BASELINE_EXPERIMENT_ROOT/data"
export RG_BASELINE_RUN_ROOT="$RG_BASELINE_EXPERIMENT_ROOT/runs"
```

### Recommended notebook order
Expand All @@ -76,16 +79,26 @@ baseline/notebooks/CIFAR10_ViT_Optimizer_Baselines.ipynb
One-head nanoGPT:

```bash
cd baseline/nanogpt_one_head
bash scripts/setup_mac.sh
bash scripts/prepare_data.sh
bash scripts/smoke_test.sh

export RG_NANOGPT_ONE_HEAD_ROOT="$HOME/rg-nanogpt-one-head"
caffeinate -dimsu bash scripts/run_all_baselines.sh \
2>&1 | tee "$RG_NANOGPT_ONE_HEAD_ROOT/run_all.log"
cd baseline/experiments/nanogpt_one_head_2026_08_21_baseline
export RG_NANOGPT_EXPERIMENT_ROOT="/tmp/rg-nanogpt-one-head-20260821"
mkdir -p "$RG_NANOGPT_EXPERIMENT_ROOT"/{cache/{home,pip,xdg/{cache,config,data,state},matplotlib},tmp}
export HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/home"
export PIP_CACHE_DIR="$RG_NANOGPT_EXPERIMENT_ROOT/cache/pip"
export XDG_CACHE_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/cache"
export XDG_CONFIG_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/config"
export XDG_DATA_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/data"
export XDG_STATE_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/state"
export MPLCONFIGDIR="$RG_NANOGPT_EXPERIMENT_ROOT/cache/matplotlib"
export TMPDIR="$RG_NANOGPT_EXPERIMENT_ROOT/tmp"
python -m pip install -e ../../nanogpt_one_head
python scripts/run_experiment.py doctor --device mps
python scripts/run_experiment.py prepare
caffeinate -dimsu python scripts/run_experiment.py run --device mps
```

The full protocol, exact commands, reporting contract, and honest results ledger
are in the [dated experiment folder](baseline/experiments/nanogpt_one_head_2026_08_21_baseline/README.md).

nanochat:

```text
Expand Down Expand Up @@ -200,7 +213,7 @@ The committed values above are strong source-backed centers. They become frozen
best baselines only after the bounded qualification protocol:

1. screen the preregistered neighborhood using validation data only;
2. run the finalists with the complete three-seed protocol;
2. run the finalists with the complete five-seed protocol;
3. select the lowest mean best-validation loss;
4. write the full winning configuration and evidence to a lock file;
5. inspect protected-test comparisons only after the lock exists.
Expand Down
14 changes: 14 additions & 0 deletions baseline/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,17 @@ __pycache__/
.ipynb_checkpoints/
data/
runs/

# The dated nanoGPT campaign deliberately archives only small, reviewed run
# records in the repository. Keep ordinary/raw run directories ignored while
# allowing this one experiment's README and timestamped archive directories.
!experiments/nanogpt_one_head_2026_08_21_baseline/runs/
!experiments/nanogpt_one_head_2026_08_21_baseline/runs/README.md
!experiments/nanogpt_one_head_2026_08_21_baseline/runs/*/
!experiments/nanogpt_one_head_2026_08_21_baseline/runs/*/**
experiments/nanogpt_one_head_2026_08_21_baseline/runs/**/*.pt
experiments/nanogpt_one_head_2026_08_21_baseline/runs/**/*.bin
experiments/nanogpt_one_head_2026_08_21_baseline/runs/**/*.partial*
experiments/nanogpt_one_head_2026_08_21_baseline/runs/.*.partial-*/
experiments/nanogpt_one_head_2026_08_21_baseline/runs/**/cache/
experiments/nanogpt_one_head_2026_08_21_baseline/runs/**/logs/
38 changes: 24 additions & 14 deletions baseline/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,21 +71,20 @@ When installing from the repository root, use:
python -m pip install -e './baseline[experiment]'
```

Set persistent locations before running long jobs:
Set one explicit experiment location before running long jobs:

```bash
export RG_BASELINE_DATA_DIR="$HOME/rg-optimizer-data"
export RG_BASELINE_RUN_ROOT="$HOME/rg-optimizer-runs"
export RG_BASELINE_EXPERIMENT_ROOT="/tmp/rg-optimizer-baselines"
export RG_BASELINE_DATA_DIR="$RG_BASELINE_EXPERIMENT_ROOT/data"
export RG_BASELINE_RUN_ROOT="$RG_BASELINE_EXPERIMENT_ROOT/runs"
```

The isolated one-head nanoGPT suite uses:
The dated one-head nanoGPT campaign uses and validates:

```bash
export RG_NANOGPT_ONE_HEAD_ROOT="$HOME/rg-nanogpt-one-head"
export RG_NANOGPT_EXPERIMENT_ROOT="/tmp/rg-nanogpt-one-head-20260821"
```

Do not put long-running results in `/tmp`.

## 1. MNIST / MLP3

Run these notebooks in order:
Expand Down Expand Up @@ -243,15 +242,26 @@ that produced that checkpoint.
Mac workflow:

```bash
cd nanogpt_one_head
bash scripts/setup_mac.sh
bash scripts/prepare_data.sh
bash scripts/smoke_test.sh

caffeinate -dimsu bash scripts/run_all_baselines.sh \
2>&1 | tee "$RG_NANOGPT_ONE_HEAD_ROOT/run_all.log"
cd experiments/nanogpt_one_head_2026_08_21_baseline
export RG_NANOGPT_EXPERIMENT_ROOT="/tmp/rg-nanogpt-one-head-20260821"
mkdir -p "$RG_NANOGPT_EXPERIMENT_ROOT"/{cache/{home,pip,xdg/{cache,config,data,state},matplotlib},tmp}
export HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/home"
export PIP_CACHE_DIR="$RG_NANOGPT_EXPERIMENT_ROOT/cache/pip"
export XDG_CACHE_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/cache"
export XDG_CONFIG_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/config"
export XDG_DATA_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/data"
export XDG_STATE_HOME="$RG_NANOGPT_EXPERIMENT_ROOT/cache/xdg/state"
export MPLCONFIGDIR="$RG_NANOGPT_EXPERIMENT_ROOT/cache/matplotlib"
export TMPDIR="$RG_NANOGPT_EXPERIMENT_ROOT/tmp"
python -m pip install -e ../../nanogpt_one_head
python scripts/run_experiment.py doctor --device mps
python scripts/run_experiment.py prepare
caffeinate -dimsu python scripts/run_experiment.py run --device mps
```

See the [dated experiment README](experiments/nanogpt_one_head_2026_08_21_baseline/README.md)
for the AdamW/MuonClip five-seed protocol, reporting, and archive steps.

## 4. nanochat

Run:
Expand Down
Loading
Loading