Skip to content

Add BoTorch hybrid tuning workflow for CIFAR search #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
WillCheung2016 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add BoTorch hybrid tuning workflow for CIFAR search #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 26 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,16 +30,26 @@ Install the Python dependencies:
pip install -r requirements.txt
```

The requirement file installs [BoTorch](https://botorch.org/) and
[GPyTorch](https://gpytorch.ai/), which must match the local PyTorch
installation. If you are using a GPU-specific wheel, install the desired
`torch`/`torchvision` versions first so that `pip install -r
requirements.txt` selects compatible builds.

### Usage

Run the script with desired arguments:
```
python train.py [-h] [--random_hparams | --llm] [--rounds ROUNDS] [--search_space SEARCH_SPACE]
[--dataset_dir DATASET_DIR] [--arch ARCH] [--save_dir SAVE_DIR]
python train.py [-h] [--random_hparams | --llm] [--llm_rounds LLM_ROUNDS]
[--bo_rounds BO_ROUNDS] [--bo_batch_size BO_BATCH_SIZE] [--hybrid_bo]
[--search_space SEARCH_SPACE] [--dataset_dir DATASET_DIR]
[--arch ARCH] [--save_dir SAVE_DIR]
[--train_batch_size TRAIN_BATCH_SIZE] [--eval_batch_size EVAL_BATCH_SIZE]
[--learning_rate LEARNING_RATE] [--weight_decay WEIGHT_DECAY] [--label_smoothing LABEL_SMOOTHING]
[--optimizer OPTIMIZER] [--num_train_epochs NUM_TRAIN_EPOCHS] [--seed SEED]
[--max_train_batches MAX_TRAIN_BATCHES] [--max_eval_batches MAX_EVAL_BATCHES]
[--learning_rate LEARNING_RATE] [--weight_decay WEIGHT_DECAY]
[--label_smoothing LABEL_SMOOTHING] [--optimizer OPTIMIZER]
[--num_train_epochs NUM_TRAIN_EPOCHS] [--seed SEED]
[--max_train_batches MAX_TRAIN_BATCHES]
[--max_eval_batches MAX_EVAL_BATCHES]
```

To run the LLM-based hyperparameter search with a single seed:
Expand All @@ -48,6 +58,17 @@ python train.py --llm --seed SEED
```
By default, this tunes the optimizer, learning rate, batch size, weight decay, and label smoothing. GPT-4 generated some ranges which we found reasonable. To reproduce our results, run the command with five different random seeds.

### Hybrid LLM + Bayesian optimization workflow

Enable `--hybrid_bo` to hand off the LLM suggestions to a [BoTorch](https://botorch.org/)-powered Bayesian optimization stage. The `--llm_rounds` flag controls the number of configurations proposed by the LLM, while `--bo_rounds` determines how many additional candidates are generated by Bayesian optimization. The BoTorch helper evaluates candidates in batches of `--bo_batch_size` (default 1).

```bash
python train.py --llm --hybrid_bo --llm_rounds 4 --bo_rounds 6 --bo_batch_size 2 \
--seed 0 --search_space constrained
```

The JSON artifact saved under `--save_dir` records which method (LLM or BoTorch) produced each configuration along with the configured round counts.

To run random hyperparameter search:
```
python train.py --random_hparams --seed SEED
Expand Down
Loading