learning rate tuning for frozen models:
- decision: use same lr for aggregation and head!
lr tuning for:
tuning this:
report mean metric over 5 seeds per learning rate
learning rate candidates (maybe):
- ~5 in total, more if the best result is at one of the ends
- stepping: factor of 3
append to your base command to execute all runs at once (replace v* and s* with useful values):
model.learning_rate=v1,v2,v3,v4,v5 seed=s1,s2,s3,s4,s5 --multirun
append this to your srun command to run for 3 days (if it is a partition from your department you can also increase to 5):
cc @tanikina @StalVars @leonhardhennig @harbecke
learning rate tuning for frozen models:
lr tuning for:
frozen pre-trained target-model + frozen bert(see this PR for the configs: add frozen-pretrained+frozen-bert experiment configs #81)tuning this:
model.learning_ratereport mean metric over 5 seeds per learning rate
learning rate candidates (maybe):
append to your base command to execute all runs at once (replace
v*ands*with useful values):append this to your srun command to run for 3 days (if it is a partition from your department you can also increase to 5):
cc @tanikina @StalVars @leonhardhennig @harbecke