metadata
license: mit
language: en
datasets:
- EleutherAI/bergson-wikitext-512-chunks
base_model: gpt2
gpt-2-wikitext-chunks-epsroot-1e-6-bs64
GPT-2 (124M) fine-tuned on EleutherAI/bergson-wikitext-512-chunks with the bergson MAGIC trainer, for use as a base model in training data attribution experiments.
Training details
- Optimizer: AdamW (beta1=0.95, beta2=0.975,
eps_root=1e-6, weight_decay=0.01) - Batch size: 64, epochs: 4, seed: 42, precision: fp32
- LR: polynomial schedule, peak 8e-4, warmup 25%, end 8e-5
- Trained with bergson 0.10.1 (git sha
27eb22b3)
Replication
The exact bergson run config is included as bergson_config.yaml. Replay it with:
bergson bergson_config.yaml
A companion model trained identically but with eps_root=0 is available at EleutherAI/gpt-2-wikitext-chunks-epsroot-0-bs64.