---
license: mit
language: en
datasets:
- EleutherAI/bergson-wikitext-512-chunks
base_model: gpt2
---

# gpt-2-wikitext-chunks-epsroot-1e-6-bs64

GPT-2 (124M) fine-tuned on [EleutherAI/bergson-wikitext-512-chunks](https://huggingface.co/datasets/EleutherAI/bergson-wikitext-512-chunks) with the [bergson](https://github.com/EleutherAI/bergson) MAGIC trainer, for use as a base model in training data attribution experiments.

## Training details

- Optimizer: AdamW (beta1=0.95, beta2=0.975, `eps_root=1e-6`, weight_decay=0.01)
- Batch size: 64, epochs: 4, seed: 42, precision: fp32
- LR: polynomial schedule, peak 8e-4, warmup 25%, end 8e-5
- Trained with bergson 0.10.1 (git sha `27eb22b3`)

## Replication

The exact bergson run config is included as [`bergson_config.yaml`](./bergson_config.yaml). Replay it with:

```bash
bergson bergson_config.yaml
```

A companion model trained identically but with `eps_root=0` is available at [EleutherAI/gpt-2-wikitext-chunks-epsroot-0-bs64](https://huggingface.co/EleutherAI/gpt-2-wikitext-chunks-epsroot-0-bs64).