Post
19
Update: models and code are online.
enwik8 test, bits per byte (lower is better):
• 39.5M params: 1.018, and 0.980 with dynamic evaluation (learning while reading)
• 86.3M params: 1.027 / 0.976
• 85.3M without the memory: 1.075 / 0.999
Honest caveat: the tables hold 82M count entries (a 1.1 GB file) on top of
the parameters. The memory is built by counting, no gradients, but the
model still runs a normal transformer.
Model: Chris4K/zeroflops-enwik8-lm
Code: https://github.com/Ckal/ZeroFLops-paper
enwik8 test, bits per byte (lower is better):
• 39.5M params: 1.018, and 0.980 with dynamic evaluation (learning while reading)
• 86.3M params: 1.027 / 0.976
• 85.3M without the memory: 1.075 / 0.999
Honest caveat: the tables hold 82M count entries (a 1.1 GB file) on top of
the parameters. The memory is built by counting, no gradients, but the
model still runs a normal transformer.
Model: Chris4K/zeroflops-enwik8-lm
Code: https://github.com/Ckal/ZeroFLops-paper