AI & ML interests

An open, programmable decision layer for models and compute.

Recent Activity

Chris4K 
posted an update about 12 hours ago
view post
Post
19
Update: models and code are online.

enwik8 test, bits per byte (lower is better):
• 39.5M params: 1.018, and 0.980 with dynamic evaluation (learning while reading)
• 86.3M params: 1.027 / 0.976
• 85.3M without the memory: 1.075 / 0.999

Honest caveat: the tables hold 82M count entries (a 1.1 GB file) on top of
the parameters. The memory is built by counting, no gradients, but the
model still runs a normal transformer.

Model: Chris4K/zeroflops-enwik8-lm
Code: https://github.com/Ckal/ZeroFLops-paper
Chris4K 
posted an update 5 days ago
view post
Post
4,992
🧠 Zero-FLOP Memory

Can a small transformer do more by remembering differently?

39.5M parameters + discrete count tables + document cache.

1.017 BPC on enwik8.

No gradient updates needed to build the memory.

Exploring a different way to scale language modeling: not just more parameters, but better memory.

Paper & code follows ... if someones interested
  • 8 replies
·