Post
1,006
🧠 Zero-FLOP Memory
Can a small transformer do more by remembering differently?
39.5M parameters + discrete count tables + document cache.
1.017 BPC on enwik8.
No gradient updates needed to build the memory.
Exploring a different way to scale language modeling: not just more parameters, but better memory.
Paper & code follows ... if someones interested
Can a small transformer do more by remembering differently?
39.5M parameters + discrete count tables + document cache.
1.017 BPC on enwik8.
No gradient updates needed to build the memory.
Exploring a different way to scale language modeling: not just more parameters, but better memory.
Paper & code follows ... if someones interested