Pondsiders/le-chaton-floppe

Le Chaton Floppe: a language model that fits on a floppy

Once upon a time we were playing with small language models and wondered if we could create a language model that speaks fluently but that's small enough to fit on a floppy disk.

We are proud to announce: kind of!

---

We started with the work of Eldan and Li 2023 who discovered, among other things, that when it comes to transformer language models, width brings knowledge and depth brings coherence. A tall, narrow model should have decent coherence with a very small vocabulary, like the imagination of a three-year-old.

We won't bore you with the story of how we got there, but we ultimately settled on this architecture.

params	n_layers	d_model	n_heads	d_head	d_ff	vocab	context
2,418,720	6	160	5	32	512	2,048	512

We called this configuration le chaton floppe.

---

The le_chaton_floppe.ipynb notebook is available on GitHub.

The GGUF artifact produced by the notebook is compatible with llama.cpp. Obviously since this model is pre-trained only, you'll want to use llama-completion and not llama-cli to interact with it as shown below.

% llama-completion \
    -hf Pondsiders/le-chaton-floppe \
    --temperature 0.8 \
    --top-p 0.95 \
    --seed 1986 \
    --prompt "Once" 2>/dev/null

Once upon a time, there was a little boy named Tim. Tim loved to play with his toy car. He would push it with his hands and watch it go slow, slow or slow, fast. Tim had so much fun with his toy car.
One day, Tim's toy car broke. He was very sad and wanted to play with it too. He thought, "I need to fix my car, and I can't play with it again." Tim went to his mom and said, "Mom, my car is broken. Can you help me fix it?"
His mom smiled and said, "Of course, Tim. Let's try to fix it together." They worked hard and made a new toy car with a new wheel. Tim was happy that he could fix his car, and he thanked his mom for helping him. From that day on, Tim played with his toy car in the toy car, and it belonged to his mom. [end of text]

As you can see, the model is fluent in English and capable of holding a thread through about the first two paragraphs, but as we get deeper into the sequence coherence drops fast.

Still. Pretty neat that it fits on a floppy!