YOCTO — World's Smallest Language Model

Yocto is a 484K parameter language model that tells children's stories. It achieves 9.58 perplexity on TinyStories — matching models 2-4× larger.

Key Innovation: Unified Attention

Standard transformers use 3 separate projections (Q, K, V). Yocto uses one unified projection that splits into [seeking|offering|content] bands:

Standard:  Q = W_Q·x,  K = W_K·x,  V = W_V·x   [3d² params]
Unified:   u = W·x → [seeking|offering|content] [d² params]

Result: 67% fewer attention parameters, better perplexity.

Quick Start

import torch
from huggingface_hub import hf_hub_download

# Download model
model_path = hf_hub_download(repo_id="Reinforce-ai/yocto", filename="model.pt")
tokenizer_path = hf_hub_download(repo_id="Reinforce-ai/yocto", filename="tokenizer.json")

# Load and generate (see GitHub for full code)

Performance

Metric	Value
Parameters	484,272
Size (fp16)	946 KB
Attention share	5.7%
Perplexity	9.58
Speed (CPU)	700+ tok/s

Example Output

Prompt: "Once upon a time"

Once upon a time, there was a little girl named Lily. She loved to play with her toys all day long. One day, she found a shiny thing on the shelf. The little girl said, "Look, mommy, look!" Her mommy explained that it's very cool, so Lily and her mommy went to the store to buy some tasty food.

Architecture

Component	Value
Embedding dim	72
Layers	4
Attention heads	3
FFN dim	288
Vocab size	4,000
Context length	512

Live Demo

Try Yocto in your browser: HuggingFace Space

Links

• 🌐 Website: reinforceai.com/yocto
• 💻 GitHub: github.com/reinforceai/yocto
• 📄 Paper: Attention Fields: Unified Projections for Efficient Language Models

Citation

@misc{deshwal2026yocto,
  title={Attention Fields: Unified Projections for Efficient Language Models},
  author={Deshwal, Viraj},
  year={2026},
  url={https://www.reinforceai.com/yocto},
  howpublished={\url{https://github.com/reinforceai/yocto}}
}

License

MIT