---
license: mit
language:
  - en
tags:
  - text-generation
  - story-generation
  - tiny-model
  - efficient-attention
  - unified-attention
library_name: pytorch
pipeline_tag: text-generation
model-index:
  - name: yocto
    results:
      - task:
          type: text-generation
        dataset:
          name: TinyStories
          type: roneneldan/TinyStories
        metrics:
          - name: Perplexity
            type: perplexity
            value: 9.58
---

# YOCTO — World's Smallest Language Model

<p align="center">
  <img src="https://img.shields.io/badge/Parameters-484K-blue" alt="Parameters">
  <img src="https://img.shields.io/badge/Size-946KB-green" alt="Size">
  <img src="https://img.shields.io/badge/Speed-700%2B%20tok%2Fs-orange" alt="Speed">
  <img src="https://img.shields.io/badge/Perplexity-9.58-purple" alt="Perplexity">
</p>

Yocto is a 484K parameter language model that tells children's stories. It achieves 9.58 perplexity on TinyStories — matching models 2-4× larger.

## Key Innovation: Unified Attention

Standard transformers use 3 separate projections (Q, K, V). Yocto uses one unified projection that splits into [seeking|offering|content] bands:

```
Standard:  Q = W_Q·x,  K = W_K·x,  V = W_V·x   [3d² params]
Unified:   u = W·x → [seeking|offering|content] [d² params]
```

Result: **67% fewer attention parameters**, better perplexity.

## Quick Start

```python
import torch
from huggingface_hub import hf_hub_download

# Download model
model_path = hf_hub_download(repo_id="Reinforce-ai/yocto", filename="model.pt")
tokenizer_path = hf_hub_download(repo_id="Reinforce-ai/yocto", filename="tokenizer.json")

# Load and generate (see GitHub for full code)
```

## Performance

| Metric | Value |
|--------|-------|
| Parameters | 484,272 |
| Size (fp16) | 946 KB |
| Attention share | 5.7% |
| Perplexity | 9.58 |
| Speed (CPU) | **700+ tok/s** |

## Example Output

**Prompt:** "Once upon a time"

> Once upon a time, there was a little girl named Lily. She loved to play with her toys all day long. One day, she found a shiny thing on the shelf. The little girl said, "Look, mommy, look!" Her mommy explained that it's very cool, so Lily and her mommy went to the store to buy some tasty food.

## Architecture

| Component | Value |
|-----------|-------|
| Embedding dim | 72 |
| Layers | 4 |
| Attention heads | 3 |
| FFN dim | 288 |
| Vocab size | 4,000 |
| Context length | 512 |

## Live Demo

Try Yocto in your browser: [HuggingFace Space](https://huggingface.co/spaces/Reinforce-ai/yocto-demo)

## Links

- 🌐 **Website**: [reinforceai.com/yocto](https://www.reinforceai.com/yocto)
- 💻 **GitHub**: [github.com/reinforceai/yocto](https://github.com/reinforceai/yocto)
- 📄 **Paper**: [Attention Fields: Unified Projections for Efficient Language Models](https://github.com/reinforceai/yocto/blob/main/README.md)

## Citation

```bibtex
@misc{deshwal2026yocto,
  title={Attention Fields: Unified Projections for Efficient Language Models},
  author={Deshwal, Viraj},
  year={2026},
  url={https://www.reinforceai.com/yocto},
  howpublished={\url{https://github.com/reinforceai/yocto}}
}
```

## License

MIT