WhirlwindAI/AtomZephyr

---

The Idea

What if a transformer became... microscopic? AtomZephyr explores one of the smallest practical transformer architectures ever built. Not because anyone asked for it. Because someone eventually had to answer the question: "How absurdly small can an AI become before it forgets how to AI?" Turns out... 27 parameters is still technically enough.

---

Why?

Most AI models compete by getting bigger.

AtomZephyr competes by removing parameters until people start questioning whether it's still a neural network.

Every parameter had to earn its place.

Most didn't.

---

Specifications

Property	Value
Parameters	27
Architecture	GPT-2
Layers	1
Attention Heads	1
Embedding Size	1
FFN Size	1
Context Length	4
Vocabulary	5 Tokens
Model Size	<5 KB
Training Time	~6 Seconds (CPU)

---

Performance

Test	Result
Understand English	❌
Write Code	❌
Solve Math	❌
Generate "abba"	✅
Break Expectations	✅

---

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("WhirlwindAI/AtomZephyr")
model = AutoModelForCausalLM.from_pretrained("WhirlwindAI/AtomZephyr")

prompt = "a"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    do_sample=True,
    temperature=1.7,
    max_length=4
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Possible output

abaa

Groundbreaking.

---

Example Conversation

User

Tell me a joke.

AtomZephyr

abba

Technically...

that's an answer.

---

Scientific Achievement

Removing parameters is easy.

Keeping a transformer alive afterwards...

isn't.

AtomZephyr exists purely to explore the absolute lower limits of transformer architectures while remaining a real, trainable language model.

Whether it's useful is a completely different discussion.

---

Awards

🥇 Smallest Model That Still Has Self-Respect

🏆 Best Binary Poetry Generator

🥈 Most Efficient Waste Of Six Seconds

🎖️ Official Representative Of Tiny AI

---

Limitations

AtomZephyr should not be used for:

• Programming
• Translation
• Question Answering
• Homework
• Anything important

It performs significantly better when asked to do absolutely nothing useful.

---

Fun Facts

• Fits inside most PNG images.
• Smaller than many neural network tutorials.
• Downloads faster than this README loads.
• Has fewer parameters than some calculator manuals have pages.

---

License

MIT

Take it apart.

Make it smaller.

Break another record.

---

Built by WhirlwindAI

"Sometimes progress isn't measured in billions... it's measured in what you can remove."