--- license: mit language: - en pipeline_tag: text-generation tags: - transformer - gpt2 - tiny - atom - experimental - humor - whirlwindai new_version: WhirlwindAI/SubatomZephyr ---



--- # The Idea
## What if a transformer became... microscopic? AtomZephyr explores one of the smallest practical transformer architectures ever built. Not because anyone asked for it. Because someone eventually had to answer the question: **"How absurdly small can an AI become before it forgets how to AI?"** Turns out... **27 parameters is still technically enough.**
--- # Why? Most AI models compete by getting bigger. AtomZephyr competes by removing parameters until people start questioning whether it's still a neural network. Every parameter had to earn its place. Most didn't. --- # Specifications | Property | Value | |-----------|-------| | Parameters | **27** | | Architecture | GPT-2 | | Layers | 1 | | Attention Heads | 1 | | Embedding Size | 1 | | FFN Size | 1 | | Context Length | 4 | | Vocabulary | 5 Tokens | | Model Size | <5 KB | | Training Time | ~6 Seconds (CPU) | --- # Performance | Test | Result | |------|--------| | Understand English | ❌ | | Write Code | ❌ | | Solve Math | ❌ | | Generate "abba" | ✅ | | Break Expectations | ✅ | --- # Quick Start ```python from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("WhirlwindAI/AtomZephyr") model = AutoModelForCausalLM.from_pretrained("WhirlwindAI/AtomZephyr") prompt = "a" inputs = tokenizer(prompt, return_tensors="pt") outputs = model.generate( **inputs, do_sample=True, temperature=1.7, max_length=4 ) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` Possible output ``` abaa ``` Groundbreaking. --- # Example Conversation **User** > Tell me a joke. **AtomZephyr** ``` abba ``` Technically... that's an answer. --- # Scientific Achievement Removing parameters is easy. Keeping a transformer alive afterwards... isn't. AtomZephyr exists purely to explore the absolute lower limits of transformer architectures while remaining a real, trainable language model. Whether it's useful is a completely different discussion. --- # Awards 🥇 Smallest Model That Still Has Self-Respect 🏆 Best Binary Poetry Generator 🥈 Most Efficient Waste Of Six Seconds 🎖️ Official Representative Of Tiny AI --- # Limitations AtomZephyr should **not** be used for: - Programming - Translation - Question Answering - Homework - Anything important It performs significantly better when asked to do absolutely nothing useful. --- # Fun Facts - Fits inside most PNG images. - Smaller than many neural network tutorials. - Downloads faster than this README loads. - Has fewer parameters than some calculator manuals have pages. --- # License MIT Take it apart. Make it smaller. Break another record. ---
### Built by WhirlwindAI *"Sometimes progress isn't measured in billions... it's measured in what you can remove."*