TinyBuddy-500K

⚠️ Educational / experimental model. TinyBuddy-500K is a from-scratch tiny Llama-style language model (~547K parameters) trained on a synthetic slice of TinyStories-style text. It is not a useful assistant — it is a working demonstration of training extremely small models from scratch. See the Limitations section.

Model description

TinyBuddy-500K is a very small decoder-only Transformer language model trained on synthetic children's stories in the style of TinyStories. The architecture follows the LLaMA design (RMSNorm, Grouped Query Attention, SiLU MLP, tied embeddings).

Hyperparameter	Value
Parameters	547,296 (~547K)
Layers	2
Attention heads	4
Key-Value heads (GQA)	2
Hidden size	96
MLP intermediate size	384
Context length	512
Vocab size	2,048 (BPE trained from scratch)
Norm	RMSNorm
Activation	SiLU
Position embeddings	Learned absolute
Weight tying	Yes (tied embeddings)
Precision	float32

Training details

• Data: Synthetic TinyStories-style corpus (~128K tokens)
• Tokenizer: Custom byte-level BPE with 2048 vocabulary
• Optimizer: AdamW
• Steps: ~300 steps on CPU
• Hardware: Single CPU core
• Final loss: ~0.17

Usage

This model uses custom modeling code, so you must pass trust_remote_code=True.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "Eeppa/TinyBuddy-500K"

tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
model.eval()

prompt = "Once upon a time, there was a little girl named Lily."
input_ids = tokenizer.encode(prompt, return_tensors="pt")

out = model.generate(input_ids, max_new_tokens=60, temperature=0.8, top_k=50)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Limitations

This model is extremely small and was trained for a very short time on limited data.

What works:

• Basic English patterns and short sentence structure
• Simple story-like generation

What's broken:

• Very limited coherence (usually breaks after 1–2 sentences)
• High repetition
• Poor long-range consistency
• No real reasoning or factual knowledge

This model exists purely for educational purposes to explore the lower limits of language model size.

License

MIT

Citation

@misc{tinybuddy500k,
  title  = {TinyBuddy-500K: An educational ~500K parameter Llama-style model trained on TinyStories},
  year   = {2026},
  note   = {Educational demonstration of extremely small language models.}
}

Built with Llama.