Atom-1.0

Atom-1.0 is a small (29.1M parameter) GPT-style language model trained from scratch on the TinyStories dataset. It writes short, simple children's stories.

Try it: Atom-1.0 demo Space

Model details

Parameters	29.1M
Architecture	Decoder-only transformer (Llama-style)
Layers / heads / width	8 layers, 8 heads, 512 hidden size
Context length	512 tokens
Vocabulary	8,192 (byte-level BPE, trained on the same data)
Normalization	RMSNorm (pre-norm)
Positions	Rotary embeddings (RoPE)
MLP	SwiGLU
Embeddings	Input and output embeddings tied
Precision	Weights stored as float32

Training

• Data: TinyStories (first 2,000 documents held out for validation), about 400M training tokens.
• Steps: 1,525 optimizer steps at 262,144 tokens per step.
• Optimizer: AdamW (betas 0.9 / 0.95), weight decay 0.1 on weight matrices only, gradient clipping at 1.0.
• Schedule: peak learning rate 6e-4, 200 warmup steps, cosine decay to 6e-5.
• Precision: mixed precision training.

Results

Metric	Value
Validation loss	1.587
Validation perplexity	4.89

Example output

Prompt: One day, a little girl named Lily found a shiny key

One day, a little girl named Lily found a shiny key under her bed. She was very happy and wanted to show her mom the pretty key.

Sampling settings used: temperature 0.8, top-k 50, top-p 0.9, repetition penalty 1.15.

Files

• model.safetensors: final weights
• config.json: model dimensions (D, L, H, T, VOCAB), final step and best validation loss
• tokenizer.json: byte-level BPE tokenizer (EOS token: <|endoftext|>)
• ckpt_last.pt / ckpt_best.pt: PyTorch training checkpoints

Usage

This is a custom architecture, so it does not load with transformers.AutoModel. The full model definition and a working generation loop are in app.py of the demo Space. Load the weights with safetensors.torch.load_model (not load_file), because the embeddings are tied.

Limitations

• Trained only on TinyStories, so it writes in the style of a young child's picture book and has a narrow vocabulary.
• It is not an instruction-following or question-answering model. Start it with a story-style opening.
• It can repeat itself and lose track of characters or objects over longer passages.
• It has not been evaluated for safety or factual accuracy and should not be used for anything beyond experimentation.