license: mit
pipeline_tag: text-generation
language: en
library_name: transformers
tags:
- tiny
- tiny-model
- slm
- small-language-model
- sub-10m
- llama
- from-scratch
metrics:
- perplexity
LDT-10M
A 10,284,480-parameter LLaMA-style text model, trained from scratch. This is a verified first checkpoint for the LDT-10M request (model-requests #12, DedeProGames) — real weights, real training, but undertrained (see the honest status below). It is not a quality release yet; the card states that plainly.
Architecture
Standard LLaMA block, no sliding window, no GQA:
| field | value |
|---|---|
| params (learnable) | 10,284,480 |
| layers | 5 |
| d_model | 320 |
| heads | 5 (head_dim 64) |
| FFN | SwiGLU, inter 896 |
| vocab | 12,288 (gollem byte-level BPE) |
| context | 512 |
| embeddings | tied (lm_head → tok.weight) |
| norm | RMSNorm, pre-norm |
| attention | causal, RoPE (base 10000) |
The parameter count is the learnable total: the raw safetensors sum is 14,216,640,
which double-counts the tied embedding (tok.weight 12288×320 = 3,932,160) that
head.weight aliases. Tied, the true count is 10,284,480.
Training
- Data: FineWeb-Edu (
30.1M tokens) + DCLM-baseline (21.4M tokens) = ~50.5M tokens. - Steps: 4000 (the requested 2.6B-token budget is not met — see below).
- Final val loss: 4.6020 (ppl 99.68) on the held-out split.
- Trained from scratch (no base model).
Honest status: undertrained, not broken
The model learned real context — val loss 4.602 is well below the 7.38 unigram floor, so it is not flatlined and not degenerate. But it has seen only ~50.5M tokens ≈ 4.9 tokens/param, far below the ~10–20+ tok/param small models typically need for coherent prose. The result is grammatical but semantically thin: wordy, repetitive, and it drifts off-topic mid-sentence.
Real samples (best.pt, step 4000, temp 0.8, top-k 40) — no broken tokens, no token-loops:
"The sun is" → "The sun is assembled by a new study because he is an associate of the study of the disease in the early years. I've been interested in having a very different study…"
"Once upon a time" → "Once upon a time, she is an attack. But he is not a good idea. But it is something that does not have a moment of his own life…"
"The cat sat on the" → "The cat sat on the ground. The sunp is a piece of light and is not a good deal of tear. The new story of the MD's Ford…"
"def hello():" → "def hello(): I have a lot of the best. I'm not sure what happened to me. I'll be able to do anything, but I think it's been just a lot to say…"
What this is: a verified, non-degenerate 10M checkpoint that learned grammar and surface fluency. What it is not: a coherent prose model. It is a first milestone on the path to the requested 2.6B-token model, not the destination.
Loading
Self-contained model.py (no transformers dependency required):
import torch
from model import LDT
m = LDT.from_pretrained("model.safetensors", device="cpu")
ids = torch.tensor([[1, 2, 3]]) # token ids
out = m.generate(ids, max_new_tokens=64, temperature=0.8, top_k=40, seed=0)
The tokenizer is a byte-level BPE (tokenizer.json, vocab 12288).
Roadmap
Continuing training toward the 2.6B-token budget is the next step; this repo will be updated (or a v2 published) once the model is coherent. This checkpoint is kept public as an honest intermediate, not a finished model.