| --- |
| license: mit |
| pipeline_tag: text-generation |
| language: en |
| library_name: transformers |
| tags: |
| - tiny |
| - tiny-model |
| - slm |
| - small-language-model |
| - sub-10m |
| - llama |
| - from-scratch |
| metrics: |
| - perplexity |
| --- |
| |
| # LDT-10M |
|
|
| A **10,284,480-parameter** LLaMA-style text model, trained from scratch. This is a |
| verified **first checkpoint** for the LDT-10M request (model-requests #12, |
| DedeProGames) — real weights, real training, but **undertrained** (see the honest |
| status below). It is not a quality release yet; the card states that plainly. |
|
|
| ## Architecture |
|
|
| Standard LLaMA block, no sliding window, no GQA: |
|
|
| | field | value | |
| |---|---| |
| | params (learnable) | **10,284,480** | |
| | layers | 5 | |
| | d_model | 320 | |
| | heads | 5 (head_dim 64) | |
| | FFN | SwiGLU, inter 896 | |
| | vocab | 12,288 (gollem byte-level BPE) | |
| | context | 512 | |
| | embeddings | tied (lm_head → tok.weight) | |
| | norm | RMSNorm, pre-norm | |
| | attention | causal, RoPE (base 10000) | |
| |
| The parameter count is the learnable total: the raw safetensors sum is 14,216,640, |
| which double-counts the tied embedding (tok.weight 12288×320 = 3,932,160) that |
| `head.weight` aliases. Tied, the true count is 10,284,480. |
| |
| ## Training |
| |
| - **Data:** FineWeb-Edu (~30.1M tokens) + DCLM-baseline (~21.4M tokens) = **~50.5M tokens**. |
| - **Steps:** 4000 (the requested 2.6B-token budget is **not** met — see below). |
| - **Final val loss:** 4.6020 (ppl 99.68) on the held-out split. |
| - Trained from scratch (no base model). |
| |
| ## Honest status: undertrained, not broken |
| |
| The model learned real context — val loss 4.602 is well below the 7.38 unigram |
| floor, so it is **not** flatlined and **not** degenerate. But it has seen only |
| ~50.5M tokens ≈ **4.9 tokens/param**, far below the ~10–20+ tok/param small |
| models typically need for coherent prose. The result is grammatical but |
| semantically thin: wordy, repetitive, and it drifts off-topic mid-sentence. |
| |
| Real samples (best.pt, step 4000, temp 0.8, top-k 40) — no broken tokens, no |
| token-loops: |
| |
| > "The sun is" → "The sun is assembled by a new study because he is an associate |
| > of the study of the disease in the early years. I've been interested in having |
| > a very different study…" |
| |
| > "Once upon a time" → "Once upon a time, she is an attack. But he is not a good |
| > idea. But it is something that does not have a moment of his own life…" |
| |
| > "The cat sat on the" → "The cat sat on the ground. The sunp is a piece of light |
| > and is not a good deal of tear. The new story of the MD's Ford…" |
| |
| > "def hello():" → "def hello(): I have a lot of the best. I'm not sure what |
| > happened to me. I'll be able to do anything, but I think it's been just a lot |
| > to say…" |
| |
| **What this is:** a verified, non-degenerate 10M checkpoint that learned grammar |
| and surface fluency. **What it is not:** a coherent prose model. It is a first |
| milestone on the path to the requested 2.6B-token model, not the destination. |
| |
| ## Loading |
| |
| Self-contained `model.py` (no transformers dependency required): |
| |
| ```python |
| import torch |
| from model import LDT |
| |
| m = LDT.from_pretrained("model.safetensors", device="cpu") |
| ids = torch.tensor([[1, 2, 3]]) # token ids |
| out = m.generate(ids, max_new_tokens=64, temperature=0.8, top_k=40, seed=0) |
| ``` |
| |
| The tokenizer is a byte-level BPE (`tokenizer.json`, vocab 12288). |
| |
| ## Roadmap |
| |
| Continuing training toward the 2.6B-token budget is the next step; this repo |
| will be updated (or a v2 published) once the model is coherent. This checkpoint |
| is kept public as an honest intermediate, not a finished model. |
