---
license: mit
pipeline_tag: text-generation
language: en
library_name: transformers
tags:
  - tiny
  - tiny-model
  - slm
  - small-language-model
  - sub-10m
  - llama
  - from-scratch
metrics:
  - perplexity
---

# LDT-10M

A **10,284,480-parameter** LLaMA-style text model, trained from scratch. This is a
verified **first checkpoint** for the LDT-10M request (model-requests #12,
DedeProGames) — real weights, real training, but **undertrained** (see the honest
status below). It is not a quality release yet; the card states that plainly.

## Architecture

Standard LLaMA block, no sliding window, no GQA:

| field | value |
|---|---|
| params (learnable) | **10,284,480** |
| layers | 5 |
| d_model | 320 |
| heads | 5 (head_dim 64) |
| FFN | SwiGLU, inter 896 |
| vocab | 12,288 (gollem byte-level BPE) |
| context | 512 |
| embeddings | tied (lm_head → tok.weight) |
| norm | RMSNorm, pre-norm |
| attention | causal, RoPE (base 10000) |

The parameter count is the learnable total: the raw safetensors sum is 14,216,640,
which double-counts the tied embedding (tok.weight 12288×320 = 3,932,160) that
`head.weight` aliases. Tied, the true count is 10,284,480.

## Training

- **Data:** FineWeb-Edu (~30.1M tokens) + DCLM-baseline (~21.4M tokens) = **~50.5M tokens**.
- **Steps:** 4000 (the requested 2.6B-token budget is **not** met — see below).
- **Final val loss:** 4.6020 (ppl 99.68) on the held-out split.
- Trained from scratch (no base model).

## Honest status: undertrained, not broken

The model learned real context — val loss 4.602 is well below the 7.38 unigram
floor, so it is **not** flatlined and **not** degenerate. But it has seen only
~50.5M tokens ≈ **4.9 tokens/param**, far below the ~10–20+ tok/param small
models typically need for coherent prose. The result is grammatical but
semantically thin: wordy, repetitive, and it drifts off-topic mid-sentence.

Real samples (best.pt, step 4000, temp 0.8, top-k 40) — no broken tokens, no
token-loops:

> "The sun is" → "The sun is assembled by a new study because he is an associate
> of the study of the disease in the early years. I've been interested in having
> a very different study…"

> "Once upon a time" → "Once upon a time, she is an attack. But he is not a good
> idea. But it is something that does not have a moment of his own life…"

> "The cat sat on the" → "The cat sat on the ground. The sunp is a piece of light
> and is not a good deal of tear. The new story of the MD's Ford…"

> "def hello():" → "def hello(): I have a lot of the best. I'm not sure what
> happened to me. I'll be able to do anything, but I think it's been just a lot
> to say…"

**What this is:** a verified, non-degenerate 10M checkpoint that learned grammar
and surface fluency. **What it is not:** a coherent prose model. It is a first
milestone on the path to the requested 2.6B-token model, not the destination.

## Loading

Self-contained `model.py` (no transformers dependency required):

```python
import torch
from model import LDT

m = LDT.from_pretrained("model.safetensors", device="cpu")
ids = torch.tensor([[1, 2, 3]])  # token ids
out = m.generate(ids, max_new_tokens=64, temperature=0.8, top_k=40, seed=0)
```

The tokenizer is a byte-level BPE (`tokenizer.json`, vocab 12288).

## Roadmap

Continuing training toward the 2.6B-token budget is the next step; this repo
will be updated (or a v2 published) once the model is coherent. This checkpoint
is kept public as an honest intermediate, not a finished model.