# YCode-LM v4 (100M parameters)

The 100M-parameter YCode-LM, **grown from YCode-LM v3** rather than trained from scratch:
v3's 13 trained layers were kept and 21 new layers were inserted between them, each a copy of its
neighbour with zeroed output projections. The grown model computed exactly what v3 computed, and
continued pretraining then trained the new layers.

| | |
| --- | --- |
| Architecture | decoder-only transformer, 34 layers × 512 wide, 8 query / 2 key-value heads (GQA), QK-norm, RoPE, SwiGLU, RMSNorm |
| Parameters | 100,047,616 |
| Context | 1,024 tokens |
| Tokenizer | byte-level BPE, 8,192 tokens (shared with v2/v3) |
| Weights | bfloat16, **sharded** into `model-0000N-of-00005.pt` files (each < 45 MB, under GitHub's file limit); `model.pt` holds the config, the tokenizer and the shard index |
| Continued pretraining | 3,000 steps × 8,192 tokens ≈ 24.6M tokens on top of v3, Muon (lr 0.015) + AdamW (lr 1e-3), warmup-stable-decay with annealing |
| Instruction tuning | 700 packed steps on the corrected 87k-pair instruction set |
| Hardware | 4-core Intel Xeon CPU, ~760 tokens/s, about 13 hours |

## Results

All four models were evaluated the same way with `ycode-lm eval`. Bits per byte is measured on 11
packages none of them saw in training. The code tasks are executed against unit tests.

| | v1 (6.9M) | v2 (7.7M) | v3 (40.8M) | **v4 (100M)** |
| --- | --- | --- | --- | --- |
| Bits per byte on held-out code (lower is better) | 1.095 | 0.966 | 0.857 | **0.814** |
| Write a function from scratch, pass@1 (30 problems) | 0 / 30 | 0 / 30 | 0 / 30 | **1 / 30** |
| Fix an injected bug, fix@1 (15 functions) | 0 / 15 | 5 / 15 | 4 / 15 | **5 / 15** |

**What this means:**
- v4 models unseen code best of all versions: 5% better than v3 and 26% better than v1.
- It is the first YCode-LM to write a correct function from a description:
  `square(x)` → `return x * x`. That answer also passes extra tests (0, 5, 1.5, -7). One problem
  out of 30 is a small step, not a reliable skill.
- It fixed 5 of 15 injected bugs (`add`, `is_even`, `filter_even`, `celsius_to_fahrenheit`,
  `is_palindrome`).
- **The trade-off:** on the training corpus's own validation split, v4's pretraining loss ended
  slightly *worse* than v3's (1.832 vs 1.805 nats/token). Raising the learning rate again after
  growing disturbed v3's tuned weights, and 24.6M tokens is far too little to fully train 60M new
  parameters. It still generalises better to unseen packages (bits per byte above). More training,
  ideally on a GPU, is the clear next step.

## Use

```bash
pip install -e ".[local]"
ycode --local models/ycode-lm-v4
```

Keep all six `model*.pt` files together. Loading reassembles the shards automatically.