YCode-LM v4 (100M parameters)
The 100M-parameter YCode-LM, grown from YCode-LM v3 rather than trained from scratch: v3's 13 trained layers were kept and 21 new layers were inserted between them, each a copy of its neighbour with zeroed output projections. The grown model computed exactly what v3 computed, and continued pretraining then trained the new layers.
| Architecture | decoder-only transformer, 34 layers × 512 wide, 8 query / 2 key-value heads (GQA), QK-norm, RoPE, SwiGLU, RMSNorm |
| Parameters | 100,047,616 |
| Context | 1,024 tokens |
| Tokenizer | byte-level BPE, 8,192 tokens (shared with v2/v3) |
| Weights | bfloat16, sharded into model-0000N-of-00005.pt files (each < 45 MB, under GitHub's file limit); model.pt holds the config, the tokenizer and the shard index |
| Continued pretraining | 3,000 steps × 8,192 tokens ≈ 24.6M tokens on top of v3, Muon (lr 0.015) + AdamW (lr 1e-3), warmup-stable-decay with annealing |
| Instruction tuning | 700 packed steps on the corrected 87k-pair instruction set |
| Hardware | 4-core Intel Xeon CPU, ~760 tokens/s, about 13 hours |
Results
All four models were evaluated the same way with ycode-lm eval. Bits per byte is measured on 11
packages none of them saw in training. The code tasks are executed against unit tests.
| v1 (6.9M) | v2 (7.7M) | v3 (40.8M) | v4 (100M) | |
|---|---|---|---|---|
| Bits per byte on held-out code (lower is better) | 1.095 | 0.966 | 0.857 | 0.814 |
| Write a function from scratch, pass@1 (30 problems) | 0 / 30 | 0 / 30 | 0 / 30 | 1 / 30 |
| Fix an injected bug, fix@1 (15 functions) | 0 / 15 | 5 / 15 | 4 / 15 | 5 / 15 |
What this means:
- v4 models unseen code best of all versions: 5% better than v3 and 26% better than v1.
- It is the first YCode-LM to write a correct function from a description:
square(x)→return x * x. That answer also passes extra tests (0, 5, 1.5, -7). One problem out of 30 is a small step, not a reliable skill. - It fixed 5 of 15 injected bugs (
add,is_even,filter_even,celsius_to_fahrenheit,is_palindrome). - The trade-off: on the training corpus's own validation split, v4's pretraining loss ended slightly worse than v3's (1.832 vs 1.805 nats/token). Raising the learning rate again after growing disturbed v3's tuned weights, and 24.6M tokens is far too little to fully train 60M new parameters. It still generalises better to unseen packages (bits per byte above). More training, ideally on a GPU, is the clear next step.
Use
pip install -e ".[local]"
ycode --local models/ycode-lm-v4
Keep all six model*.pt files together. Loading reassembles the shards automatically.