File size: 2,642 Bytes
5ef3875 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | # YCode-LM v3 (40.8M parameters)
The largest YCode-LM, trained **from scratch** with this repository's `ycode-lm` pipeline: its own
BPE tokenizer, its own transformer and training loop, no pretrained weights.
| | |
| --- | --- |
| Architecture | decoder-only transformer, 13 layers × 512 wide, 8 query / 2 key-value heads (GQA), QK-norm, RoPE, SwiGLU, RMSNorm |
| Parameters | 40,836,608 |
| Context | 1,024 tokens |
| Tokenizer | byte-level BPE, 8,192 tokens (shared with v2, embedded in `model.pt`) |
| Weights | bfloat16 (81.8 MB), converted to float32 on load |
| Pretraining | 7,300 steps × 8,192 tokens ≈ 60M tokens of Python (same 176 MB corpus as v2), **Muon** optimizer for the transformer matrices + AdamW for embeddings/norms, warmup-stable-decay LR, annealing with 20% instruction data |
| Instruction tuning | 1,400 packed steps on 87k pairs generated from real code. Answers use compact one-line docstrings and only standalone functions for write/complete tasks |
| Hardware | 4-core Intel Xeon CPU (bf16, `torch.compile`), about 10 hours in total |
## Results
All three models were evaluated the same way with `ycode-lm eval`. Bits per byte is measured on 11
packages none of them saw in training (click, anyio, jsonschema, markdown_it, starlette, uvicorn, h11,
attr, pluggy, filelock, idna). The code tasks are executed against unit tests.
| | v1 (6.9M) | v2 (7.7M) | **v3 (40.8M)** |
| --- | --- | --- | --- |
| Bits per byte on held-out code (lower is better) | 1.095 | 0.966 | **0.857** |
| Base model before instruction tuning (pretraining held-out loss) | 2.07 nats/token* | 2.063 | **1.805** |
| Write a function from scratch, pass@1 (30 problems) | 0 / 30 | 0 / 30 | 0 / 30 |
| Fix an injected bug, fix@1 (15 functions) | 0 / 15 | 5 / 15 | 4 / 15 |
\*v1 used a different tokenizer and held-out split, so its per-token loss is not directly comparable.
**What this means:**
- v3 models real code clearly best: it predicts unseen code 11% better than v2 and 22% better than v1.
- It writes short, well-formed answers and can find and fix simple bugs. For example, given
`result = 1` in a summing loop it answers "The bug is in `result = 1`. It should be `result = 0`."
and returns the corrected function.
- It does **not** yet write correct functions from a description. On bug fixing, v2 and v3 are
within noise of each other on 15 problems.
- More data and compute (a GPU) are the way forward. The pipeline supports both unchanged.
In YCode it runs in answer-only mode (no tools).
## Use
```bash
pip install -e ".[local]"
ycode --local models/ycode-lm-v3
ycode-lm chat --model models/ycode-lm-v3
```
|