File size: 2,088 Bytes
5ef3875 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | # YCode-LM v2 (7.7M parameters)
See also the larger [YCode-LM v3 (40.8M)](../ycode-lm-v3/README.md).
A small programming language model trained **from scratch** with this repository's `ycode-lm`
pipeline: its own BPE tokenizer, its own transformer and training loop, no pretrained weights.
It is published here so `ycode --local` works right after cloning, with no API key and no training.
| | |
| --- | --- |
| Architecture | decoder-only transformer, 8 layers × 256 wide, 8 query / 2 key-value heads (GQA), QK-norm, RoPE, SwiGLU, RMSNorm |
| Parameters | 7,738,112 |
| Context | 512 tokens |
| Tokenizer | byte-level BPE, 8,192 tokens (embedded in `model.pt`) |
| Weights | bfloat16 (15.6 MB), converted to float32 on load |
| Pretraining | 10,000 steps, ~82M tokens of Python (176 MB: the standard library plus installed packages), warmup-stable-decay LR, annealing with 20% instruction data |
| Instruction tuning | 2,400 packed steps on 109k pairs generated from real code: write and explain functions, complete stubs, write docstrings, explain classes, fix injected bugs |
| Hardware | 4-core Intel Xeon CPU (bf16, `torch.compile`) |
## Results
Measured with `ycode-lm eval` on 11 packages the model never saw (click, anyio, jsonschema,
markdown_it, starlette, uvicorn, h11, attr, pluggy, filelock, idna):
| | v1 (6.9M) | **v2 (this model)** |
| --- | --- | --- |
| Bits per byte on held-out code (lower is better) | 1.095 | **0.966** |
| pass@1 on 30 executed coding problems | 0 / 30 | 0 / 30 |
| fix@1 on 15 buggy functions (fixes executed against tests) | 0 / 15 | 5 / 15 |
It writes fluent, well-formed Python and follows the instruction formats, but its code is usually
wrong. It is a learning/research model, not a coding assistant. In YCode it runs in answer-only
mode (no tools).
## Use
```bash
pip install -e ".[local]"
ycode --local models/ycode-lm-v2
ycode-lm chat --model models/ycode-lm-v2
ycode-lm eval --model models/ycode-lm-v2 --heldout path/to/code/it/never/saw
```
To train your own (or continue from this one), see "Your own LLM" in the main README.
|