File size: 3,698 Bytes
5ef3875 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | # YCode-LM v5 (1.5B parameters)
The 1.5-billion-parameter YCode-LM, **grown from YCode-LM v4** and trained entirely on a CPU.
v4's 34 layers × 512 channels were widened 4× to 2048 channels: new channels and attention heads
started with zeroed outputs, so the grown model computed exactly what v4 computed (checked: identical
loss on the same batches before training). Continued pretraining then trained the new capacity.
| | |
| --- | --- |
| Architecture | decoder-only transformer, 34 layers × 2048 wide, 32 query / 8 key-value heads (GQA), QK-norm, RoPE, SwiGLU, RMSNorm |
| Parameters | 1,523,202,304 |
| Context | 1,024 tokens |
| Tokenizer | byte-level BPE, 8,192 tokens (shared with v2–v4) |
| Weights | bfloat16, **sharded** into 103 `model-*-of-00103.pt` files (each < 45 MB, under GitHub's file limit); `model.pt` holds the config, the tokenizer and the shard index |
| Continued pretraining | 600 steps × 4,096 tokens ≈ 2.5M tokens, Lion (lr 3e-5), warmup-stable-decay with 20% instruction data during the decay |
| Instruction tuning | 60 packed steps, Lion (lr 1.5e-5) |
| Hardware | 4-core Intel Xeon CPU, 15 GB of RAM. 55–75 tokens/s on a CPU with AMX bf16 units, 8–9 tokens/s on one without |
## Training a 1.5B model in 15 GB of RAM
Weights, gradients and AdamW state for 1.5B parameters take ~24 GB in fp32. v5 used a low-memory
mode (`ycode-lm train --optimizer lion`):
- **Lion** keeps one momentum value per weight (AdamW keeps two), stored in bf16: 3 GB.
- **Updates inside backward:** each weight is updated as soon as its gradient is ready, and the
gradient is freed. A full set of gradients (6 GB) never exists.
- Gradient checkpointing, bf16 autocast without the cast-weight cache, memory-mapped checkpoints.
Peak memory was about 12 GB. Training ran in restartable segments that saved checkpoints, and it
survived three container restarts.
## Results
Evaluated with `ycode-lm eval`, the same way as every earlier version. Bits per byte is measured on
11 Python packages none of the models saw in training. The code tasks are executed against unit tests.
| | v1 (6.9M) | v2 (7.7M) | v3 (40.8M) | v4 (100M) | **v5 (1,523M)** |
| --- | --- | --- | --- | --- | --- |
| Bits per byte on held-out code (lower is better) | 1.095 | 0.966 | 0.857 | 0.814 | **0.758** |
| Write a function from scratch, pass@1 (30 problems) | 0 / 30 | 0 / 30 | 0 / 30 | **1 / 30** | 0 / 30 |
| Fix an injected bug, fix@1 (15 functions) | 0 / 15 | **5 / 15** | 4 / 15 | **5 / 15** | 4 / 15 |
**What this means:**
- v5 models unseen code best of all versions: **7% better than v4**, the largest step between any two
versions, after only ~2.5M tokens of extra training.
- That has not yet turned into solving more tasks. v5 fixed 4 bugs (`total`, `filter_even`,
`celsius_to_fahrenheit`, `is_palindrome`) and wrote no fully correct function. With 30 and 15 tasks,
a difference of one is within noise. Its instruction tuning was cut to 60 steps by the slow CPU.
- A 1.5B model would ideally see tens of billions of training tokens; v5 saw a few million on top of
v4. It is badly under-trained, and more training (ideally on GPUs: `ycode-lm autotrain`, `--shard`)
is the clear next step.
- YCode uses v5 as its default model on machines with at least 4 GB of free memory, and v4 otherwise. v4 is just as good at the benchmark tasks and much faster on a CPU, so `ycode --local v4` is a good choice on slow machines.
## Use
```bash
pip install -e .
ycode # v5 is the default; loads in bfloat16: about 3 GB of RAM
ycode-lm chat --model models/ycode-lm-v5
```
Keep all 104 `model*.pt` files together. Loading reassembles the shards automatically.
|