README.md

---
license: mit
language:
  - code
tags:
  - code-generation
  - python
  - instruction-tuned
  - supervised-fine-tuning
  - from-scratch
library_name: pytorch
---

# code-llm-435m · sft — the instruction-tuned variant

The instruction-tuned version of the 435M from-scratch code model whose pretrained weights are in
[`TightX/code-llm-435m-base`](https://huggingface.co/TightX/code-llm-435m-base). Same architecture, same 32,000-token byte-level BPE vocabulary, same 1,024-token
context — only the weights differ. It reads a plain-English request (*"write a function that checks
whether a number is prime"*) or continues a bare `def` signature, and writes Python.

This is the checkpoint the post-training log selects for release: **first of seven candidates on a
task set that never took part in any selection decision**, and still in the top tier on the tasks
that did.

## Architecture

Identical to [`TightX/code-llm-435m-base`](https://huggingface.co/TightX/code-llm-435m-base) — see that card for the full table. In short: 434,680,832
parameters, 22 layers, d_model 1,024 / 16 heads, SwiGLU (d_ff 4,096), RoPE (theta 500,000), RMSNorm,
causal SDPA, untied embeddings, context 1,024, released file in bf16.

## Post-training

| | |
|---|---|
| Base | `base/model.safetensors` — the weight-soup of the pretraining run, not its last step |
| Data | 147 M tokens of **gold-standard instruction data** (only examples whose reference solution passes its unit tests) interleaved with 29.4 M tokens of signature→body pairs, **5:1** |
| Epochs | 1 |
| Learning rate | 2e-5, cosine to 10%, 50-step warmup |
| Batch | micro 8 × 4 accumulation = 32,768 tokens per step |
| Precision | bf16 + TF32 + `torch.compile` |
| Seed | 46 |
| Hardware / time | one RTX 5090 (32 GB), ~90 minutes |

Roughly 30 post-training runs were evaluated to reach this one. The approaches that did **not** work —
instruction-free training, teaching the two objectives in two stages, self-distillation, weight
averaging ("soups"), ratio sweeps — are documented with their numbers, including the mistakes, in the
GitHub repository at https://github.com/TightXu/Simple_Code_LLM.

## Results

| Suite | Protocol | Strict | Lenient |
|---|---|---|---|
| The 11 tuning tasks | 20 attempts per task per prompt style (reported on a 55-point-per-column scale: 82.2 / 110 = 74.7%) | **82.2 / 110** | 85.8 / 110 |
| **12 held-out tasks** (never used for selection) | 10 attempts per task per prompt style | **71.0%** of 240 | 76.5% |

Per column on the tuning suite: natural-language requests **49.8 / 55**, bare signatures **32.5 / 55**.

The held-out row is the point of the exercise. A model that led the tuning board at 81.6 (89.8 on the
5-attempt protocol) finished **last** there — 55.8 against 59–65 for the rest — because part of its
advantage was a formatting habit that those 11 tasks happened to reward. Selecting on a fixed set of
tasks repeatedly is how a fake champion is manufactured; the honest number is the one from tasks the
selection never saw.

## Limits

1. **The benchmark is 23 short algorithmic tasks.** It is a proxy for "writes small functions
   correctly" and says nothing about debugging, library use, or multi-file changes.
2. **Seed variance is the largest effect measured: about ±6 points.** The released checkpoint is a
   single seed, so that is the honest interval around its quality.
3. **The held-out set was used exactly once, but it is only 12 tasks.** It rules out the specific
   failure described above; it does not prove broad generalisation.
4. **`quicksort` and `two_sum` are near-zero for every arm, including this one.** That is a capability
   boundary of a 435M model trained on 31 B tokens, not a tuning miss.
5. **The signature column is sensitive to output convention** — returning a new list versus mutating in
   place — which is why a lenient metric is reported next to the strict one.

## Files

| File | What |
|---|---|
| `model.safetensors` | bf16 weights, 157 tensors — verified bit-identical to the training checkpoint after the fp32→bf16 cast |
| `config.json` | architecture config exactly as stored in the training checkpoint |
| `tokenizer.json` | byte-level BPE, 32,000 tokens (the same tokenizer as `../base`) |
| `SHA256SUMS.txt` | artifact hash |

Loading it requires the model class from the training repository (`src/train.py`, class `CodeLLM` with
`ModelConfig(**config.json)`); a state dict built by hand will not reproduce the forward pass.

## Intended use and limits

Research and education: what a few-hundred-million-parameter model can learn end to end on real data,
and what makes the difference between a checkpoint that looks good and one that is good. Not for
production code generation, not a coding assistant, not a substitute for a competent developer: at
1,024 tokens of context and 435 M parameters it neither plans nor reasons. Trained only on public
code and instruction data; no personal data. Outputs may reproduce patterns (and licensing quirks)
from the training corpus despite filtering and deduplication.

## Links

`DESIGN.md` (decisions and why), `CHALLENGES.md` (every bug and contaminant), the ablation report, the
post-training log with its corrections, and the evaluation harness are in the GitHub repository at
https://github.com/TightXu/Simple_Code_LLM. MIT licensed.