| --- |
| language: en |
| license: apache-2.0 |
| tags: |
| - causal-lm |
| - code |
| - python |
| - pretrain |
| base_model: gpt2 |
| --- |
| |
| # code-1b-pretrain-v3 |
|
|
| A 1.13B parameter GPT-2 architecture causal language model pretrained from |
| scratch on a curated mix of Python code and programming literature. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| tokenizer = AutoTokenizer.from_pretrained("rovdetection/code-1b-pretrain-v3") |
| model = AutoModelForCausalLM.from_pretrained("rovdetection/code-1b-pretrain-v3") |
| |
| inputs = tokenizer("def fibonacci(n):", return_tensors="pt") |
| out = model.generate(**inputs, max_new_tokens=100, temperature=0.8, do_sample=True) |
| print(tokenizer.decode(out[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Training summary |
|
|
| | | | |
| |---|---| |
| | Architecture | GPT-2 (1.13B params) | |
| | Total steps | 30,000 | |
| | Peak LR | 3e-5 (cosine with warmup) | |
| | Effective batch size | 32 (gradient accumulation) | |
| | Precision | fp16 + 8-bit Adam (bitsandbytes) | |
| | Eval perplexity (held-out Python) | **3.65** | |
|
|
| ## Dataset mix (Phase 4 — final 7k steps) |
|
|
| | Dataset | Weight | |
| |---|---| |
| | bigcode/starcoderdata (Python) | 35% | |
| | codeparrot/codeparrot-clean | 25% | |
| | open-phi/programming_books_llama | 25% | |
| | greengerong/leetcode | 15% | |
|
|
| ## Training phases |
|
|
| | Phase | Steps | LR range | Notes | |
| |---|---|---|---| |
| | 1 | 0 – 10,000 | 0 → 3e-5 | Warmup + early descent | |
| | 2–3 | 10,000 – 23,000 | 3e-5 → 4e-6 | Cosine decay, baseline mix | |
| | 4 | 23,000 – 30,000 | 4e-6 → ~0 | Quality shift: StarCoder ↑35% | |
|
|
| ## Repo structure |
|
|
| The repo root contains inference weights only (`model.safetensors`, tokenizer, |
| `config.json`). The `last-checkpoint/` subfolder contains the full training |
| state (optimizer, scheduler, scaler, RNG) for resuming training. |
|
|
