---
base_model: allenai/OLMo-2-1124-7B
library_name: transformers
tags: [code, olmo2, synthcode, code-editing]
---
# OLMo-2 7B, SynthCode round-5 arms, all milestones

Continued-pretraining checkpoints of [allenai/OLMo-2-1124-7B](https://huggingface.co/allenai/OLMo-2-1124-7B) on the
SynthCode round-5 synthetic code-edit strategies (ICLR 2027 submission). Each model was trained on one strategy for up to
1B tokens with a warmup-stable-decay schedule; at each milestone a branch was annealed (decay over the next 10% of tokens,
min lr ratio 0.005) and that annealed branch is what is stored here.

All 22 arms (11 strategies x weak/strong) at all three milestones (250M, 500M, 1000M): 66 models, ~14.6 GB each in bf16.

Layout: `<arm>/<milestone>/`, e.g. `g1_cleanup_code__edit/500M/`. Load with
`AutoModelForCausalLM.from_pretrained("SynthCode-Project/OLMo2-7B-all-milestones", subfolder="g1_cleanup_code__edit/500M")`.

| Arm folder | Strategy | | Arm folder | Strategy |
|---|---|---|---|---|
| g1_0_baseline_original_code | Raw | | g2_3_plan_oldcode_newcode | Plan+Old |
| g1_cleanup_code | Clean | | g2_4_plan_cm_oldcode_newcode | Plan+CM+Old |
| g1_jupyter_notebook | Notebook | | g2_5_plan_newcode | Plan |
| g1_blog_post | Blog | | g2_6_oldcode_review_newcode_v4 | Old+Review |
| g2_1_oldcode_cm_newcode | Old+CM | | g2_7_oldcode_synthcm_newcode_v4 | Old+FormatCM |
| g2_2_oldcode_synthcm_newcode | Old+TerseCM | | | |

Weak (control) arms (no suffix) mix the strategy data with 9.8% instruction-replay tokens (evol-codealpaca +
Magicoder-OSS); strong arms (`__edit`) additionally mix 6.5% edit-instruction tokens.

Training: LLaMA-Factory, packed 8192-token sequences, lr 2e-5, 10,176 steps = 1B tokens per arm. Prompt format used for
instruction data and evaluation: `<|user|>\n{prompt}\n<|assistant|>\n`. Each folder keeps the trainer logs
(`trainer_log.jsonl`, `trainer_state.json`, `training_loss.png`) for its anneal.