| --- |
| base_model: allenai/OLMo-2-1124-7B |
| library_name: transformers |
| tags: [code, olmo2, synthcode, code-editing] |
| --- |
| # OLMo-2 7B, SynthCode round-5 arms, all milestones |
|
|
| Continued-pretraining checkpoints of [allenai/OLMo-2-1124-7B](https://huggingface.co/allenai/OLMo-2-1124-7B) on the |
| SynthCode round-5 synthetic code-edit strategies (ICLR 2027 submission). Each model was trained on one strategy for up to |
| 1B tokens with a warmup-stable-decay schedule; at each milestone a branch was annealed (decay over the next 10% of tokens, |
| min lr ratio 0.005) and that annealed branch is what is stored here. |
|
|
| All 22 arms (11 strategies x weak/strong) at all three milestones (250M, 500M, 1000M): 66 models, ~14.6 GB each in bf16. |
|
|
| Layout: `<arm>/<milestone>/`, e.g. `g1_cleanup_code__edit/500M/`. Load with |
| `AutoModelForCausalLM.from_pretrained("SynthCode-Project/OLMo2-7B-all-milestones", subfolder="g1_cleanup_code__edit/500M")`. |
|
|
| | Arm folder | Strategy | | Arm folder | Strategy | |
| |---|---|---|---|---| |
| | g1_0_baseline_original_code | Raw | | g2_3_plan_oldcode_newcode | Plan+Old | |
| | g1_cleanup_code | Clean | | g2_4_plan_cm_oldcode_newcode | Plan+CM+Old | |
| | g1_jupyter_notebook | Notebook | | g2_5_plan_newcode | Plan | |
| | g1_blog_post | Blog | | g2_6_oldcode_review_newcode_v4 | Old+Review | |
| | g2_1_oldcode_cm_newcode | Old+CM | | g2_7_oldcode_synthcm_newcode_v4 | Old+FormatCM | |
| | g2_2_oldcode_synthcm_newcode | Old+TerseCM | | | | |
|
|
| Weak (control) arms (no suffix) mix the strategy data with 9.8% instruction-replay tokens (evol-codealpaca + |
| Magicoder-OSS); strong arms (`__edit`) additionally mix 6.5% edit-instruction tokens. |
|
|
| Training: LLaMA-Factory, packed 8192-token sequences, lr 2e-5, 10,176 steps = 1B tokens per arm. Prompt format used for |
| instruction data and evaluation: `<|user|>\n{prompt}\n<|assistant|>\n`. Each folder keeps the trainer logs |
| (`trainer_log.jsonl`, `trainer_state.json`, `training_loss.png`) for its anneal. |
|
|
