Pre-to-Post-2/sft_200m_generated_fixed_pe

Chess SFT checkpoints with the positional-encoding fix (20M / 50M / 200M)

Multi-turn chess SFT checkpoints trained after the YaRN positional-encoding fix in sft_trainer.py (2026-09). Before the fix, the block_size / rope_scaling change was applied after from_pretrained(), so runs with a 3072-token context trained with plain RoPE extrapolation while their saved config claimed YaRN. These runs load the config first and build the model from it, so training and inference both use YaRN (factor 3, 1024 → 3072). Thanks to Jens Tuyls for spotting the bug.

Setup

• SFT data: 200m_generated (qwen06b generator, nc8/nt10/nta8), CoT format trajectory_sep_no_labels, 85 train files
• Context: 3072 tokens via YaRN on a 1024-token pretrained base
• LR 3e-4, final checkpoint only (step 771)

Layout

Folders mirror the local save tree. Each model lives at <model_id>/final/, where model_id = C{compute}_{size}_alpha{α}_beta{β}. Alongside final/ are combined_eval.json (when evaluated) and config_snapshot/ (training config). Optimizer states are not included.

from transformers import AutoModelForCausalLM, AutoTokenizer
repo, sub = "Pre-to-Post-2/sft_200m_generated_fixed_pe", "C6p5e18_50m_alpha0.400_beta0.023/final"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub, trust_remote_code=True)
tok = AutoTokenizer.from_pretrained(repo, subfolder=sub, trust_remote_code=True)

Models

• C6p5e18_200m_alpha0.100_beta0.100
• C6p5e18_200m_alpha0.200_beta0.100
• C6p5e18_200m_alpha0.750_beta0.100
• C6p5e18_200m_alpha1.000_beta0.100
• C6p5e18_20m_alpha0.010_beta0.008
• C6p5e18_20m_alpha0.030_beta0.008
• C6p5e18_20m_alpha0.050_beta0.008
• C6p5e18_20m_alpha0.100_beta0.008
• C6p5e18_20m_alpha0.200_beta0.008
• C6p5e18_20m_alpha0.300_beta0.008
• C6p5e18_20m_alpha0.600_beta0.008
• C6p5e18_20m_alpha0.750_beta0.008
• C6p5e18_20m_alpha1.000_beta0.008
• C6p5e18_50m_alpha0.010_beta0.023
• C6p5e18_50m_alpha0.030_beta0.023
• C6p5e18_50m_alpha0.050_beta0.023
• C6p5e18_50m_alpha0.100_beta0.023
• C6p5e18_50m_alpha0.200_beta0.023
• C6p5e18_50m_alpha0.400_beta0.023
• C6p5e18_50m_alpha0.750_beta0.023
• C6p5e18_50m_alpha1.000_beta0.023
• C6p5e19_200m_alpha0.200_beta0.007
• C6p5e19_200m_alpha0.400_beta0.007
• C6p5e19_200m_alpha0.750_beta0.007
• C6p5e19_200m_alpha1.000_beta0.007
• C6p5e19_50m_alpha0.180_beta0.002

C6p5e18_20m_alpha1.000_beta0.008 and C6p5e18_50m_alpha0.200_beta0.023 were retrained on 2026-10-05 and have no combined_eval.json yet.