RaceBench-MiniCPM5-v1.1

Quality over quantity — and v1.1 is the receipt.

Full-parameter fine-tune of openbmb/MiniCPM5-1B on RaceBench-v1.1 (376,736 rows: 353,456 distilled + 23,280 premium agent traces at 4x upsample).

The honest headline: v1.1 does NOT beat base. The premium-agent experiment reversed v1.0's reasoning gain (BBH -9.4, z=-12.3) while keeping a smaller math regression (GSM8K -5.7, z=-3.0). The 4x-upsampled long agent traces look harmful to a 1B model. This is a negative result, published honestly - and it cleanly redirects v1.2.

Results: RaceBench-MiniCPM5-v1.1 vs base

lm-evaluation-harness v0.4.12, identical settings head-to-head (fp16, seed 42, single NVIDIA H100). Deltas in points; z = delta / pooled SE; |z| > 2 is significant at ~95%.

Task	v1.1	MiniCPM5-1B (base)	Delta	z
BBH (CoT few-shot, n=6511)	50.94 ± 0.55	60.34 ± 0.53	-9.40	-12.3	significant LOSS
GSM8K (strict-match, n=1319)	34.34 ± 1.31	40.03 ± 1.35	-5.69	-3.0	significant LOSS
MMLU (full, n=14042)	51.78 ± 0.40	53.38 ± 0.40	-1.60	-2.8	significant small LOSS
Winogrande (n=1267)	56.27 ± 1.39	56.35 ± 1.39	-0.08	-0.04	noise
HellaSwag acc_norm (n=10042)	48.41 ± 0.50	48.82 ± 0.50	-0.41	-0.58	noise
TruthfulQA MC2 (n=817)	44.77 ± 1.49	45.99 ± 1.49	-1.22	-0.58	noise
GPQA-Main zeroshot (n=448)	27.90 ± 2.12	27.01 ± 2.10	+0.89	+0.30	noise
ARC-Challenge (n=1172)	34.98 ± 1.39	34.22 ± 1.39	+0.76	+0.39	noise

Reading this correctly

• Significant regressions: BBH -9.4, GSM8K -5.7, MMLU -1.6. Nothing improves significantly.
• Do not compare these absolutes to the v1.0 model card table - that run used different task configs (zero-shot BBH variant, MMLU subset, 2xT4 venue). Within THIS run, both models were evaluated identically, so deltas are apples-to-apples.
• vs v1.0 directionally: v1.0 delivered BBH +2.7 / GSM8K -9.9. v1.1 delivers BBH -9.4 / GSM8K -5.7. Adding 4x-upsampled premium traces traded the reasoning gain away for a smaller math hit. Bad trade at 1B scale.

Why this happened (hypothesis)

The premium agent traces average 46 turns / ~2.7k chars (max 135k). At 1B scale, 6.2% of training mass in very long multi-turn traces appears to have taught verbose drift, not planning. Per the pre-registered decision tree this is Branch B: ablate premium threshold (>=0.85 or >=0.90 instead of >=0.8), cap trace length (<8192 chars, <=12 messages), re-test.

Deployment implications (read before adopting)

• Do not use v1.1 over the base model today. Every significant delta is negative.
• Use saidutta69/RaceBench-MiniCPM5 (v1.0) if you want the reasoning-gain checkpoint - with its documented GSM8K cost.
• IFEval/HumanEval/MBPP were attempted on H100 and are pending harness fixes; not reported here and no claims are made about instruction-following or code.

Training details

• Base model: openbmb/MiniCPM5-1B
• Dataset: saidutta69/RaceBench-v1.1
• Method: full-parameter SFT, seq_len 4096, sample packing, assistant-only loss masking
• Precision: bf16, SDPA attention, adamw_bnb_8bit
• Schedule: lr 2e-05 cosine, warmup 100, wd 0, seed 42, effective batch 8 x 4096 tokens (32k tok/step), 1 epoch = 10,445 steps, final train_loss 1.301
• Data volume: 376,736 rows -> 342.2M tokens -> 83,557 packed sequences
• Hardware: NVIDIA H100 80GB, ~3h wall

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "saidutta69/RaceBench-MiniCPM5-v1.1",
    trust_remote_code=True,
    torch_dtype="bfloat16",
)
tokenizer = AutoTokenizer.from_pretrained("saidutta69/RaceBench-MiniCPM5-v1.1")

License

Apache-2.0 (base model weights); RaceBench data is MIT.

Citation

@misc{racebench-minicpm5-v11,
  author = {Sai Dutta},
  title = {RaceBench-MiniCPM5-v1.1 -- Premium-Agent Ablation Checkpoint},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/saidutta69/RaceBench-MiniCPM5-v1.1}}
}