mine-o2-small (355M, base)

mine-o2-small is a 355M-parameter base (pretrained, not instruction-tuned) causal language model from MineAI Technology. This card documents its benchmark results under a single reproducible harness (lm-eval 0.4.13, fp32), with the 124M SFT model mine:o1-nano evaluated the same way as a reference point.

⚠️ Important: evaluation status

In our evaluation, this checkpoint failed the model-health sanity gate. Its WikiText-2 cross-entropy (11.58 nats/token) is worse than uniform random guessing (10.82), and the loss rises with token position. All benchmark scores below are at or near chance level.

The conversion sweep found no better variant, and the converted model matches its reference forward pass, so the most likely cause is the checkpoint itself (weights or conversion), not yet verified against the original JAX model. Until that is confirmed, treat these numbers as a description of the checkpoint as evaluated, not as the true capability of the mine-o2-small architecture.

Model summary

Developer	MineAI Technology
Parameters	355M
Stage	Base (no SFT / no chat tuning)
License	Apache 2.0
Eval precision	float32
Best conversion variant found	rope=half+, tanh-GELU, eps=1e-6
Sanity gate	❌ Failed (WikiText-2 CE 11.58 vs 10.82 uniform)

Evaluation results

All models run with the same harness and settings. Scores are acc_norm where available, otherwise acc. "Better" is declared only when the gap exceeds 2 combined standard errors. Note the comparison is not stage-matched: o2-small is a base model, o1-nano is SFT-tuned.

Leaderboard-style and zero-shot tasks

Task	Protocol	Chance %	o2-small 355M	o1-nano 124M (ref.)	Better
ARC-Challenge	25-shot	25	26.02 ±1.28	21.42 ±1.20	o2-small
HellaSwag	10-shot	25	26.79 ±0.44	28.14 ±0.45	o1-nano
MMLU	5-shot	25	25.37 ±0.37	25.90 ±0.37	tie
TruthfulQA MC2	0-shot	–	46.79 ±1.67	44.47 ±1.55	tie
Winogrande	5-shot	50	51.07 ±1.40	50.36 ±1.41	tie
GSM8K (strict)	5-shot	0	0.00	0.00	–
GSM8K (flexible)	5-shot	0	1.82 ±0.37	2.27 ±0.41	tie
ARC-Challenge	0-shot	25	24.74 ±1.26	23.55 ±1.24	tie
ARC-Easy	0-shot	25	31.65 ±0.95	38.05 ±1.00	o1-nano
BoolQ	0-shot	62.2 (majority)	60.58 ±0.85	42.84 ±0.87	o2-small
HellaSwag	0-shot	25	27.14 ±0.44	28.10 ±0.45	tie
OpenBookQA	0-shot	25	26.00 ±1.96	28.40 ±2.02	tie
PIQA	0-shot	50	56.86 ±1.16	58.81 ±1.15	tie
SciQ	0-shot	25	38.20 ±1.54	63.30 ±1.52	o1-nano
Winogrande	0-shot	50	51.22 ±1.40	51.70 ±1.40	tie
LAMBADA (acc)	0-shot	–	1.67 ±0.18	26.82 ±0.62	o1-nano

Leaderboard v1-style average (6 tasks): o2-small 29.34 (GSM8K strict) / 29.64 (GSM8K flexible), vs o1-nano 28.38 / 28.76. Both averages are close to chance; the 1-point gap is not meaningful. The two o2-small "wins" (ARC-Challenge 25-shot, BoolQ) are marginal: BoolQ is just the majority-class rate.

Language modeling

Model	WikiText-2 word ppl	WikiText-2 CE	LAMBADA ppl	WikiText word ppl (lm-eval)
o2-small 355M	1,399,373	10.58	5,123,713	132,945
o1-nano 124M (ref.)	250.3	4.13	132.3	94.2

Own implementation: non-overlapping 1024-token windows, EOS prefix, whitespace-word perplexity, WikiText detokenizer. A healthy model of this size would be expected to land far below these o2-small values.

Model health (sanity gate)

Model	WikiText-2 CE (uniform 10.82)	Loss by position bucket	Gate
o2-small 355M	11.582	8.33 → 10.27 → 12.02 → 11.64 → 11.61	❌ Failed
o1-nano 124M (ref.)	4.541	7.56 → 6.07 → 4.88 → 4.36 → 4.27	✅ Passed

A healthy model's loss falls as more context is available (as o1-nano's does). o2-small's rises, which points to a broken checkpoint or conversion rather than genuinely weak language modeling.

Behaviour probes (small, indicative only)

40 arithmetic problems and 25 short factual questions, few-shot format for the base model, greedy decoding.

Model	Prompt format	Arithmetic	Facts
o2-small 355M	few-shot	1/40	0/25
o1-nano 124M (ref.)	few-shot	1/40	3/25
o1-nano 124M (ref.)	chat template	18/40	9/25

Known issues

• Fails the sanity gate (loss above uniform); the converted weights may be wrong or the checkpoint may not be trained.
• The converted state dict initially raised a key-mismatch error during preparation.
• Not verified against the original JAX model.
• Base model only: no instruction following or chat behaviour.

Intended use

Research and experimentation only, after the checkpoint has been verified. Not suitable for factual, medical, legal, or safety-critical use.

Limitations

• Scores near chance on every task; GSM8K ≈ 0–2%, LAMBADA accuracy 1.7%.
• Custom probes are small and indicative only.
• No contamination check was run. No GPT-2 medium or Pythia-410M reference runs are included.

Evaluation details

• Harness: lm-eval 0.4.13, transformers 5.16.1, torch 2.11.0+cu128
• Hardware: 2× Tesla T4, float32; full test sets (no sample limit)
• Protocols: ARC-C 25-shot, HellaSwag 10-shot, MMLU 5-shot, Winogrande 5-shot, GSM8K 5-shot, TruthfulQA MC2 0-shot, plus 0-shot ARC-C/E, BoolQ, HellaSwag, OpenBookQA, PIQA, SciQ, Winogrande, LAMBADA
• Converted weights SHA-256: 96e8685922725df66c0862a68505ca5e0a9a8db9290642f27d059d028ec5879b
• Run date: 2026-10-20

Citation

@misc{mineai_o2_small,
  title        = {mine-o2-small: a 355M base language model},
  author       = {{MineAI Technology}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/MineAITechnology/mine-o2-small}}
}