📄 Paper | 📝 Blog | 🚀 Recipe Leaderboard | 📊 Benchmark (MMLBD-C)

OriOn-Mistral

OriOn-Mistral is a LC version of Mistral-Small-3.1-24B-Instruct trained with CPT + SFT on synthetic data for long-context visual document performance (PDF VQA / multi-page reasoning) while also massively boosting text long-context capabilities.

---

Highlights

• Strong long-document VQA gains 46.6 on MMLongBenchDoc (+16.8%)
• Strong text long-context gains 53.1 on HELMET (+43.5%).
• 344K context length with training on up to 336-page documents.
• Drop-in serving with vLLM: vllm serve lightonai/OriOn-Mistral (see Serving below).
• Reproducibility + ablations available via the checkpoint leaderboard.

---

Related

• Checkpoint Leaderboard: lightonai/OriOn-Leaderboard includes extensive information for exploration and reproducibility of our training recipes.
• OriOn-Qwen: lightonai/OriOn-Qwen LongPO short-stage finetune on Qwen3-VL-32B-Instruct for long-document QA and reasoning over PDFs.
• Manually Corrected MMLongBenchDoc: lightonai/MMLBD-C improves upon MMLongBenchDoc by flagging inconsistencies between the question, answer and source document. We correct errors related to typos, poor grammar, incorrect question-document pairing and ambiguous phrasing.

---

Benchmarks

Scores (accuracy / task metric, higher is better).

The table below compares OriOn-Mistral to a strong Mistral baseline and other key checkpoints from our paper.

Model / checkpoint	VA	LCA	MMLBD-C	MMLB 128K	SlideVQA	Helmet	LongBench v2	DUDE
OriOn-Qwen (LongPO)	94.6	93.1	56.4	75.6	75.5	62.9	42.0	56.0
OriOn-Mistral (Plain Distill)	84.9	83.0	47.4	65.7	71.2	53.1	38.0	54.0
Mistral 3.1 Small (24B)	80.2	76.7	41.4	66.4	67.8	37.0	39.0	52.8

---

Intended use

OriOn-Mistral is intended for:

• Long PDF / slide-deck QA and understanding with one-shot QA over the full document.
• Long-context reasoning where visual long-context training improves both visual and text long-context behavior.

---

Training details (high level)

• Method: Long document and LC CPT extends context length to 344K. SFT with challenging synthetic single-page and multi-page questions on long PDFs and Qwen3-VL-235B-A22B-Instruct as a teacher model.
• Context length: 344K tokens. Mistral’s RoPE θ is already 1e9, so we do not modify it during CPT for context extension.
• Documents: up to 336 pages

---

Serving

We recommend serving with vLLM (adjust for your setup):

vllm serve lightonai/OriOn-Mistral -tp 2 --quantization fp8

---

Citation

If you use OriOn-Mistrak or MMLBD-C in your work, please cite:

@misc{orion_longdoc_vlm_2026,
  title={How to Train Your Long-Context Visual Document Model}, 
  author={Austin Veselka},
  year={2026},
  eprint={2602.15257},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2602.15257}, 
}
@misc{mistral31small,
      title={Mistral Small 3.1},
      year={2025},
      author={MistralAI},
}
@misc{mmlbd,
  title={MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations},
  author={Yubo Ma and Yuhang Zang and Liangyu Chen and Meiqi Chen and Yizhu Jiao and Xinze Li and Xinyuan Lu and Ziyu Liu and Yan Ma and Xiaoyi Dong and Pan Zhang and Liangming Pan and Yu-Gang Jiang and Jiaqi Wang and Yixin Cao and Aixin Sun},
  year={2024},
  eprint={2407.01523},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2407.01523},
}