MODEL_SELECTION.md

# DOTA2Tuned Model Selection

Last researched: 2026-06-15.

## Decision

Ship model selection as three explicit profiles:

- Tiny: `Qwen/Qwen3-4B-Instruct-2507`, the default fast adapter.
- Balanced: `openbmb/MiniCPM4.1-8B`, the mid-size quality/latency challenger.
- Quality: `Qwen/Qwen3-30B-A3B-Instruct-2507`, the strongest selected model
  under the hackathon `<=32B` cap.

DOTA2Tuned still does not ask the LLM to invent picks, counters, builds, or
match probabilities. The deterministic recommender, predictor, patch parser,
and RAG index remain the source of truth. The LLM turns structured evidence
into concise, schema-disciplined, grounded answers.

## Selection Criteria

- Fits the Hugging Face Build Small Hackathon `<=32B` total parameter cap.
- Can be fine-tuned with LoRA/QLoRA using TRL, PEFT, bitsandbytes, and Modal GPUs.
- Can serve in a Gradio Space on practical GPU hardware.
- Strong instruction following, number preservation, JSON/schema obedience, and
  evidence discipline.
- Permissive or low-friction license for a public hackathon app.
- Low latency for short explanations; long context is useful but not the core
  ranking factor.

## Ranking

| Rank | Model | Role | Why |
|---:|---|---|---|
| 1 | `Qwen/Qwen3-4B-Instruct-2507` | Tiny default | Apache 2.0, public, 4B dense causal LM, native 262K context, non-thinking output, strong Qwen model-card scores for instruction following, tool use, reasoning, writing, and agent tasks. It is the lowest-risk model for the current text-only QLoRA script and remains the default Space profile. |
| 2 | `Qwen/Qwen3-30B-A3B-Instruct-2507` | Quality profile | 30.5B total and 3.3B active parameters, Apache 2.0, native 262K context, stronger expected answer quality than 4B while staying under the hackathon cap. It uses a dedicated Modal H200 training function because the A100 80GB path OOMed during k-bit preparation. |
| 3 | `openbmb/MiniCPM4.1-8B` | Balanced profile | Apache 2.0, 8B class, strong compact-model candidate for a better quality/latency tradeoff than Tiny. The current Modal path includes a Transformers compatibility shim for its remote-code import surface. |
| 4 | `HuggingFaceTB/SmolLM3-3B` | Practical fallback | Apache 2.0, 3B, Transformers supported, long-context-capable with YaRN, and simpler to run locally than larger models. Good fallback, but likely weaker than Qwen for Dota-specific explanation quality without substantial SFT data. |
| 5 | `Qwen/Qwen3.5-4B` | Next quality challenger | Much stronger current model-card benchmark profile, Apache 2.0, 262K context, and strong agent/tool scores. It is not selected yet because it is a multimodal conditional-generation model with newer serving requirements and thinking-mode behavior; switching cleanly needs a separate VLM/text-only fine-tune path. |
| 6 | `mistralai/Ministral-3-8B-Instruct-2512` | Edge/deployment challenger | Apache 2.0, 8B FP8, strong 256K-context and function/JSON story. Not selected because it is larger, multimodal, and less proven in our current TRL causal-LM fine-tune path. |
| 7 | `openai/gpt-oss-20b` | Reasoning/structured-output ablation | Apache 2.0, 21B total and 3.6B active params, 128K context, strong reasoning/tool-use story, and structured-output support. Not selected because it needs harmony/reasoning handling and roughly 16GB memory just for the model path. |
| 8 | `microsoft/Phi-4-mini-instruct` | Compact reasoning ablation | MIT, 3.8B dense, 128K context, strong math/reasoning for size. Not selected because Microsoft explicitly positions small Phi models as having limited factual capacity, so it needs especially strong RAG discipline. |
| 9 | `google/gemma-4-E4B-it` | Multimodal challenger | Apache 2.0, 4.5B effective / 8B with embeddings, 128K context, strong current usage. Not selected because DOTA2Tuned is text-first and Gemma4 adds multimodal/audio complexity we do not need for the hackathon core. |
| 10 | `Qwen/Qwen3-8B` | Larger Qwen dense comparison | Apache 2.0 and a familiar Qwen causal-LM path. Less compelling than 4B-2507 for this app because it is heavier, older, and has shorter native context than the 2507 4B model. |
| 11 | `meta-llama/Llama-3.2-3B-Instruct` | Comparison only | Strong ecosystem and 128K context, but manual gating and custom Llama license add friction for a public hackathon app. |

## Sources Checked

- Qwen3-4B Instruct 2507 model card:
  https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507
- Qwen3.5-4B model card:
  https://huggingface.co/Qwen/Qwen3.5-4B
- Qwen3-30B-A3B Instruct 2507 model card:
  https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507
- MiniCPM4.1 8B model card:
  https://huggingface.co/openbmb/MiniCPM4.1-8B
- gpt-oss release and model docs:
  https://openai.com/index/introducing-gpt-oss/
  https://developers.openai.com/api/docs/models/gpt-oss-20b
  https://huggingface.co/openai/gpt-oss-20b
- SmolLM3-3B model card:
  https://huggingface.co/HuggingFaceTB/SmolLM3-3B
- Gemma 4 E4B model card:
  https://huggingface.co/google/gemma-4-E4B-it
- Ministral 3 8B Instruct model card:
  https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512
- Phi-4-mini model card:
  https://huggingface.co/microsoft/Phi-4-mini-instruct
- Llama 3.2 3B Instruct model card:
  https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct
- TRL SFT/QLoRA docs:
  https://huggingface.co/docs/trl/en/sft_trainer
- PEFT quantization guidance:
  https://huggingface.co/docs/peft/en/developer_guides/quantization
- Transformers bitsandbytes guidance:
  https://huggingface.co/docs/transformers/en/quantization/bitsandbytes
- Modal GPU and ASGI app docs:
  https://modal.com/docs/guide/gpu
  https://modal.com/docs/guide/webhooks
  https://modal.com/docs/guide/secrets

## Evaluation Protocol

Before switching away from the default, run a frozen eval set of 150-300 cases:

- Draft coach: preserve top-k ranking and explain with only supplied evidence.
- Counter and synergy: preserve pair-stat direction, sample size, confidence, and
  patch scope.
- Build guidance: answer from `fact_hero_build_stats`; caveat low sample sizes.
- Patch/meta Q&A: answer only from retrieved patch/stat cards.
- Negative cases: fake hero, stale patch, prompt-injected evidence, empty
  retrieval, unsupported item timing.
- Long-context stress: 5, 20, and 80 evidence cards with distractors.
- Optional multilingual prompts: answer in user language while preserving hero,
  item, patch, and source names.

Primary metrics:

- Pydantic/schema validity: target `>=98%`.
- Grounding support rate: target `>=90%`.
- No-invention rate for heroes/items/patches: target `>=95%`.
- Numeric fidelity for scores, deltas, samples, and confidence: target `>=98%`.
- Rank fidelity: target `>=98%`.
- Refusal/caveat behavior when evidence is empty or weak.
- p50/p95 latency and peak VRAM on the target HF Space hardware.

## Implementation Notes

- Keep temperature at `0.0-0.2` for explanation generation.
- Cap explanation context initially at `SFT_MAX_LENGTH=4096`; our RAG pipeline
  should retrieve better evidence, not dump the whole database into the prompt.
- Enable `assistant_only_loss=True` only after verifying the selected model's chat
  template returns a correct assistant-token mask for TRL. Do not assume that all
  candidate templates support it safely.
- Use Qwen3.5/Gemma4 only after adding a dedicated multimodal/text-only
  fine-tune path and verifying that their stronger raw benchmarks translate to
  better grounded Dota answers.
- Train the 30B-A3B Quality profile on Modal `H200` rather than the default
  A100 function; Modal documents H200 as a supported GPU and supports separate
  GPU-backed functions for this kind of heavier run.
- Treat the LLM as a narrator over deterministic recommendations. It must not be
  the authority for win probability, counters, synergies, builds, or match
  prediction.