--- license: mit language: - en base_model: - mistralai/Ministral-3-3B-Base-2512 pipeline_tag: text-generation tags: - text-generation - chat - conversational - instruct - mistral - ministral - 3b datasets: - HuggingFaceH4/ultrachat_200k - nvidia/OpenMathInstruct-2 - m-a-p/CodeFeedback-Filtered-Instruction model-index: - name: Toffee-3B-β results: - task: type: text-generation dataset: name: MMLU type: cais/mmlu metrics: - type: accuracy value: 63.10 - task: type: text-generation dataset: name: HellaSwag type: Rowan/hellaswag metrics: - type: accuracy value: 72.04 - task: type: text-generation dataset: name: Winogrande type: allenai/winogrande metrics: - type: accuracy value: 68.11 - task: type: text-generation dataset: name: GSM8K type: openai/gsm8k metrics: - type: accuracy value: 26.54 - task: type: text-generation dataset: name: BoolQ type: google/boolq metrics: - type: accuracy value: 84.98 - task: type: text-generation dataset: name: GPQA (Main) type: Idavidrein/gpqa config: gpqa_main metrics: - type: accuracy value: 31.92 - task: type: text-generation dataset: name: GPQA (Extended) type: Idavidrein/gpqa config: gpqa_extended metrics: - type: accuracy value: 28.39 - task: type: text-generation dataset: name: GPQA Diamond (generative, flexible extract) type: Idavidrein/gpqa config: gpqa_diamond metrics: - type: exact_match value: 36.36 name: exact_match (flexible) - task: type: text-generation dataset: name: GPQA (Diamond) type: Idavidrein/gpqa config: gpqa_diamond metrics: - type: accuracy value: 32.83 - task: type: text-generation dataset: name: Winogender (All) type: kh4dien/winogender config: winogender_all metrics: - type: accuracy value: 63.75 - task: type: text-generation dataset: name: Winogender (Neutral) type: kh4dien/winogender config: winogender_neutral metrics: - type: accuracy value: 68.33 - task: type: text-generation dataset: name: Winogender (Gotcha) type: kh4dien/winogender config: winogender_gotcha metrics: - type: accuracy value: 58.33 - task: type: text-generation dataset: name: PIQA type: ybisk/piqa config: acc_norm metrics: - type: accuracy value: 79.22 --- ![image](https://cdn-uploads.huggingface.co/production/uploads/6a535822f7c1c2b44a684b39/l_vqeXocyqDlkrt-QcP6v.png) # Model Card for Toffee 3b β We've finally completed **Toffee β**, a series of language models trained to act as helpful assistants. Toffee 3B β is the first model in the series, a refined version of [mistralai/Ministral-3-3B-Base-2512](https://huggingface.co/mistralai/Ministral-3-3B-Base-2512), which was post-trained on high-quality instructional datasets. In fact, we were trying to achieve the title of **"The Gold Standard of Small Chat-models"** and you know, we **succeeded**, the model scored excellent results on benchmarks for its number of parameters. ## Model description - **Model:** maxzt/toffee-3b-beta - **Language(s) (NLP):** Primarily English - **License:** MIT - **Finetuned from model:** [mistralai/Ministral-3-3B-Base-2512](https://huggingface.co/mistralai/Ministral-3-3B-Base-2512) ## 📊 MMLU & HellaSwag benchmarks results ![image](https://cdn-uploads.huggingface.co/production/uploads/6a535822f7c1c2b44a684b39/othLaqxdT8VRjCKPCQmNa.png) --- ![image](https://cdn-uploads.huggingface.co/production/uploads/6a535822f7c1c2b44a684b39/xun37-L11C2bHU622KcwY.png) | Model | Params | MMLU | HellaSwag | BoolQ | Winogrande | | :--- | :---: | :---: | :---: | :---: | :---: | | **Toffee-3B-β** | 3B | **63.10%** | **72.04%** | **84.98%** | **68.11%** | | Qwen2.5-3B-Instruct | 3B | 65.40% | 71.56% | 73.40% | 66.90% | | Llama 3.2 3B-Instruct | 3B | 63.40% | 69.80% | — | 67.96% | | Falcon3-3B-Instruct | 3B | 56.90% | — | — | 68.11% | | Gemma 3 4B PT | 4B | 59.60% | **77.20%** | 72.30% | 64.70% | | Phi-3.5-mini-Instruct | 3.8B | **69.00%** | 69.40% | 78.00% (2-shot) | **68.50%** | > **Toffee-3B-β:** MMLU **63.10%**, HellaSwag **72.04%**, BoolQ **84.98%**, Winogrande **68.11%**. > Results for the other models are taken from published model cards / benchmark reports. Evaluation settings may differ between models (shots, prompt format, evaluator version), so the table is an approximate comparison rather than a strictly controlled leaderboard. ## Technical Specifications | Property | Value | |---|---| | Parameters | 3B | | Architecture | Ministral 3 / Mistral-family | | Context Length | 8,192 tokens | | Vocabulary Size | 131,072 | | Hidden Size | 3,072 | | Layers | 26 | | Attention Heads | 32 | | KV Heads | 8 | | Precision | BF16 | | Base Model | Ministral-3-3B-Base-2512 | | License | MIT | ## Training Details | Parameter | Value | | ---------------------- | --------------------------------------------------------------------------- | | Base model | `mistralai/Ministral-3-3B-Base-2512` | | Fine-tuning | Supervised Fine-Tuning (SFT) + LoRA | | Training examples | Up to **80,000** | | Train / eval split | **98% / 2%** | | Max sequence length | **8,192 tokens** | | LoRA rank | **32** | | LoRA alpha | **64** | | LoRA dropout | **0.05** | | LoRA targets | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` | | Epochs | **1** | | Learning rate | **1e-4** | | LR scheduler | **Cosine** | | Warmup ratio | **0.05** | | Weight decay | **0.01** | | Per-device batch size | **4** | | Gradient accumulation | **8** | | Effective batch size* | **32** | | Optimizer | **AdamW (`adamw_torch`)** | | Precision | **BF16**, fallback to **FP16** | | Packing | **Enabled** | | Loss | **Assistant-only** | | Random seed | **42** | | Framework | **Unsloth + Hugging Face** | ## Usage ```python import torch from transformers import ( Mistral3ForConditionalGeneration, MistralCommonBackend, ) model_id = "maxzt/toffee-3b-beta" tokenizer = MistralCommonBackend.from_pretrained( model_id, ) model = Mistral3ForConditionalGeneration.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", ) user_message = "Explain why the sky is blue." prompt = f"[INST] {user_message} [/INST] " inputs = tokenizer.encode( prompt, return_tensors="pt", ).to(model.device) with torch.no_grad(): outputs = model.generate( inputs, max_new_tokens=512, temperature=0.7, top_p=0.9, do_sample=True, ) response = tokenizer.decode( outputs[0][inputs.shape[-1]:] ) print(response.strip()) ``` ## Training Data Toffee-3B-β was trained on a curated mixture of instruction-following, mathematical reasoning, and programming data. | Dataset | Target examples | Purpose | |---|---:|---| | [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) | 35,000 | General instruction following and conversation | | [nvidia/OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2) | 25,000 | Mathematical reasoning and problem solving | | [m-a-p/CodeFeedback-Filtered-Instruction](https://huggingface.co/datasets/m-a-p/CodeFeedback-Filtered-Instruction) | 20,000 | Programming and code instruction following | | **Total** | **80,000** | | The datasets were converted into a unified prompt/completion format, shuffled with a fixed seed (`42`), and split into **98% training / 2% evaluation** data. Examples longer than **8,192 tokens** were removed. During SFT, loss was applied **only to assistant completion tokens**. ## Citation ```bibtex @misc{toffee3bbeta, title={Toffee 3B β}, author={Max ZT}, year={2026}, publisher={Hugging Face}, url={https://huggingface.co/maxzt/toffee-3b-beta} }