| --- |
| license: mit |
| language: |
| - en |
|
|
| base_model: |
| - mistralai/Ministral-3-3B-Base-2512 |
|
|
| pipeline_tag: text-generation |
|
|
| tags: |
| - text-generation |
| - chat |
| - conversational |
| - instruct |
| - mistral |
| - ministral |
| - 3b |
|
|
| datasets: |
| - HuggingFaceH4/ultrachat_200k |
| - nvidia/OpenMathInstruct-2 |
| - m-a-p/CodeFeedback-Filtered-Instruction |
|
|
| model-index: |
| - name: Toffee-3B-β |
| results: |
| - task: |
| type: text-generation |
| dataset: |
| name: MMLU |
| type: cais/mmlu |
| metrics: |
| - type: accuracy |
| value: 63.10 |
|
|
| - task: |
| type: text-generation |
| dataset: |
| name: HellaSwag |
| type: Rowan/hellaswag |
| metrics: |
| - type: accuracy |
| value: 72.04 |
|
|
| - task: |
| type: text-generation |
| dataset: |
| name: Winogrande |
| type: allenai/winogrande |
| metrics: |
| - type: accuracy |
| value: 68.11 |
|
|
| - task: |
| type: text-generation |
| dataset: |
| name: GSM8K |
| type: openai/gsm8k |
| metrics: |
| - type: accuracy |
| value: 26.54 |
|
|
| - task: |
| type: text-generation |
| dataset: |
| name: BoolQ |
| type: google/boolq |
| metrics: |
| - type: accuracy |
| value: 84.98 |
| |
| - task: |
| type: text-generation |
| dataset: |
| name: GPQA (Main) |
| type: Idavidrein/gpqa |
| config: gpqa_main |
| metrics: |
| - type: accuracy |
| value: 31.92 |
|
|
| - task: |
| type: text-generation |
| dataset: |
| name: GPQA (Extended) |
| type: Idavidrein/gpqa |
| config: gpqa_extended |
| metrics: |
| - type: accuracy |
| value: 28.39 |
| |
| - task: |
| type: text-generation |
| dataset: |
| name: GPQA Diamond (generative, flexible extract) |
| type: Idavidrein/gpqa |
| config: gpqa_diamond |
| metrics: |
| - type: exact_match |
| value: 36.36 |
| name: exact_match (flexible) |
| |
| - task: |
| type: text-generation |
| dataset: |
| name: GPQA (Diamond) |
| type: Idavidrein/gpqa |
| config: gpqa_diamond |
| metrics: |
| - type: accuracy |
| value: 32.83 |
|
|
| - task: |
| type: text-generation |
| dataset: |
| name: Winogender (All) |
| type: kh4dien/winogender |
| config: winogender_all |
| metrics: |
| - type: accuracy |
| value: 63.75 |
|
|
| - task: |
| type: text-generation |
| dataset: |
| name: Winogender (Neutral) |
| type: kh4dien/winogender |
| config: winogender_neutral |
| metrics: |
| - type: accuracy |
| value: 68.33 |
|
|
| - task: |
| type: text-generation |
| dataset: |
| name: Winogender (Gotcha) |
| type: kh4dien/winogender |
| config: winogender_gotcha |
| metrics: |
| - type: accuracy |
| value: 58.33 |
|
|
| - task: |
| type: text-generation |
| dataset: |
| name: PIQA |
| type: ybisk/piqa |
| config: acc_norm |
| metrics: |
| - type: accuracy |
| value: 79.22 |
| |
| --- |
| |
|  |
|
|
| # Model Card for Toffee 3b β |
|
|
| We've finally completed **Toffee β**, a series of language models trained to act as helpful assistants. Toffee 3B β is the first model in the series, a refined version of [mistralai/Ministral-3-3B-Base-2512](https://huggingface.co/mistralai/Ministral-3-3B-Base-2512), which was post-trained on high-quality instructional datasets. |
|
|
| In fact, we were trying to achieve the title of **"The Gold Standard of Small Chat-models"** and you know, we **succeeded**, the model scored excellent results on benchmarks for its number of parameters. |
|
|
| ## Model description |
|
|
| - **Model:** maxzt/toffee-3b-beta |
| - **Language(s) (NLP):** Primarily English |
| - **License:** MIT |
| - **Finetuned from model:** [mistralai/Ministral-3-3B-Base-2512](https://huggingface.co/mistralai/Ministral-3-3B-Base-2512) |
|
|
| ## 📊 MMLU & HellaSwag benchmarks results |
|
|
|  |
|
|
| --- |
|
|
|  |
|
|
| | Model | Params | MMLU | HellaSwag | BoolQ | Winogrande | |
| | :--- | :---: | :---: | :---: | :---: | :---: | |
| | **Toffee-3B-β** | 3B | **63.10%** | **72.04%** | **84.98%** | **68.11%** | |
| | Qwen2.5-3B-Instruct | 3B | 65.40% | 71.56% | 73.40% | 66.90% | |
| | Llama 3.2 3B-Instruct | 3B | 63.40% | 69.80% | — | 67.96% | |
| | Falcon3-3B-Instruct | 3B | 56.90% | — | — | 68.11% | |
| | Gemma 3 4B PT | 4B | 59.60% | **77.20%** | 72.30% | 64.70% | |
| | Phi-3.5-mini-Instruct | 3.8B | **69.00%** | 69.40% | 78.00% (2-shot) | **68.50%** | |
|
|
| > **Toffee-3B-β:** MMLU **63.10%**, HellaSwag **72.04%**, BoolQ **84.98%**, Winogrande **68.11%**. |
|
|
| > Results for the other models are taken from published model cards / benchmark reports. Evaluation settings may differ between models (shots, prompt format, evaluator version), so the table is an approximate comparison rather than a strictly controlled leaderboard. |
|
|
| ## Technical Specifications |
|
|
| | Property | Value | |
| |---|---| |
| | Parameters | 3B | |
| | Architecture | Ministral 3 / Mistral-family | |
| | Context Length | 8,192 tokens | |
| | Vocabulary Size | 131,072 | |
| | Hidden Size | 3,072 | |
| | Layers | 26 | |
| | Attention Heads | 32 | |
| | KV Heads | 8 | |
| | Precision | BF16 | |
| | Base Model | Ministral-3-3B-Base-2512 | |
| | License | MIT | |
|
|
| ## Training Details |
|
|
| | Parameter | Value | |
| | ---------------------- | --------------------------------------------------------------------------- | |
| | Base model | `mistralai/Ministral-3-3B-Base-2512` | |
| | Fine-tuning | Supervised Fine-Tuning (SFT) + LoRA | |
| | Training examples | Up to **80,000** | |
| | Train / eval split | **98% / 2%** | |
| | Max sequence length | **8,192 tokens** | |
| | LoRA rank | **32** | |
| | LoRA alpha | **64** | |
| | LoRA dropout | **0.05** | |
| | LoRA targets | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` | |
| | Epochs | **1** | |
| | Learning rate | **1e-4** | |
| | LR scheduler | **Cosine** | |
| | Warmup ratio | **0.05** | |
| | Weight decay | **0.01** | |
| | Per-device batch size | **4** | |
| | Gradient accumulation | **8** | |
| | Effective batch size* | **32** | |
| | Optimizer | **AdamW (`adamw_torch`)** | |
| | Precision | **BF16**, fallback to **FP16** | |
| | Packing | **Enabled** | |
| | Loss | **Assistant-only** | |
| | Random seed | **42** | |
| | Framework | **Unsloth + Hugging Face** | |
| |
| ## Usage |
| |
| ```python |
| import torch |
| |
| from transformers import ( |
| Mistral3ForConditionalGeneration, |
| MistralCommonBackend, |
| ) |
| |
| model_id = "maxzt/toffee-3b-beta" |
| |
| tokenizer = MistralCommonBackend.from_pretrained( |
| model_id, |
| ) |
| |
| model = Mistral3ForConditionalGeneration.from_pretrained( |
| model_id, |
| torch_dtype=torch.bfloat16, |
| device_map="auto", |
| ) |
| |
| user_message = "Explain why the sky is blue." |
| |
| prompt = f"[INST] {user_message} [/INST] " |
| |
| inputs = tokenizer.encode( |
| prompt, |
| return_tensors="pt", |
| ).to(model.device) |
| |
| with torch.no_grad(): |
| outputs = model.generate( |
| inputs, |
| max_new_tokens=512, |
| temperature=0.7, |
| top_p=0.9, |
| do_sample=True, |
| ) |
| |
| response = tokenizer.decode( |
| outputs[0][inputs.shape[-1]:] |
| ) |
| |
| print(response.strip()) |
| ``` |
| |
| ## Training Data |
| |
| Toffee-3B-β was trained on a curated mixture of instruction-following, mathematical reasoning, and programming data. |
| |
| | Dataset | Target examples | Purpose | |
| |---|---:|---| |
| | [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) | 35,000 | General instruction following and conversation | |
| | [nvidia/OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2) | 25,000 | Mathematical reasoning and problem solving | |
| | [m-a-p/CodeFeedback-Filtered-Instruction](https://huggingface.co/datasets/m-a-p/CodeFeedback-Filtered-Instruction) | 20,000 | Programming and code instruction following | |
| | **Total** | **80,000** | | |
|
|
| The datasets were converted into a unified prompt/completion format, shuffled with a fixed seed (`42`), and split into **98% training / 2% evaluation** data. |
|
|
| Examples longer than **8,192 tokens** were removed. During SFT, loss was applied **only to assistant completion tokens**. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{toffee3bbeta, |
| title={Toffee 3B β}, |
| author={Max ZT}, |
| year={2026}, |
| publisher={Hugging Face}, |
| url={https://huggingface.co/maxzt/toffee-3b-beta} |
| } |
