Unify model card: shared English template, remove internal identifiers

README.md

@@ -1,51 +1,151 @@
 ---
 license: mit
-base_model: mikecovlee/tinymixtral
+library_name: transformers
+pipeline_tag: text-generation
+language: en
 tags:
   - tinymixtral
   - moe
+  - mixture-of-experts
   - instruction-tuned
+  - sft
+  - text-generation
+  - custom_code
+model_type: tinymixtral
+base_model: mikecovlee/tinymixtral
+datasets:
+  - allenai/tulu-3-sft-mixture
+  - teknium/OpenHermes-2.5
+  - Open-Orca/SlimOrca
+  - Open-Orca/OpenOrca
+  - HuggingFaceH4/ultrachat_200k
+  - meta-math/MetaMathQA
+  - microsoft/orca-math-word-problems-200k
+  - nvidia/OpenMathInstruct-2
+  - rajpurkar/squad_v2
+  - mandarjoshi/trivia_qa
 ---
 
-# tinymixtral-it
+# tinymixtral-it — instruction-tuned MoE
+
+`tinymixtral-it` (a.k.a. **v3.0-it**) is the **instruction-tuned** version of the MoE flagship
+[`mikecovlee/tinymixtral`](https://huggingface.co/mikecovlee/tinymixtral): 477.5M total / 276.1M
+active parameters (top-2 of 4 experts), 2048 context, 32k SentencePiece vocabulary.
+
+It is the MoE counterpart of the dense
+[`mikecovlee/tinymistral-276m-it`](https://huggingface.co/mikecovlee/tinymistral-276m-it): the
+**exact same 3M-tier SFT recipe** (same data, hyper-parameters and schedule) is applied to both
+the MoE and the dense base, so the two instruction-tuned models can be compared at the same
+active-parameter count and the same SFT budget.
+
+## Model details
+
+| Parameter | Value |
+|-----------|-------|
+| hidden_size | 1024 |
+| num_layers | 16 |
+| Attention | Grouped Query Attention (16 heads / 4 KV heads) |
+| Head dim | 64 |
+| RoPE theta | 1,000,000 |
+| Norm | RMSNorm + per-head QK-Norm (pre-RoPE) |
+| Experts | 4 routed (top-2), aux loss 1e-3 |
+| Expert FFN | SwiGLU, intermediate = 2048 |
+| Vocab size | 32,000 (tied embeddings) |
+| Max position | 2,048 |
+| **Total params** | **~477.5M** |
+| **Active params** | **~276.1M** |
+
+## Training
+
+- **Initialization:** the public base `mikecovlee/tinymixtral` (v3.0).
+- **Data:** 2,168,835 deduplicated and eval-decontaminated English instruction conversations,
+  blended from 10 public sources (Tulu3, OpenHermes, SlimOrca, OpenOrca, UltraChat, MetaMath,
+  OrcaMath, OpenMathInstruct-2, SQuAD2, TriviaQA).
+- **Recipe:** 1 epoch, sequence packing to 1024 tokens, batch 24, bf16, AdamW, lr 2e-5
+  (cosine, 100-step warmup), weight decay 0.1, seed 42.
+
+## Evaluation
+
+Measured with `lm-evaluation-harness` v0.4.12, 0-shot (cuda, bf16).
+
+Harness = mean of the 7 primary metrics (hellaswag acc_norm, piqa acc, winogrande acc,
+arc_easy acc, arc_challenge acc_norm, openbookqa acc_norm, lambada acc).
+
+| Model | 7-task harness | 7-task all-acc | GSM8K strict / flex | IFEval prompt / inst |
+|---|---|---|---|---|
+| **`tinymixtral-it`** (MoE + 3M SFT) | **0.3994** | 0.3691 | 0.0182 / 0.0205 | 0.1756 / 0.2782 |
+| `tinymixtral` (MoE base) | 0.3992 | — | 0.0000 / 0.0159 | — |
+| `tinymistral-276m-it` (dense + 3M SFT) | 0.3892 | 0.3631 | 0.0174 / 0.0205 | 0.1460 / 0.2602 |
+| `tinymistral-276m` (dense base) | 0.3904 | 0.3890 | 0.0000 / 0.0136 | — |
 
-Instruction-tuned **TinyMixtral**: 477.5M total / 276.1M active parameters
-(MoE, top-2 of 4 experts), 2048 context, 32k SentencePiece vocab.
+> **Harness note.** Means quoted here use the **7-task** harness (excludes BoolQ); the base
+> v1.0/v3.0 cards report an **8-task** mean (includes BoolQ). The two are not directly comparable.
 
-Fine-tuned for one epoch from the public `mikecovlee/tinymixtral` base on
-2,168,835 deduplicated and eval-decontaminated English instruction conversations
-blended from public sources (Tulu3, OpenHermes, SlimOrca, OpenOrca, UltraChat,
-MetaMath, OrcaMath, OpenMathInstruct-2, SQuAD2, TriviaQA).
-Sequence packing to 1024 tokens, bf16, AdamW, lr 2e-5 cosine with 100-step warmup.
+Per-task (`tinymixtral-it`): hellaswag 0.340, piqa 0.634, winogrande 0.528, arc_easy 0.448,
+arc_challenge 0.260, openbookqa 0.302, lambada 0.289 — plus **BoolQ 0.426**, reported separately
+(BoolQ is excluded from the 7-task mean above, to match the dense iso-active comparison).
 
-## Load
+Additional metric:
+
+| Metric | Score |
+|---|---|
+| Open-ended answer quality (LLM rubric, 4,955 held-out prompts, 0–100) | 15.0 ± 0.3 |
+
+Note: relative to the base model, instruction following and open-ended answer quality improve
+substantially while multiple-choice common-sense accuracy drops (BoolQ is the most sensitive task).
+Arithmetic remains far below practical use. The dense counterpart receives the same SFT recipe and
+lands close on the 7-task harness (0.3892 vs 0.3994) but is less steerable (IFEval 0.1460 / 0.2602).
+
+## Usage
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-model = AutoModelForCausalLM.from_pretrained(
-    "mikecovlee/tinymixtral-it", trust_remote_code=True, dtype="bfloat16", device_map="cuda"
-)
-tokenizer = AutoTokenizer.from_pretrained("mikecovlee/tinymixtral-it")
+model_id = "mikecovlee/tinymixtral-it"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype="bfloat16", device_map="auto")
+
+messages = [{"role": "user", "content": "What is 12% of 250?"}]
+ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
+out = model.generate(ids, max_new_tokens=256, do_sample=False)
+print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
 ```
 
-Conversational use via the bundled chat template
-(`tokenizer.apply_chat_template`); greedy decoding works well at this scale.
+Conversational use via the bundled chat template (`tokenizer.apply_chat_template`); greedy
+decoding works well at this scale.
 
-## Benchmarks (0-shot)
+## Limitations
 
-| metric | score |
-|---|---|
-| Instruction following (IFEval, prompt-strict / inst-strict) | 0.170 / 0.280 |
-| Open-ended answer quality (LLM rubric, 4,955 held-out prompts, 0-100) | 15.0 ± 0.3 |
-| GSM8K (strict / flexible) | 2.1% / 2.3% |
-| 8-task harness (canonical mean: hellaswag/piqa/arc_c/obqa acc_norm, rest acc) | 0.403 |
+- Absolute numbers are bounded by the 477.5M MoE total / 276.1M active scale and the
+  8.05B-token pretraining budget; arithmetic remains far below practical use.
+- English-centric, no safety alignment.
+- Multiple-choice common-sense accuracy drops slightly after SFT (BoolQ most sensitive).
 
-Per-task harness: hellaswag 0.340, piqa 0.634, winogrande 0.528, arc_easy 0.448,
-arc_challenge 0.260, openbookqa 0.302, boolq 0.426, lambada 0.289.
+## Family
+
+- [`mikecovlee/tinymixtral`](https://huggingface.co/mikecovlee/tinymixtral) — 477.5M MoE base (v3.0 flagship)
+- **`mikecovlee/tinymixtral-it`** — this model (MoE instruction-tuned, v3.0-it, 3M SFT)
+- [`mikecovlee/tinymistral-276m`](https://huggingface.co/mikecovlee/tinymistral-276m) — 276M dense base (iso-active ablation)
+- [`mikecovlee/tinymistral-276m-it`](https://huggingface.co/mikecovlee/tinymistral-276m-it) — 276M dense instruction-tuned (3M SFT)
+- [`mikecovlee/tinymixtral-v1.1-1b`](https://huggingface.co/mikecovlee/tinymixtral-v1.1-1b) — 1B MoE (earlier flagship)
+- [`mikecovlee/tinymixtral-v1.1-0.5b`](https://huggingface.co/mikecovlee/tinymixtral-v1.1-0.5b) — 0.5B-class MoE (data-quality ablation)
+- [`mikecovlee/tinymixtral-v2.0-beta`](https://huggingface.co/mikecovlee/tinymixtral-v2.0-beta) — shared-expert experiment (beta)
+- [`mikecovlee/tinymixtral-v1.0`](https://huggingface.co/mikecovlee/tinymixtral-v1.0) — legacy (C4)
+
+> **Naming.** The MoE family is published under `tinymixtral`; the dense 276M iso-active
+> ablation companions use the `tinymistral` spelling. Both belong to the same project.
+
+## Citation
+
+```bibtex
+@misc{tinymixtralit2026,
+  title  = {TinyMixtral: a small Mixture-of-Experts language-model family},
+  author = {Michael Lee},
+  year   = {2026},
+  howpublished = {\url{https://huggingface.co/mikecovlee/tinymixtral-it}}
+}
+```
 
-Note: relative to the base model, instruction following and open-ended answer
-quality improve substantially while multiple-choice common-sense accuracy drops
-(boolq is the most sensitive task). Arithmetic remains far below practical use.
+## License
 
-License: MIT (Copyright (C) 2026 Michael Lee).
+MIT (Copyright (C) 2026 Michael Lee).