| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| base_model: DedeProGames/Wisp-5M |
| datasets: |
| - HuggingFaceTB/smol-smoltalk |
| tags: |
| - wisp |
| - tiny |
| - instruct |
| - sft |
| - conversational |
| - llama |
| --- |
| |
| # Wisp-5M-Instruct |
|
|
| An instruction-tuned version of [Wisp-5M](https://huggingface.co/DedeProGames/Wisp-5M), a 5.1M-parameter LLaMA-style model pretrained from scratch. It was fully fine-tuned for two epochs on [smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk) with the loss computed only on assistant tokens (end-of-turn tokens weighted 5×, so the model learns to stop). |
|
|
| At 5M parameters this is a tiny research model. It follows the chat format and answers simple requests, but factual knowledge, reasoning and arithmetic are very weak. See the limitations below. |
|
|
| ## Quickstart |
|
|
| Use the recommended sampling settings (they are stored in `generation_config.json`). With near-greedy decoding this model often falls into repetition loops. |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| repo = "DedeProGames/Wisp-5M-Instruct" |
| tokenizer = AutoTokenizer.from_pretrained(repo) |
| model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float32).eval() |
| |
| messages = [{"role": "user", "content": "What is photosynthesis?"}] |
| input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=False) |
| out = model.generate(input_ids, max_new_tokens=256) # temperature 0.7, top-p 0.9, repetition penalty 1.1 from generation_config.json |
| print(tokenizer.decode(out[0, input_ids.shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| Keep prompt + answer within the 2,048-token context. |
|
|
| ## Chat format |
|
|
| The Wisp tokenizer already has `<|im_start|>` and `<|im_end|>`, so the model uses ChatML. `apply_chat_template` renders it: |
|
|
| ```text |
| <|endoftext|><|im_start|>user |
| {user message}<|im_end|> |
| <|im_start|>assistant |
| {assistant reply}<|im_end|> |
| ``` |
|
|
| `<|im_end|>` (id 3) ends a turn, and `<|endoftext|>` (id 0) is the BOS. The model stops on either one. |
|
|
| ## Architecture |
|
|
| Same as Wisp-5M: `LlamaForCausalLM`, 9 layers, hidden size 192, 6 heads with 2 KV heads, SwiGLU (intermediate 512), RMSNorm, RoPE (θ = 10,000), tied embeddings, 8,192-token byte-level BPE vocabulary, 2,048-token context. Weights in BF16 (`model.safetensors`, 20 MB). |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Base | DedeProGames/Wisp-5M | |
| | Data | smol-smoltalk train split, conversations of up to 1,024 tokens (with this tokenizer) | |
| | Epochs | 2 (14,502 steps, batch 32 = 16 × 2 gradient accumulation) | |
| | Optimizer | AdamW (β = 0.9, 0.95), weight decay 0.1 on 2D matrices, grad clip 1.0 | |
| | Schedule | lr 2e-4, 300 warmup steps, cosine to 2e-5 | |
| | Loss | assistant tokens; `<|im_end|>` weighted 5× | |
| | Precision | bf16 autocast, fp32 master weights | |
| | Hardware | 1× NVIDIA RTX 3060 12 GB, 47 minutes | |
|
|
| Held-out loss on assistant tokens (smol-smoltalk test split): 1.791 at the end of training. |
|
|
| ## Evaluation |
|
|
| Benchmarks follow the protocol of the [Kiyo-230M-Instruct](https://huggingface.co/DedeProGames/Kiyo-230M-Instruct) card: FP16, the model's own chat template, no system prompt, new-token limits IFEval 1,024, GPQA 384, LiveBench 512, and the model's full 2,048-token context. The main setting uses temperature 0.1, top-k 50 and repetition penalty 1.05, as on the card (GPQA is sampled at T = 0.6, as on the card). The second setting is the recommended one shipped in `generation_config.json`. |
|
|
| | Benchmark | Metric | Wisp-5M-Instruct, protocol | Wisp-5M-Instruct, recommended sampling | |
| |---|---|---:|---:| |
| | GPQA Diamond | pass@1 (198 × 5 samples, T = 0.6) | 9.80 | not run | |
| | IFEval | prompt-level strict | 12.38 | 11.65 | |
| | LiveBench instruction following | strict, 200 questions | 16.53 | not run | |
| | CaseReportBench | token F1 | not run (not on the Hub) | not run | |
| | EOS rate (IFEval) | share of answers that stop | 0.82 | 0.98 | |
|
|
| For reference, the Kiyo card reports (not re-run here): Supra2-100M-Instruct GPQA 5.56, IFEval 9.06, LiveBench 11.28; SmolLM2-135M-Instruct GPQA 16.57, IFEval 20.89, LiveBench 24.23. |
|
|
| Notes: |
|
|
| - With the protocol setting, about 18% of IFEval answers never stop and loop on a sentence. Under the recommended sampling this drops to about 2%. |
| - GPQA: among answered questions, accuracy is 19.5%, which is below the 25% chance level. The model cannot solve these questions. |
| - The GPQA prompt was written for this run, not taken from the Kiyo card, so the number is not strictly comparable. |
| - Single run, single seed. Differences of a point or two are within noise. |
|
|
| Per-sample outputs and summaries are in [`eval/`](eval/). |
|
|
| ## Limitations |
|
|
| - It is tiny (5.1M parameters). Expect factual errors, invented details, arithmetic mistakes, weak multi-step reasoning, and repetitive text in long answers. |
| - Under near-greedy decoding it can loop. Keep the recommended sampling settings. |
| - Context is 2,048 tokens, but it was fine-tuned on conversations of up to 1,024 tokens. |
| - English only. |
| - No safety tuning and no safety evaluation. Do not use it for medical, legal, financial or other high-stakes decisions. |
|
|
| ## License |
|
|
| Apache-2.0, as for the base model. The SFT data (smol-smoltalk) is also Apache-2.0. |
|
|
