Blame for README.md

---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
base_model: DedeProGames/Wisp-5M
datasets:
- HuggingFaceTB/smol-smoltalk
tags:
- wisp
- tiny
- instruct
- sft
- conversational
- llama
---

# Wisp-5M-Instruct

An instruction-tuned version of [Wisp-5M](https://huggingface.co/DedeProGames/Wisp-5M), a 5.1M-parameter LLaMA-style model pretrained from scratch. It was fully fine-tuned for two epochs on [smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk) with the loss computed only on assistant tokens (end-of-turn tokens weighted 5×, so the model learns to stop).

At 5M parameters this is a tiny research model. It follows the chat format and answers simple requests, but factual knowledge, reasoning and arithmetic are very weak. See the limitations below.

## Quickstart

Use the recommended sampling settings (they are stored in `generation_config.json`). With near-greedy decoding this model often falls into repetition loops.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "DedeProGames/Wisp-5M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float32).eval()

messages = [{"role": "user", "content": "What is photosynthesis?"}]
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=False)
out = model.generate(input_ids, max_new_tokens=256)  # temperature 0.7, top-p 0.9, repetition penalty 1.1 from generation_config.json
print(tokenizer.decode(out[0, input_ids.shape[1]:], skip_special_tokens=True))
```

Keep prompt + answer within the 2,048-token context.

## Chat format

The Wisp tokenizer already has `<|im_start|>` and `<|im_end|>`, so the model uses ChatML. `apply_chat_template` renders it:

```text
<|endoftext|><|im_start|>user
{user message}<|im_end|>
<|im_start|>assistant
{assistant reply}<|im_end|>
```

`<|im_end|>` (id 3) ends a turn, and `<|endoftext|>` (id 0) is the BOS. The model stops on either one.

## Architecture

Same as Wisp-5M: `LlamaForCausalLM`, 9 layers, hidden size 192, 6 heads with 2 KV heads, SwiGLU (intermediate 512), RMSNorm, RoPE (θ = 10,000), tied embeddings, 8,192-token byte-level BPE vocabulary, 2,048-token context. Weights in BF16 (`model.safetensors`, 20 MB).

## Training

| | |
|---|---|
| Base | DedeProGames/Wisp-5M |
| Data | smol-smoltalk train split, conversations of up to 1,024 tokens (with this tokenizer) |
| Epochs | 2 (14,502 steps, batch 32 = 16 × 2 gradient accumulation) |
| Optimizer | AdamW (β = 0.9, 0.95), weight decay 0.1 on 2D matrices, grad clip 1.0 |
| Schedule | lr 2e-4, 300 warmup steps, cosine to 2e-5 |
| Loss | assistant tokens; `<|im_end|>` weighted 5× |
| Precision | bf16 autocast, fp32 master weights |
| Hardware | 1× NVIDIA RTX 3060 12 GB, 47 minutes |

Held-out loss on assistant tokens (smol-smoltalk test split): 1.791 at the end of training.

## Evaluation

Benchmarks follow the protocol of the [Kiyo-230M-Instruct](https://huggingface.co/DedeProGames/Kiyo-230M-Instruct) card: FP16, the model's own chat template, no system prompt, new-token limits IFEval 1,024, GPQA 384, LiveBench 512, and the model's full 2,048-token context. The main setting uses temperature 0.1, top-k 50 and repetition penalty 1.05, as on the card (GPQA is sampled at T = 0.6, as on the card). The second setting is the recommended one shipped in `generation_config.json`.

| Benchmark | Metric | Wisp-5M-Instruct, protocol | Wisp-5M-Instruct, recommended sampling |
|---|---|---:|---:|
| GPQA Diamond | pass@1 (198 × 5 samples, T = 0.6) | 9.80 | not run |
| IFEval | prompt-level strict | 12.38 | 11.65 |
| LiveBench instruction following | strict, 200 questions | 16.53 | not run |
| CaseReportBench | token F1 | not run (not on the Hub) | not run |
| EOS rate (IFEval) | share of answers that stop | 0.82 | 0.98 |

For reference, the Kiyo card reports (not re-run here): Supra2-100M-Instruct GPQA 5.56, IFEval 9.06, LiveBench 11.28; SmolLM2-135M-Instruct GPQA 16.57, IFEval 20.89, LiveBench 24.23.

Notes:

- With the protocol setting, about 18% of IFEval answers never stop and loop on a sentence. Under the recommended sampling this drops to about 2%.
- GPQA: among answered questions, accuracy is 19.5%, which is below the 25% chance level. The model cannot solve these questions.
- The GPQA prompt was written for this run, not taken from the Kiyo card, so the number is not strictly comparable.
- Single run, single seed. Differences of a point or two are within noise.

Per-sample outputs and summaries are in [`eval/`](eval/).

## Limitations

- It is tiny (5.1M parameters). Expect factual errors, invented details, arithmetic mistakes, weak multi-step reasoning, and repetitive text in long answers.
- Under near-greedy decoding it can loop. Keep the recommended sampling settings.
- Context is 2,048 tokens, but it was fine-tuned on conversations of up to 1,024 tokens.
- English only.
- No safety tuning and no safety evaluation. Do not use it for medical, legal, financial or other high-stakes decisions.

## License

Apache-2.0, as for the base model. The SFT data (smol-smoltalk) is also Apache-2.0.