Add Wisp-15M-Instruct: 2-epoch SFT on smol-smoltalk with ChatML, EOS-weighted loss

README.md

@@ -0,0 +1,110 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+base_model: DedeProGames/Wisp-15M
+datasets:
+- HuggingFaceTB/smol-smoltalk
+tags:
+- wisp
+- tiny
+- instruct
+- sft
+- conversational
+- llama
+---
+
+# Wisp-15M-Instruct
+
+An instruction-tuned version of [Wisp-15M](https://huggingface.co/DedeProGames/Wisp-15M), a 15.5M-parameter LLaMA-style model pretrained from scratch. It was fully fine-tuned for two epochs on [smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk) with the loss computed only on assistant tokens (end-of-turn tokens weighted 5×, so the model learns to stop).
+
+At 15.5M parameters this is a small research model. It follows the chat format and answers simple requests, but factual knowledge, reasoning and arithmetic are weak. See the limitations below.
+
+## Quickstart
+
+Use the recommended sampling settings (stored in `generation_config.json`). With near-greedy decoding this model can fall into repetition loops.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "DedeProGames/Wisp-15M-Instruct"
+tokenizer = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float32).eval()
+
+messages = [{"role": "user", "content": "What is photosynthesis?"}]
+input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=False)
+out = model.generate(input_ids, max_new_tokens=256)  # temperature 0.7, top-p 0.9, repetition penalty 1.1 from generation_config.json
+print(tokenizer.decode(out[0, input_ids.shape[1]:], skip_special_tokens=True))
+```
+
+Keep prompt + answer within the 2,048-token context.
+
+## Chat format
+
+The Wisp tokenizer already has `<|im_start|>` and `<|im_end|>`, so the model uses ChatML. `apply_chat_template` renders it:
+
+```text
+<|endoftext|><|im_start|>user
+{user message}<|im_end|>
+<|im_start|>assistant
+{assistant reply}<|im_end|>
+```
+
+`<|im_end|>` (id 3) ends a turn, and `<|endoftext|>` (id 0) is the BOS. The model stops on either one.
+
+## Architecture
+
+Same as Wisp-15M: `LlamaForCausalLM`, 12 layers, hidden size 320, 10 heads with 2 KV heads (head dim 32), SwiGLU (intermediate 864), RMSNorm, RoPE (θ = 10,000), tied embeddings, 8,192-token byte-level BPE vocabulary, 2,048-token context. 15,531,840 parameters. Weights in BF16 (`model.safetensors`, 31 MB).
+
+## Training
+
+| | |
+|---|---|
+| Base | DedeProGames/Wisp-15M |
+| Data | smol-smoltalk train split, conversations of up to 1,024 tokens (232,008 with this tokenizer; 71.0M supervised tokens) |
+| Epochs | 2 (14,502 steps, batch 32 = 16 × 2 gradient accumulation) |
+| Optimizer | AdamW (β = 0.9, 0.95), weight decay 0.1 on 2D matrices, grad clip 1.0 |
+| Schedule | lr 2e-4, 300 warmup steps, cosine to 2e-5 |
+| Loss | assistant tokens; `<|im_end|>` weighted 5× |
+| Precision | bf16 autocast, fp32 master weights |
+| Hardware | 1× NVIDIA RTX 3060 12 GB, 96 minutes |
+
+Held-out loss on assistant tokens (smol-smoltalk test split): 1.486 at the end of training.
+
+## Evaluation
+
+Benchmarks follow the protocol of the [Kiyo-230M-Instruct](https://huggingface.co/DedeProGames/Kiyo-230M-Instruct) card: FP16, the model's own chat template, no system prompt, new-token limits IFEval 1,024, GPQA 384, LiveBench 512, and the model's full 2,048-token context. The main setting uses temperature 0.1, top-k 50 and repetition penalty 1.05, as on the card. The second setting is the recommended one shipped in `generation_config.json`.
+
+| Benchmark | Metric | Wisp-15M-Instruct, protocol | Wisp-15M-Instruct, recommended sampling |
+|---|---|---:|---:|
+| GPQA Diamond | pass@1 (198 × 5 samples, T = 0.6) | 12.32 | not run |
+| IFEval | prompt-level strict | 18.30 | 15.90 |
+| LiveBench instruction following | strict, 200 questions | 17.55 | not run |
+| CaseReportBench | token F1 | not run (not on the Hub) | not run |
+| EOS rate (IFEval) | share of answers that stop | 0.89 | 0.99 |
+
+For reference, the Kiyo card reports (not re-run here): Kiyo-230M-Instruct GPQA 21.11, IFEval 17.38, LiveBench 18.77; SmolLM2-135M-Instruct GPQA 16.57, IFEval 20.89, LiveBench 24.23; BananaMind-2-Pro GPQA 15.05, IFEval 20.15, LiveBench 19.75; Supra2-100M-Instruct GPQA 5.56, IFEval 9.06, LiveBench 11.28.
+
+Notes:
+
+- With the protocol setting, about 11% of IFEval answers never stop and loop. Under the recommended sampling this drops to about 1%, at a cost of about 2 IFEval points.
+- GPQA: among answered questions, accuracy is 22.1%, close to the 25% chance level. The model cannot solve these questions.
+- The GPQA prompt was written for this run, not taken from the Kiyo card, so the number is not strictly comparable.
+- Single run, single seed. Differences of a point or two are within noise.
+
+Per-sample outputs and summaries are in [`eval/`](eval/).
+
+## Limitations
+
+- It is small (15.5M parameters). Expect factual errors, invented details, arithmetic mistakes, weak multi-step reasoning, and repetitive text in long answers.
+- Under near-greedy decoding it can loop. Keep the recommended sampling settings.
+- Context is 2,048 tokens, but it was fine-tuned on conversations of up to 1,024 tokens.
+- English only.
+- No safety tuning and no safety evaluation. Do not use it for medical, legal, financial or other high-stakes decisions.
+
+## License
+
+Apache-2.0, as for the base model. The SFT data (smol-smoltalk) is also Apache-2.0.

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "head_dim": 32,
+  "hidden_act": "silu",
+  "hidden_size": 320,
+  "initializer_range": 0.02,
+  "intermediate_size": 864,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 10,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 2,
+  "pad_token_id": 1,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.53.3",
+  "use_cache": true,
+  "vocab_size": 8192
+}

eval/protocol/summary.json

@@ -0,0 +1,25 @@
+{
+  "ifeval": {
+    "metric": "prompt-level strict",
+    "score": 18.299445471349355,
+    "n": 541,
+    "eos_rate": 0.8927911275415896,
+    "minutes": 10.05
+  },
+  "gpqa": {
+    "metric": "pass@1 (990 attempts, no-letter counts wrong)",
+    "score": 12.323232323232324,
+    "n_attempts": 990,
+    "answered_rate": 55.75757575757576,
+    "accuracy_among_answered": 22.10144927536232,
+    "prompts_truncated": 5,
+    "minutes": 6.83
+  },
+  "livebench": {
+    "metric": "per-question mean(all followed, fraction followed), strict",
+    "score": 17.554166666666664,
+    "n": 200,
+    "eos_rate": 0.86,
+    "minutes": 2.28
+  }
+}

eval/recommended_sampling/summary.json

@@ -0,0 +1,9 @@
+{
+  "ifeval": {
+    "metric": "prompt-level strict",
+    "score": 15.89648798521257,
+    "n": 541,
+    "eos_rate": 0.9944547134935305,
+    "minutes": 4.74
+  }
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 0,
+  "pad_token_id": 1,
+  "eos_token_id": [
+    3,
+    0
+  ],
+  "do_sample": true,
+  "temperature": 0.7,
+  "top_p": 0.9,
+  "repetition_penalty": 1.1,
+  "max_new_tokens": 512
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d1ff841c37b26b40ae9e52b42e2f0fd36ca69efb00695bfd7c2ab2cfba9b971
+size 31075696

special_tokens_map.json

@@ -0,0 +1,9 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|pad|>",
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ]
+}

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|im_end|>",
+  "pad_token": "<|pad|>",
+  "unk_token": null,
+  "model_max_length": 2048,
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "clean_up_tokenization_spaces": false,
+  "chat_template": "{{- '<|endoftext|>' -}}{%- for message in messages -%}{{- '<|im_start|>' + message['role'] + '\n' + message['content'].strip() + '<|im_end|>\n' -}}{%- endfor -%}{%- if add_generation_prompt -%}{{- '<|im_start|>assistant\n' -}}{%- endif -%}"
+}