| --- |
| license: other |
| language: en |
| tags: |
| - littlelearner |
| - qwen3 |
| - k5 |
| --- |
| |
| # littlelearner-5b-chatty |
| 5B K-5-bounded chat model with general chat, model identity, and format steerability installed by a behavior SFT on the blend base (chatty v2). |
|
|
| Part of the [**LittleLearner**](https://arxiv.org/abs/2608.13545) scale-up study (*pedagogically-controlled knowledge exposure*): Qwen3 dense LMs trained on a corpus filtered to U.S. K-5 material (**bounded**) vs an unfiltered FineWeb-Edu corpus (**unbounded**), to measure what an interpretable knowledge boundary costs and grants. |
|
|
| ## Model |
| - **Architecture:** Qwen3 dense (`Qwen3ForCausalLM`). |
| - **Size:** 5.04B params, hidden 3072, 44 layers, 24 query / 8 KV heads, FFN 9216. **Context:** 4096. |
| - **Tokenizer:** custom 64k byte-level BPE with per-digit splitting (ChatML special tokens). |
| - **Pretraining:** 88B tokens on K-5 **LittleCurriculum** (FineWeb-Edu filtered to U.S. grades K-5). WSD schedule, sharded Muon, MXFP8, Megatron-Core on 8xB200. |
| - **SFT:** behavior SFT directly on the cooloff-blend base (no intermediate SFT stage): K-5 math CoT (30k) + smoltalk general chat (15k) + K-5 GSM8K + format-control pairs (answer-only, show-steps, length constraints; user-turn and system-turn variants) + LittleLearner identity data. fp32 master parameters, lr 1e-5, 1 epoch. The model chats on casual prompts, states that it is LittleLearner, and follows answer-format instructions given in the user turn or the system prompt. |
|
|
| ## Evaluation |
| MathCAMPS (paper-filtered): |
| - K-5 pass@64 **78.7** / pass@1 **31.4** |
| Behavior probes (greedy): |
| - casual prompts get conversational replies; identity answered as LittleLearner |
| - answer-format instruction obedience: user turn **0.90**, held-out system prompt **0.85** |
|
|
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| repo = "littlelearner/littlelearner-5b-bounded-sft-chatty" |
| tok = AutoTokenizer.from_pretrained(repo) |
| model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto") |
| msgs = [{"role": "user", "content": "If Sarah has 12 apples and gives 5 to Tom, how many does she have left?"}] |
| ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device) |
| out = model.generate(ids) |
| print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ```python |
| # vLLM |
| from vllm import LLM |
| repo = "littlelearner/littlelearner-5b-bounded-sft-chatty" |
| llm = LLM(repo) |
| msgs = [{"role": "user", "content": "Liam has 3 apples and buys 4 more. How many apples does he have?"}] |
| print(llm.chat(msgs)[0].outputs[0].text) |
| ``` |
