ffli commited on
Commit
2cb8bbd
·
verified ·
1 Parent(s): 0f2642f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +9 -1
README.md CHANGED
@@ -7,7 +7,7 @@ tags:
7
  - k5
8
  ---
9
 
10
- # littlelearner-5b-bounded-sft-chatty
11
 
12
  5B K-5-bounded chat model with general chat, model identity, and format steerability installed by a behavior SFT on the blend base (chatty v2).
13
 
@@ -20,6 +20,14 @@ Part of the **LittleLearner** scale-up study (*pedagogically-controlled knowledg
20
  - **Pretraining:** 88B tokens on K-5 **LittleCurriculum** (FineWeb-Edu filtered to U.S. grades K-5). WSD schedule, sharded Muon, MXFP8, Megatron-Core on 8xB200.
21
  - **SFT:** behavior SFT directly on the cooloff-blend base (no intermediate SFT stage): K-5 math CoT (30k) + smoltalk general chat (15k) + K-5 GSM8K + format-control pairs (answer-only, show-steps, length constraints; user-turn and system-turn variants) + LittleLearner identity data. fp32 master parameters, lr 1e-5, 1 epoch. The model chats on casual prompts, states that it is LittleLearner, and follows answer-format instructions given in the user turn or the system prompt.
22
 
 
 
 
 
 
 
 
 
23
  ## Usage
24
 
25
  ```python
 
7
  - k5
8
  ---
9
 
10
+ # littlelearner-5b-chatty
11
 
12
  5B K-5-bounded chat model with general chat, model identity, and format steerability installed by a behavior SFT on the blend base (chatty v2).
13
 
 
20
  - **Pretraining:** 88B tokens on K-5 **LittleCurriculum** (FineWeb-Edu filtered to U.S. grades K-5). WSD schedule, sharded Muon, MXFP8, Megatron-Core on 8xB200.
21
  - **SFT:** behavior SFT directly on the cooloff-blend base (no intermediate SFT stage): K-5 math CoT (30k) + smoltalk general chat (15k) + K-5 GSM8K + format-control pairs (answer-only, show-steps, length constraints; user-turn and system-turn variants) + LittleLearner identity data. fp32 master parameters, lr 1e-5, 1 epoch. The model chats on casual prompts, states that it is LittleLearner, and follows answer-format instructions given in the user turn or the system prompt.
22
 
23
+ ## Evaluation
24
+ MathCAMPS (paper-filtered):
25
+ - K-5 pass@64 **78.7** / pass@1 **31.4**
26
+ Behavior probes (greedy):
27
+ - casual prompts get conversational replies; identity answered as LittleLearner
28
+ - answer-format instruction obedience: user turn **0.90**, held-out system prompt **0.85**
29
+
30
+
31
  ## Usage
32
 
33
  ```python