Update README.md
Browse files
README.md
CHANGED
|
@@ -7,7 +7,7 @@ tags:
|
|
| 7 |
- k5
|
| 8 |
---
|
| 9 |
|
| 10 |
-
# littlelearner-5b-
|
| 11 |
|
| 12 |
5B K-5-bounded chat model with general chat, model identity, and format steerability installed by a behavior SFT on the blend base (chatty v2).
|
| 13 |
|
|
@@ -20,6 +20,14 @@ Part of the **LittleLearner** scale-up study (*pedagogically-controlled knowledg
|
|
| 20 |
- **Pretraining:** 88B tokens on K-5 **LittleCurriculum** (FineWeb-Edu filtered to U.S. grades K-5). WSD schedule, sharded Muon, MXFP8, Megatron-Core on 8xB200.
|
| 21 |
- **SFT:** behavior SFT directly on the cooloff-blend base (no intermediate SFT stage): K-5 math CoT (30k) + smoltalk general chat (15k) + K-5 GSM8K + format-control pairs (answer-only, show-steps, length constraints; user-turn and system-turn variants) + LittleLearner identity data. fp32 master parameters, lr 1e-5, 1 epoch. The model chats on casual prompts, states that it is LittleLearner, and follows answer-format instructions given in the user turn or the system prompt.
|
| 22 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
## Usage
|
| 24 |
|
| 25 |
```python
|
|
|
|
| 7 |
- k5
|
| 8 |
---
|
| 9 |
|
| 10 |
+
# littlelearner-5b-chatty
|
| 11 |
|
| 12 |
5B K-5-bounded chat model with general chat, model identity, and format steerability installed by a behavior SFT on the blend base (chatty v2).
|
| 13 |
|
|
|
|
| 20 |
- **Pretraining:** 88B tokens on K-5 **LittleCurriculum** (FineWeb-Edu filtered to U.S. grades K-5). WSD schedule, sharded Muon, MXFP8, Megatron-Core on 8xB200.
|
| 21 |
- **SFT:** behavior SFT directly on the cooloff-blend base (no intermediate SFT stage): K-5 math CoT (30k) + smoltalk general chat (15k) + K-5 GSM8K + format-control pairs (answer-only, show-steps, length constraints; user-turn and system-turn variants) + LittleLearner identity data. fp32 master parameters, lr 1e-5, 1 epoch. The model chats on casual prompts, states that it is LittleLearner, and follows answer-format instructions given in the user turn or the system prompt.
|
| 22 |
|
| 23 |
+
## Evaluation
|
| 24 |
+
MathCAMPS (paper-filtered):
|
| 25 |
+
- K-5 pass@64 **78.7** / pass@1 **31.4**
|
| 26 |
+
Behavior probes (greedy):
|
| 27 |
+
- casual prompts get conversational replies; identity answered as LittleLearner
|
| 28 |
+
- answer-format instruction obedience: user turn **0.90**, held-out system prompt **0.85**
|
| 29 |
+
|
| 30 |
+
|
| 31 |
## Usage
|
| 32 |
|
| 33 |
```python
|