Update README.md
Browse files
README.md
CHANGED
|
@@ -8,4 +8,22 @@ Part of the **LittleLearner** scale-up study (*pedagogically-controlled knowledg
|
|
| 8 |
- **Architecture:** Qwen3 dense (`Qwen3ForCausalLM`).
|
| 9 |
- **Size:** 1.358B params, hidden 2048, 26 layers, 16 query / 8 KV heads, FFN 5632. **Context:** 4096.
|
| 10 |
- **Tokenizer:** custom 64k byte-level BPE with per-digit splitting (ChatML special tokens).
|
| 11 |
-
- **Pretraining:** 88B tokens on K-5 **LittleCurriculum** (FineWeb-Edu filtered to U.S. grades K-5). WSD schedule, sharded Muon, MXFP8, Megatron-Core on 8xB200.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
- **Architecture:** Qwen3 dense (`Qwen3ForCausalLM`).
|
| 9 |
- **Size:** 1.358B params, hidden 2048, 26 layers, 16 query / 8 KV heads, FFN 5632. **Context:** 4096.
|
| 10 |
- **Tokenizer:** custom 64k byte-level BPE with per-digit splitting (ChatML special tokens).
|
| 11 |
+
- **Pretraining:** 88B tokens on K-5 **LittleCurriculum** (FineWeb-Edu filtered to U.S. grades K-5). WSD schedule, sharded Muon, MXFP8, Megatron-Core on 8xB200.
|
| 12 |
+
|
| 13 |
+
## Usage
|
| 14 |
+
```python
|
| 15 |
+
# transformers (completion)
|
| 16 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 17 |
+
repo = "manueldeprada/littlelearner-1.3b-base"
|
| 18 |
+
tok = AutoTokenizer.from_pretrained(repo)
|
| 19 |
+
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="cuda")
|
| 20 |
+
ids = tok("The sum of 2 and 3 is", return_tensors="pt").to(model.device)
|
| 21 |
+
print(tok.decode(model.generate(**ids)[0], skip_special_tokens=True))
|
| 22 |
+
```
|
| 23 |
+
|
| 24 |
+
```python
|
| 25 |
+
# vLLM
|
| 26 |
+
from vllm import LLM
|
| 27 |
+
llm = LLM("manueldeprada/littlelearner-1.3b-base")
|
| 28 |
+
print(llm.generate(["The sum of 2 and 3 is"])[0].outputs[0].text)
|
| 29 |
+
```
|