ffli commited on
Commit
9945d2d
·
verified ·
1 Parent(s): 7a6c26d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +19 -1
README.md CHANGED
@@ -8,4 +8,22 @@ Part of the **LittleLearner** scale-up study (*pedagogically-controlled knowledg
8
  - **Architecture:** Qwen3 dense (`Qwen3ForCausalLM`).
9
  - **Size:** 1.358B params, hidden 2048, 26 layers, 16 query / 8 KV heads, FFN 5632. **Context:** 4096.
10
  - **Tokenizer:** custom 64k byte-level BPE with per-digit splitting (ChatML special tokens).
11
- - **Pretraining:** 88B tokens on K-5 **LittleCurriculum** (FineWeb-Edu filtered to U.S. grades K-5). WSD schedule, sharded Muon, MXFP8, Megatron-Core on 8xB200.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
  - **Architecture:** Qwen3 dense (`Qwen3ForCausalLM`).
9
  - **Size:** 1.358B params, hidden 2048, 26 layers, 16 query / 8 KV heads, FFN 5632. **Context:** 4096.
10
  - **Tokenizer:** custom 64k byte-level BPE with per-digit splitting (ChatML special tokens).
11
+ - **Pretraining:** 88B tokens on K-5 **LittleCurriculum** (FineWeb-Edu filtered to U.S. grades K-5). WSD schedule, sharded Muon, MXFP8, Megatron-Core on 8xB200.
12
+
13
+ ## Usage
14
+ ```python
15
+ # transformers (completion)
16
+ from transformers import AutoModelForCausalLM, AutoTokenizer
17
+ repo = "manueldeprada/littlelearner-1.3b-base"
18
+ tok = AutoTokenizer.from_pretrained(repo)
19
+ model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="cuda")
20
+ ids = tok("The sum of 2 and 3 is", return_tensors="pt").to(model.device)
21
+ print(tok.decode(model.generate(**ids)[0], skip_special_tokens=True))
22
+ ```
23
+
24
+ ```python
25
+ # vLLM
26
+ from vllm import LLM
27
+ llm = LLM("manueldeprada/littlelearner-1.3b-base")
28
+ print(llm.generate(["The sum of 2 and 3 is"])[0].outputs[0].text)
29
+ ```