| # littlelearner-1.3b-base |
|
|
| 1.36B K-5-bounded base model (pretraining only). |
|
|
| Part of the [**LittleLearner**](https://arxiv.org/abs/2608.13545) scale-up study (*pedagogically-controlled knowledge exposure*): Qwen3 dense LMs trained on a corpus filtered to U.S. K-5 material (**bounded**) vs an unfiltered FineWeb-Edu corpus (**unbounded**), to measure what an interpretable knowledge boundary costs and grants. |
|
|
| ## Model |
| - **Architecture:** Qwen3 dense (`Qwen3ForCausalLM`). |
| - **Size:** 1.358B params, hidden 2048, 26 layers, 16 query / 8 KV heads, FFN 5632. **Context:** 4096. |
| - **Tokenizer:** custom 64k byte-level BPE with per-digit splitting (ChatML special tokens). |
| - **Pretraining:** 88B tokens on K-5 **LittleCurriculum** (FineWeb-Edu filtered to U.S. grades K-5). WSD schedule, sharded Muon, MXFP8, Megatron-Core on 8xB200. |
|
|
| ## Usage |
| ```python |
| # transformers (completion) |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| repo = "manueldeprada/littlelearner-1.3b-base" |
| tok = AutoTokenizer.from_pretrained(repo) |
| model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="cuda") |
| ids = tok("The sum of 2 and 3 is", return_tensors="pt").to(model.device) |
| print(tok.decode(model.generate(**ids)[0], skip_special_tokens=True)) |
| ``` |
|
|
| ```python |
| # vLLM |
| from vllm import LLM |
| llm = LLM("manueldeprada/littlelearner-1.3b-base") |
| print(llm.generate(["The sum of 2 and 3 is"])[0].outputs[0].text) |
| ``` |
|
|
