Upload folder using huggingface_hub

README.md

@@ -25,11 +25,11 @@ tags:
 
 # LittleKedi-mini-base
 
-> **Training in progress.** This is a checkpoint at **1.25B of 3.2B tokens** (39%). The weights and results below will be replaced as training continues, and the final model will be better than this checkpoint.
+> **Training in progress.** This is a checkpoint at **1.50B of 3.2B tokens** (47%). The weights and results below will be replaced as training continues, and the final model will be better than this checkpoint.
 
 `LittleKedi-mini-base` is a **small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch**. It has
 **9.4M parameters** in total, of which **4.8M are active per token**
-(3.18M excluding the embedding table). It was pretrained on a 1,250M-token mix (FineWeb-Edu, Cosmopedia (Stanford), Cosmopedia (OpenStax), Cosmopedia (WikiHow), Cosmopedia (Khan Academy), FineMath 4+, github-code-clean, SmolTalk).
+(3.18M excluding the embedding table). It was pretrained on a 1,500M-token mix (FineWeb-Edu, Cosmopedia (Stanford), Cosmopedia (OpenStax), Cosmopedia (WikiHow), Cosmopedia (Khan Academy), FineMath 4+, github-code-clean, SmolTalk).
 
 *Part of the LittleKedi family. Not affiliated with DeepSeek.*
 
@@ -98,14 +98,14 @@ between models.
 
 | | |
 |---|---|
-| Data | 1,250M tokens (~133 tokens per parameter, ~263 per active parameter), the first 30% of a 4,200M-token dataset, sources interleaved throughout (mix below) |
+| Data | 1,500M tokens (~160 tokens per parameter, ~316 per active parameter), the first 36% of a 4,200M-token dataset, sources interleaved throughout (mix below) |
 | Tokenizer | Byte-level BPE, 8,192 tokens (~3.70 bytes/token on this mix); digits split individually; chat special tokens built in |
-| Steps | 19,074 × 65,536 tokens (batch 16 × 1024 × 4 accumulation) |
+| Steps | 22,889 × 65,536 tokens (batch 16 × 1024 × 4 accumulation) |
 | Optimizer | **Muon** (momentum 0.95, Nesterov, 5 Newton-Schulz steps; each expert's matrix orthogonalized separately) for hidden matrices, AdamW (β = 0.9, 0.95) for embeddings, norms and the router; weight decay 0.1, gradient clipping 1.0 |
 | Learning rate | 2e-3 peak; **WSD**: 200 warm-up steps, constant, then linear decay to 0 over the last 20% of steps (schedule set for 48,828 steps) |
 | Precision | bf16 autocast, `torch.compile` |
-| Compute | ~2.4e16 FLOPs (6 × active non-embedding parameters × tokens) |
-| Hardware | One AMD Radeon 890M integrated GPU (Ryzen AI 300 laptop), ROCm on Windows: about 37.6 hours at a median ~9.2K tokens/s |
+| Compute | ~2.9e16 FLOPs (6 × active non-embedding parameters × tokens) |
+| Hardware | One AMD Radeon 890M integrated GPU (Ryzen AI 300 laptop), ROCm on Windows: about 44.8 hours at a median ~9.3K tokens/s |
 
 | Source | What it is | Tokens | Share |
 |---|---|---|---|
@@ -122,28 +122,20 @@ between models.
 
 | Metric | Value |
 |---|---|
-| Validation loss (nats/token, held-out mix) | **3.115** |
-| Perplexity | 22.5 |
-| Bits per byte | 1.229 |
-| Validation loss: code | 1.991 |
-| Validation loss: cosmo-khanacademy | 2.432 |
-| Validation loss: cosmo-openstax | 2.858 |
-| Validation loss: cosmo-stanford | 2.784 |
-| Validation loss: cosmo-wikihow | 2.746 |
-| Validation loss: finemath | 2.668 |
-| Validation loss: fineweb-edu | 3.397 |
-| Validation loss: smoltalk | 2.314 |
+| Validation loss (nats/token, held-out mix) | **3.106** |
+| Perplexity | 22.3 |
+| Bits per byte | 1.226 |
 | Experts never chosen on the validation sample | 0 |
 
-Validation loss over training: 5.13 (16M tok) → 3.34 (197M tok) → 3.23 (361M tok) → 3.18 (541M tok) → 3.15 (721M tok) → 3.14 (901M tok) → 3.12 (1,065M tok) → 3.11 (1,245M tok)
+Validation loss over training: 5.13 (16M tok) → 3.31 (229M tok) → 3.21 (442M tok) → 3.16 (655M tok) → 3.14 (852M tok) → 3.12 (1,065M tok) → 3.11 (1,278M tok) → 3.11 (1,491M tok)
 
 ### Zero-shot benchmarks (`evals.py`)
 
 | Task | acc_norm | acc | Items | Chance |
 |---|---|---|---|---|
-| SciQ | 57.9% ± 1.6% | 65.7% | 1,000 | 25% |
-| ARC-Easy | 34.9% ± 1.0% | 34.7% | 2,376 | 25% |
-| HellaSwag | 27.4% ± 0.4% | 26.5% | 10,042 | 25% |
+| SciQ | 57.3% ± 1.6% | 62.5% | 1,000 | 25% |
+| ARC-Easy | 33.8% ± 1.0% | 34.7% | 2,376 | 25% |
+| HellaSwag | 27.4% ± 0.4% | 26.4% | 10,042 | 25% |
 
 Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness
 prompts (SciQ includes its supporting passage). **acc_norm** divides by the answer's length in characters;
@@ -154,16 +146,16 @@ prompts (SciQ includes its supporting passage). **acc_norm** divides by the answ
 | Format | Accuracy | Questions scored | Average few-shot examples |
 |---|---|---|---|
 | letter | 24.9% | 14,037 (skipped 5) | 4.5 |
-| cloze | 25.5% | 14,037 (skipped 5) | 0.0 |
+| cloze | 25.6% | 14,037 (skipped 5) | 0.0 |
 
 | Category | Subjects | Questions | Letter | Cloze |
 |---|---|---|---|---|
-| STEM | 19 | 3,153 | 27.5% | 24.2% |
-| Humanities | 13 | 4,705 | 23.4% | 25.6% |
-| Social Sciences | 12 | 3,077 | 23.4% | 25.9% |
-| Other | 13 | 3,102 | 25.9% | 26.2% |
+| STEM | 19 | 3,153 | 26.9% | 24.0% |
+| Humanities | 13 | 4,705 | 23.4% | 25.5% |
+| Social Sciences | 12 | 3,077 | 24.2% | 26.6% |
+| Other | 13 | 3,102 | 25.7% | 26.5% |
 
-**Biology & medicine (cloze): 27.3% on 2,089 questions**, +2.3 points against 25% chance (about 2.4 standard errors, significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.
+**Biology & medicine (cloze): 26.7% on 2,089 questions**, +1.7 points against 25% chance (about 1.8 standard errors, not yet significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.
 
 In the **letter** format the model compares " A"–" D", with same-subject examples added while they fit the
 context. In the **cloze** format each answer's text is scored by log-probability per byte; at this size the
@@ -175,13 +167,13 @@ cloze results are the meaningful ones. Chance is 25%.
 |---|---|---|---|---|
 | Parameters (total) | 9.4M | 70.4M | 124.4M | 51.8M |
 | Active non-embedding parameters | 3.2M | 18.9M | 85.1M | 35.4M |
-| Training tokens | 1,250M | 300B (the Pile) | undisclosed (WebText) | 20B (FineWeb-Edu) |
-| SciQ (acc_norm) | 57.9% | 55.2% | 53.2% | 77.2% |
-| ARC-Easy (acc_norm) | 34.9% | 35.0% | 42.0% | 52.2% |
+| Training tokens | 1,500M | 300B (the Pile) | undisclosed (WebText) | 20B (FineWeb-Edu) |
+| SciQ (acc_norm) | 57.3% | 55.2% | 53.2% | 77.2% |
+| ARC-Easy (acc_norm) | 33.8% | 35.0% | 42.0% | 52.2% |
 | HellaSwag (acc_norm) | 27.4% | – | 29.5% | 31.8% |
-| MMLU (cloze) | 25.5% | – | – | – |
-| MMLU biology & medicine (cloze) | 27.3% | – | – | – |
-| Validation loss (held-out mix) | 3.111 | – | – | – |
+| MMLU (cloze) | 25.6% | – | – | – |
+| MMLU biology & medicine (cloze) | 26.7% | – | – | – |
+| Validation loss (held-out mix) | 3.106 | – | – | – |
 
 LittleKedi columns are measured with this repo's `evals.py` and `mmlu_eval.py` on the checkpoints named. Validation
 losses are only comparable between models that share a tokenizer and validation set. The other columns are
@@ -233,4 +225,4 @@ GGUF builds come with the instruct models.
 - Pretraining data: **github-code-clean** (Apache 2.0, with each file under its own open-source license).
 - Pretraining data: **SmolTalk** (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).
 
-*Card generated 2026-10-13 by `prepare_release.py` from `mini-1250M.pt` (step 19073).*
+*Card generated 2026-10-14 by `prepare_release.py` from `mini-1500M.pt` (step 22888).*

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7bc4283beed488046d74670405939fb5cd26cbd0555d67800b5245594ab03b8c
+oid sha256:29fc3cb48bdec72527559727f979933bd5d586a10192cd4320b39c20186f60ac
 size 37593760