| ---
|
| license: apache-2.0
|
| language:
|
| - en
|
| pipeline_tag: text-generation
|
| datasets:
|
| - HuggingFaceFW/fineweb-edu
|
| - HuggingFaceTB/cosmopedia
|
| - HuggingFaceTB/finemath
|
| - codeparrot/github-code-clean
|
| - HuggingFaceTB/smoltalk
|
| tags:
|
| - mixture-of-experts
|
| - moe
|
| - deepseek
|
| - multi-head-latent-attention
|
| - mla
|
| - from-scratch
|
| - tiny
|
| - littlekedi
|
| - base
|
| ---
|
|
|
| <p align="center"><img src="little_kedi.jpg" alt="LittleKedi logo" width="640"></p>
|
|
|
| # LittleKedi-mini-base
|
|
|
| > **Training in progress.** This is a checkpoint at **1.50B of 3.2B tokens** (47%). The weights and results below will be replaced as training continues, and the final model will be better than this checkpoint.
|
|
|
| `LittleKedi-mini-base` is a **small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch**. It has
|
| **9.4M parameters** in total, of which **4.8M are active per token**
|
| (3.18M excluding the embedding table). It was pretrained on a 1,500M-token mix (FineWeb-Edu, Cosmopedia (Stanford), Cosmopedia (OpenStax), Cosmopedia (WikiHow), Cosmopedia (Khan Academy), FineMath 4+, github-code-clean, SmolTalk).
|
|
|
| *Part of the LittleKedi family. Not affiliated with DeepSeek.*
|
|
|
| This is the **base model**, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but **knows few facts and confidently makes things up**. Don't use it for anything that matters.
|
|
|
| ## The LittleKedi family
|
|
|
| LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from
|
| scratch. Every size comes in three variants that differ in **how sparse** they are: how many routed experts
|
| they have, and what share of their parameters a token actually passes through.
|
|
|
| | Variant | Routed experts |
|
| |---|---|
|
| | **standard** (`<size>`) | a few wide experts; the densest variant |
|
| | **sparse** (`<size>-sparse`) | many narrower, fine-grained experts, more of them active per token |
|
| | **wide-sparse** (`<size>-wide-sparse`) | more experts again; the largest total and smallest active share |
|
|
|
| More experts means more total capacity to store knowledge, but each expert
|
| sees fewer tokens (tokens Γ top-k Γ· experts per layer), so sparser variants need more training data before
|
| their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be
|
| compared directly. Shapes change as the family scales; the table below is generated from the current
|
| presets.
|
|
|
| This card is for **`mini`**, the **standard** mini model: 4.8M of its 9.4M parameters (51%) are active per token. Its siblings are `mini-sparse` and `mini-wide-sparse`.
|
|
|
| | Model | Variant | Vocab | Total | Active | Active non-emb. | Active % | Routed experts |
|
| |---|---|---|---|---|---|---|---|
|
| | `tiny` | standard | 8K | 2.0M | 1.5M | 0.5M | 78% | 8 Γ 64, top-2 |
|
| | `tiny-sparse` | sparse | 8K | 2.6M | 1.6M | 0.5M | 60% | 32 Γ 32, top-4 |
|
| | `tiny-wide-sparse` | wide-sparse | 8K | 3.8M | 1.6M | 0.5M | 42% | 64 Γ 32, top-4 |
|
| | **`mini` β this model** | **standard** | **8K** | **9.4M** | **4.8M** | **3.2M** | **51%** | **12 Γ 128, top-3** |
|
| | `mini-sparse` | sparse | 8K | 19.8M | 4.8M | 3.2M | 24% | 64 Γ 64, top-6 |
|
| | `mini-wide-sparse` | wide-sparse | 8K | 36.4M | 4.9M | 3.3M | 13% | 128 Γ 64, top-6 |
|
| | `small` | standard | 8K | 12.2M | 6.3M | 4.2M | 52% | 16 Γ 128, top-4 |
|
| | `small-sparse` | sparse | 8K | 24.1M | 6.4M | 4.3M | 27% | 80 Γ 64, top-8 |
|
| | `small-wide-sparse` | wide-sparse | 8K | 43.9M | 6.5M | 4.4M | 15% | 160 Γ 64, top-8 |
|
| | `moderate` | standard | 16K | 49.8M | 18.8M | 12.5M | 38% | 24 Γ 192, top-4 |
|
| | `moderate-sparse` | sparse | 16K | 105.8M | 19.1M | 12.8M | 18% | 120 Γ 96, top-8 |
|
| | `moderate-wide-sparse` | wide-sparse | 16K | 199.0M | 19.4M | 13.1M | 10% | 240 Γ 96, top-8 |
|
| | `medium` | standard | 16K | 85.9M | 30.8M | 22.4M | 36% | 24 Γ 256, top-4 |
|
| | `medium-sparse` | sparse | 16K | 185.3M | 31.2M | 22.8M | 17% | 120 Γ 128, top-8 |
|
| | `medium-wide-sparse` | wide-sparse | 16K | 350.9M | 31.6M | 23.2M | 9% | 240 Γ 128, top-8 |
|
| | `base` | standard | 32K | 258.7M | 105.4M | 80.2M | 41% | 32 Γ 256, top-6 |
|
| | `base-sparse` | sparse | 32K | 542.8M | 106.3M | 81.2M | 20% | 160 Γ 128, top-12 |
|
| | `base-wide-sparse` | wide-sparse | 32K | 1015.9M | 107.6M | 82.4M | 11% | 320 Γ 128, top-12 |
|
|
|
| "Active" counts everything a token passes through: embeddings, attention, the dense layer, shared
|
| experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the
|
| output head), which costs memory but almost no compute; it's the fair number for comparing compute
|
| between models.
|
|
|
| ## Architecture
|
|
|
| | Component | This model |
|
| |---|---|
|
| | Parameters | 9.40M total, 4.75M active per token (51%), 3.18M active non-embedding |
|
| | Layers / hidden size | 8 / 192 (layer 0 has a dense SwiGLU FFN of 512; layers 1β7 are MoE) |
|
| | Attention | **Multi-head Latent Attention (MLA)**: 6 heads, KV latent 48, decoupled RoPE dim 16, head dims 24+16 (q/k) and 24 (v). The KV cache stores only the 48-dim latent plus a 16-dim RoPE key per token |
|
| | MoE | **DeepSeekMoE**: 12 routed experts (SwiGLU, 128 hidden), top-3, plus 1 shared expert; sigmoid gating; group-limited routing (3 groups, top-2) |
|
| | Load balancing | **Auxiliary-loss-free** bias balancing (update speed 0.001) plus a small sequence-wise loss (Ξ± = 0.0001). Over the second half of training the median batch put 1.09Γ the mean load on the busiest expert (1.14Γ in the worst layer), and 0.00% of expert assignments were dropped for capacity. |
|
| | Expert dispatch | Training: fixed capacity (capacity factor 2.0; overflow assignments are dropped). Inference is always dropless |
|
| | Embeddings | Input embedding and output head **tied** |
|
| | Context | 1,024 tokens |
|
|
|
| ## Training
|
|
|
| | | |
|
| |---|---|
|
| | Data | 1,500M tokens (~160 tokens per parameter, ~316 per active parameter), the first 36% of a 4,200M-token dataset, sources interleaved throughout (mix below) |
|
| | Tokenizer | Byte-level BPE, 8,192 tokens (~3.70 bytes/token on this mix); digits split individually; chat special tokens built in |
|
| | Steps | 22,889 Γ 65,536 tokens (batch 16 Γ 1024 Γ 4 accumulation) |
|
| | Optimizer | **Muon** (momentum 0.95, Nesterov, 5 Newton-Schulz steps; each expert's matrix orthogonalized separately) for hidden matrices, AdamW (Ξ² = 0.9, 0.95) for embeddings, norms and the router; weight decay 0.1, gradient clipping 1.0 |
|
| | Learning rate | 2e-3 peak; **WSD**: 200 warm-up steps, constant, then linear decay to 0 over the last 20% of steps (schedule set for 48,828 steps) |
|
| | Precision | bf16 autocast, `torch.compile` |
|
| | Compute | ~2.9e16 FLOPs (6 Γ active non-embedding parameters Γ tokens) |
|
| | Hardware | One AMD Radeon 890M integrated GPU (Ryzen AI 300 laptop), ROCm on Windows: about 44.8 hours at a median ~9.3K tokens/s |
|
|
|
| | Source | What it is | Tokens | Share |
|
| |---|---|---|---|
|
| | FineWeb-Edu `sample/10BT` | educational web pages | 2,734M | 65% |
|
| | Cosmopedia (Stanford) | synthetic textbooks | 340M | 8% |
|
| | Cosmopedia (OpenStax) | synthetic textbooks | 120M | 3% |
|
| | Cosmopedia (WikiHow) | synthetic how-to articles | 130M | 3% |
|
| | Cosmopedia (Khan Academy) | synthetic lessons | 23M | 1% |
|
| | FineMath 4+ | maths web pages | 424M | 10% |
|
| | github-code-clean | source code | 172M | 4% |
|
| | SmolTalk | chat conversations | 256M | 6% |
|
|
|
| ## Evaluation
|
|
|
| | Metric | Value |
|
| |---|---|
|
| | Validation loss (nats/token, held-out mix) | **3.106** |
|
| | Perplexity | 22.3 |
|
| | Bits per byte | 1.226 |
|
| | Experts never chosen on the validation sample | 0 |
|
|
|
| Validation loss over training: 5.13 (16M tok) β 3.31 (229M tok) β 3.21 (442M tok) β 3.16 (655M tok) β 3.14 (852M tok) β 3.12 (1,065M tok) β 3.11 (1,278M tok) β 3.11 (1,491M tok)
|
|
|
| ### Zero-shot benchmarks (`evals.py`)
|
|
|
| | Task | acc_norm | acc | Items | Chance |
|
| |---|---|---|---|---|
|
| | SciQ | 57.3% Β± 1.6% | 62.5% | 1,000 | 25% |
|
| | ARC-Easy | 33.8% Β± 1.0% | 34.7% | 2,376 | 25% |
|
| | HellaSwag | 27.4% Β± 0.4% | 26.4% | 10,042 | 25% |
|
|
|
| Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness
|
| prompts (SciQ includes its supporting passage). **acc_norm** divides by the answer's length in characters;
|
| **acc** doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.
|
|
|
| ### MMLU (`mmlu_eval.py`)
|
|
|
| | Format | Accuracy | Questions scored | Average few-shot examples |
|
| |---|---|---|---|
|
| | letter | 24.9% | 14,037 (skipped 5) | 4.5 |
|
| | cloze | 25.6% | 14,037 (skipped 5) | 0.0 |
|
|
|
| | Category | Subjects | Questions | Letter | Cloze |
|
| |---|---|---|---|---|
|
| | STEM | 19 | 3,153 | 26.9% | 24.0% |
|
| | Humanities | 13 | 4,705 | 23.4% | 25.5% |
|
| | Social Sciences | 12 | 3,077 | 24.2% | 26.6% |
|
| | Other | 13 | 3,102 | 25.7% | 26.5% |
|
|
|
| **Biology & medicine (cloze): 26.7% on 2,089 questions**, +1.7 points against 25% chance (about 1.8 standard errors, not yet significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.
|
|
|
| In the **letter** format the model compares " A"β" D", with same-subject examples added while they fit the
|
| context. In the **cloze** format each answer's text is scored by log-probability per byte; at this size the
|
| cloze results are the meaningful ones. Chance is 25%.
|
|
|
| ### Compared with its siblings and other small models
|
|
|
| | | **This model** | [Pythia-70M](https://huggingface.co/EleutherAI/pythia-70m) | [GPT-2](https://huggingface.co/openai-community/gpt2) | [Supra-50M](https://huggingface.co/SupraLabs/Supra-50M-Base) |
|
| |---|---|---|---|---|
|
| | Parameters (total) | 9.4M | 70.4M | 124.4M | 51.8M |
|
| | Active non-embedding parameters | 3.2M | 18.9M | 85.1M | 35.4M |
|
| | Training tokens | 1,500M | 300B (the Pile) | undisclosed (WebText) | 20B (FineWeb-Edu) |
|
| | SciQ (acc_norm) | 57.3% | 55.2% | 53.2% | 77.2% |
|
| | ARC-Easy (acc_norm) | 33.8% | 35.0% | 42.0% | 52.2% |
|
| | HellaSwag (acc_norm) | 27.4% | β | 29.5% | 31.8% |
|
| | MMLU (cloze) | 25.6% | β | β | β |
|
| | MMLU biology & medicine (cloze) | 26.7% | β | β | β |
|
| | Validation loss (held-out mix) | 3.106 | β | β | β |
|
|
|
| LittleKedi columns are measured with this repo's `evals.py` and `mmlu_eval.py` on the checkpoints named. Validation
|
| losses are only comparable between models that share a tokenizer and validation set. The other columns are
|
| **published figures**: Pythia-70M from EleutherAI's evaluations as quoted on the
|
| [Wisp-15M card](https://huggingface.co/DedeProGames/Wisp-15M), GPT-2 and Supra-50M from the
|
| [Supra-50M-Base card](https://huggingface.co/SupraLabs/Supra-50M-Base). Their prompts may differ from ours,
|
| so treat gaps of a few points as noise.
|
|
|
| ## What it can and can't do
|
|
|
| *To be written when training completes.*
|
|
|
| ## Usage
|
|
|
| **PyTorch** (needs the `littlekedi` package from [the training code](https://github.com/EdmundMartin/LittleKedi)):
|
|
|
| ```python
|
| import json, torch
|
| from safetensors.torch import load_file
|
| from littlekedi import LittleKediModel, ModelConfig
|
| from littlekedi.tokenizer import BPETokenizer
|
|
|
| model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
|
| model.load_state_dict(load_file("model.safetensors"))
|
| tok = BPETokenizer.from_file("tokenizer.json")
|
| ids = torch.tensor([[tok.eot_id] + tok.encode("Photosynthesis is")])
|
| print(tok.decode(model.generate(ids, 60, temperature=0.7, top_p=0.9, repetition_penalty=1.15)[0].tolist()))
|
| ```
|
|
|
| Or interactively: `python sample.py --ckpt <checkpoint>.pt`. Base models are released as PyTorch weights only;
|
| GGUF builds come with the instruct models.
|
|
|
| ## Files
|
|
|
| | File | Contents |
|
| |---|---|
|
| | `model.safetensors` | PyTorch weights (embedding tied with the output head), `littlekedi` parameter names |
|
| | `config.json` | `ModelConfig` for `littlekedi` |
|
| | `tokenizer.json` | Byte-level BPE tokenizer (Hugging Face `tokenizers` format) |
|
| | `little_kedi.jpg` | The LittleKedi logo |
|
|
|
| ## Acknowledgements
|
|
|
| - The architecture follows **DeepSeek-V3** (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
|
| - **Muon** (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
|
| - Pretraining data: **FineWeb-Edu** (ODC-BY 1.0).
|
| - Pretraining data: **Cosmopedia** (Apache 2.0).
|
| - Pretraining data: **FineMath 4+** (ODC-BY 1.0).
|
| - Pretraining data: **github-code-clean** (Apache 2.0, with each file under its own open-source license).
|
| - Pretraining data: **SmolTalk** (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).
|
|
|
| *Card generated 2026-10-14 by `prepare_release.py` from `mini-1500M.pt` (step 22888).*
|
|
|
