---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/smollm-corpus
- HuggingFaceTB/finemath
- HuggingFaceGECLM/REDDIT_comments
- codeparrot/github-code-clean
- HuggingFaceTB/smoltalk
- wikimedia/wikipedia
base_model: edededdy/LittleKedi-medium-wide-sparse-base
base_model_relation: finetune
tags:
- mixture-of-experts
- moe
- deepseek
- multi-head-latent-attention
- mla
- from-scratch
- tiny
- littlekedi
- chat
- instruction-tuned
---

<p align="center"><img src="little_kedi.jpg" alt="LittleKedi logo" width="640"></p>

# LittleKedi-medium-wide-sparse-instruct

`LittleKedi-medium-wide-sparse-instruct` is the **instruction-tuned (chat) version of [LittleKedi-medium-wide-sparse-base](https://huggingface.co/edededdy/LittleKedi-medium-wide-sparse-base)**, a **small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch**. It has
**305.4M parameters** in total, of which **31.7M are active per token**
(23.29M excluding the embedding table). It was pretrained on a 20,400M-token mix (FineWeb-Edu, Cosmopedia v2 (SmolLM-Corpus), FineMath 4+, Reddit comments, github-code-clean, SmolTalk, Wikipedia (English, 2023-11-01)).

*Part of the LittleKedi family. Not affiliated with DeepSeek.*

This is the **instruction-tuned (chat) model**. It answers in a chat format and stops when it's done, but it's still a very small model: answers are fluent and on-topic but often **wrong or vague**, and it **invents facts confidently**. It's an educational and research artifact; don't use it for anything that matters.

## Chat format

```
<|system|>You are a helpful assistant.<|end|><|user|>What causes the seasons?<|end|><|assistant|>
```

The model writes its reply and ends it with `<|end|>`. The system turn is optional. The GGUF files embed this
template, and `chat_template.jinja` / `tokenizer_config.json` carry it for other tools.

## The LittleKedi family

LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from
scratch. Every size comes in three variants that differ in **how sparse** they are: how many routed experts
they have, and what share of their parameters a token actually passes through.

| Variant | Routed experts |
|---|---|
| **standard** (`<size>`) | a few wide experts; the densest variant |
| **sparse** (`<size>-sparse`) | many narrower, fine-grained experts, more of them active per token |
| **wide-sparse** (`<size>-wide-sparse`) | more experts again; the largest total and smallest active share |

More experts means more total capacity to store knowledge, but each expert
sees fewer tokens (tokens × top-k ÷ experts per layer), so sparser variants need more training data before
their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be
compared directly. Shapes change as the family scales; the table below is generated from the current
presets.

This card is for **`medium-wide-sparse`**, the **wide-sparse** medium model: 31.7M of its 305.4M parameters (10%) are active per token. Its siblings are `medium` and `medium-sparse`.

| Model | Variant | Vocab | Total | Active | Active non-emb. | Active % | Routed experts |
|---|---|---|---|---|---|---|---|
| `tiny` | standard | 8K | 2.0M | 1.5M | 0.5M | 78% | 8 × 64, top-2 |
| `tiny-sparse` | sparse | 8K | 2.6M | 1.6M | 0.5M | 60% | 32 × 32, top-4 |
| `tiny-wide-sparse` | wide-sparse | 8K | 3.8M | 1.6M | 0.5M | 42% | 64 × 32, top-4 |
| `mini` | standard | 8K | 9.4M | 4.8M | 3.2M | 51% | 12 × 128, top-3 |
| `mini-sparse` | sparse | 8K | 19.8M | 4.8M | 3.2M | 24% | 64 × 64, top-6 |
| `mini-wide-sparse` | wide-sparse | 8K | 36.4M | 4.9M | 3.3M | 13% | 128 × 64, top-6 |
| `small` | standard | 8K | 12.2M | 6.3M | 4.2M | 52% | 16 × 128, top-4 |
| `small-sparse` | sparse | 8K | 24.1M | 6.4M | 4.3M | 27% | 80 × 64, top-8 |
| `small-wide-sparse` | wide-sparse | 8K | 43.9M | 6.5M | 4.4M | 15% | 160 × 64, top-8 |
| `moderate` | standard | 16K | 45.5M | 18.9M | 12.6M | 42% | 24 × 192, top-4 |
| `moderate-sparse` | sparse | 16K | 93.5M | 19.1M | 12.8M | 20% | 120 × 96, top-8 |
| `moderate-wide-sparse` | wide-sparse | 16K | 173.4M | 19.4M | 13.1M | 11% | 240 × 96, top-8 |
| `medium` | standard | 16K | 78.2M | 31.0M | 22.6M | 40% | 24 × 256, top-4 |
| `medium-sparse` | sparse | 16K | 163.4M | 31.3M | 22.9M | 19% | 120 × 128, top-8 |
| **`medium-wide-sparse` ← this model** | **wide-sparse** | **16K** | **305.4M** | **31.7M** | **23.3M** | **10%** | **240 × 128, top-8** |
| `base` | standard | 32K | 258.7M | 105.4M | 80.2M | 41% | 32 × 256, top-6 |
| `base-sparse` | sparse | 32K | 542.8M | 106.3M | 81.2M | 20% | 160 × 128, top-12 |
| `base-wide-sparse` | wide-sparse | 32K | 1015.9M | 107.6M | 82.4M | 11% | 320 × 128, top-12 |

"Active" counts everything a token passes through: embeddings, attention, the dense layer, shared
experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the
output head), which costs memory but almost no compute; it's the fair number for comparing compute
between models.

## Architecture

| Component | This model |
|---|---|
| Parameters | 305.36M total, 31.68M active per token (10%), 23.29M active non-embedding |
| Layers / hidden size | 8 / 512 (the first 2 layers have a dense SwiGLU FFN of 1408; layers 2–7 are MoE) |
| Attention | **Multi-head Latent Attention (MLA)**: 8 heads, KV latent 128, query latent 256, decoupled RoPE dim 32, head dims 64+32 (q/k) and 64 (v). The KV cache stores only the 128-dim latent plus a 32-dim RoPE key per token |
| MoE | **DeepSeekMoE**: 240 routed experts (SwiGLU, 128 hidden), top-8, plus 2 shared experts; sigmoid gating; group-limited routing (8 groups, top-4) |
| Load balancing | **Auxiliary-loss-free** bias balancing (update speed 0.001) plus a small sequence-wise loss (α = 0.0001). Over the second half of training the median batch put 1.92× the mean load on the busiest expert (2.28× in the worst layer), and 0.26% of expert assignments were dropped for capacity. |
| Expert dispatch | Training: fixed capacity (capacity factor 2.0; overflow assignments are dropped). Inference is always dropless |
| Embeddings | Input embedding and output head **tied** |
| Context | 2,048 tokens |

## Training

| | |
|---|---|
| Data | 20,400M tokens, 27,145,344 documents (~67 tokens per parameter, ~644 per active parameter), one pass, sources interleaved throughout (mix below) |
| Tokenizer | Byte-level BPE, 16,384 tokens (~4.00 bytes/token on this mix); digits split individually; chat special tokens built in |
| Steps | 311,279 × 65,536 tokens (batch 16 × 2048 × 2 accumulation) |
| Optimizer | **Muon** (momentum 0.95, Nesterov, 5 Newton-Schulz steps; each expert's matrix orthogonalized separately) for hidden matrices, AdamW (β = 0.9, 0.95) for embeddings, norms and the router; weight decay 0.1, gradient clipping 1.0 |
| Learning rate | 2e-3 peak; **WSD**: 200 warm-up steps, constant, then linear decay to 0 over the last 20% of steps (schedule set for 311,279 steps) |
| Precision | bf16 autocast, `torch.compile` |
| Compute | ~2.9e18 FLOPs (6 × active non-embedding parameters × tokens) |
| Hardware | One NVIDIA GeForce RTX 4090 (RunPod): about 76.7 hours at a median ~73.9K tokens/s |

| Source | What it is | Tokens | Share |
|---|---|---|---|
| FineWeb-Edu `sample/100BT` | educational web pages | 11,820M | 58% |
| Cosmopedia v2 (SmolLM-Corpus) | synthetic textbooks and stories | 2,446M | 12% |
| FineMath 4+ | maths web pages | 1,875M | 9% |
| Reddit comments | forum comments | 1,630M | 8% |
| github-code-clean | source code | 958M | 5% |
| SmolTalk | chat conversations | 917M | 4% |
| Wikipedia (English, 2023-11-01) | encyclopedia articles | 754M | 4% |

## Fine-tuning (SFT)

| | |
|---|---|
| Base model | [LittleKedi-medium-wide-sparse-base](https://huggingface.co/edededdy/LittleKedi-medium-wide-sparse-base) |
| Conversations | 22,841 (22,841 train / 1,201 validation), 2.25M assistant tokens trained on |
| Data | docs/medium-sft-data.md |
| Loss | On assistant replies and their closing `<|end|>` only; prompts and padding are masked |
| Schedule | 2,856 steps, batch 8, 1 epoch, Muon + AdamW, peak LR 5e-5, cosine decay |
| MoE routing | Load-balancing biases **frozen** at their pretrained values, with padding excluded from expert-load statistics; experts run dropless |

Validation loss on 1,201 held-out conversations (assistant tokens only): **1.973 → 1.859**.

## Evaluation

### Zero-shot benchmarks (`evals.py`)

| Task | acc_norm | acc | Items | Chance |
|---|---|---|---|---|
| SciQ | 73.8% ± 1.4% | 79.0% | 1,000 | 25% |
| ARC-Easy | 46.1% ± 1.0% | 49.5% | 2,376 | 25% |
| HellaSwag | 35.6% ± 0.5% | 31.0% | 10,042 | 25% |

Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness
prompts (SciQ includes its supporting passage). **acc_norm** divides by the answer's length in characters;
**acc** doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.

### MMLU (`mmlu_eval.py`)

| Format | Accuracy | Questions scored | Average few-shot examples |
|---|---|---|---|
| letter | 24.6% | 14,042 (skipped 0) | 4.9 |
| cloze | 27.4% | 14,042 (skipped 0) | 0.0 |

| Category | Subjects | Questions | Letter | Cloze |
|---|---|---|---|---|
| STEM | 19 | 3,153 | 24.3% | 26.0% |
| Humanities | 13 | 4,705 | 24.9% | 26.0% |
| Social Sciences | 12 | 3,077 | 23.7% | 29.4% |
| Other | 13 | 3,107 | 25.2% | 29.0% |

**Biology & medicine (cloze): 28.6% on 2,094 questions**, +3.6 points against 25% chance (about 3.8 standard errors, significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.

In the **letter** format the model compares " A"–" D", with same-subject examples added while they fit the
context. In the **cloze** format each answer's text is scored by log-probability per byte; at this size the
cloze results are the meaningful ones. Chance is 25%.

### Base vs instruct

Both are scored on the same raw-text prompts, without the chat template, so the numbers are directly comparable. SFT mainly teaches format, turn-taking and when to stop, so large changes in knowledge benchmarks aren't expected. Chance on the multiple-choice tasks is 25%.

| Metric | Base | **Instruct** |
|---|---|---|
| SciQ (acc_norm) | 72.2% | 73.8% |
| ARC-Easy (acc_norm) | 47.1% | 46.1% |
| HellaSwag (acc_norm) | 36.0% | 35.6% |
| MMLU (letter) | 24.5% | 24.6% |
| MMLU (cloze) | 27.8% | 27.4% |
| MMLU biology & medicine (cloze) | 29.9% | 28.6% |

### Compared with its siblings and other small models

| | **This model** | [Pythia-70M](https://huggingface.co/EleutherAI/pythia-70m) | [GPT-2](https://huggingface.co/openai-community/gpt2) | [Supra-50M](https://huggingface.co/SupraLabs/Supra-50M-Base) |
|---|---|---|---|---|
| Parameters (total) | 305.4M | 70.4M | 124.4M | 51.8M |
| Active non-embedding parameters | 23.3M | 18.9M | 85.1M | 35.4M |
| Training tokens | 20,400M | 300B (the Pile) | undisclosed (WebText) | 20B (FineWeb-Edu) |
| SciQ (acc_norm) | 73.8% | 55.2% | 53.2% | 77.2% |
| ARC-Easy (acc_norm) | 46.1% | 35.0% | 42.0% | 52.2% |
| HellaSwag (acc_norm) | 35.6% | – | 29.5% | 31.8% |
| MMLU (cloze) | 27.4% | – | – | – |
| MMLU biology & medicine (cloze) | 28.6% | – | – | – |
| Validation loss (held-out mix) | – | – | – | – |

LittleKedi columns are measured with this repo's `evals.py` and `mmlu_eval.py` on the checkpoints named. Validation
losses are only comparable between models that share a tokenizer and validation set. The other columns are
**published figures**: Pythia-70M from EleutherAI's evaluations as quoted on the
[Wisp-15M card](https://huggingface.co/DedeProGames/Wisp-15M), GPT-2 and Supra-50M from the
[Supra-50M-Base card](https://huggingface.co/SupraLabs/Supra-50M-Base). Their prompts may differ from ours,
so treat gaps of a few points as noise.

## What it can and can't do

This is the instruction-tuned **sparsest medium model**: `medium-wide-sparse-base` (23.3M active non-embedding
parameters per token, 240 experts, 305M in total) fine-tuned for one epoch on ~24K conversations, most of them
written by DeepSeek V4.1 Flash for medium's 2,048-token context. It's the largest LittleKedi chat model: it
answers in fuller, better-organised paragraphs than `moderate-wide-sparse-instruct`, gets the gist of
common-knowledge questions right more often, and writes the simplest Python functions correctly. It still makes
up specific details freely, with the same confidence as the right ones: a research artifact for studying sparsity
and instruction tuning at small scale, not a source of information.

**Who it is**

It answers as **LittleKedi, a small cat from Istanbul who has read a lot** ("kedi" is Turkish for cat), and says
it's a small language model that makes mistakes: *"Who are you?"* → *"I'm LittleKedi! I'm a little cat from
Istanbul, and I've spent a lot of time curled up with books. I'm happy to chat, explain things or help you think
something through."* The persona stays out of ordinary answers.

**What it does**

- ✅ Replies in the chat format and stops cleanly; greets back (*"Hi!"* → *"Hello! I'm LittleKedi. How can I help
  you today?"*, 3 of 3 seeds).
- ✅ Gets the core of common-knowledge questions right more often than not: *"What causes the seasons on
  Earth?"* → *"the tilt of the Earth's axis… about 23.5 degrees"* (3 of 3 seeds name the tilt); *"What is
  photosynthesis?"* → *"the process by which plants convert light energy into chemical energy. It uses sunlight,
  carbon dioxide, and water to create glucose"*; a black hole is *"a region in the universe where gravity is so
  strong that nothing can escape it, even light itself"*.
- ✅ Gives sensible, well-structured lists for everyday advice (sleep: a consistent schedule, a relaxing
  environment, no caffeine, alcohol or screens before bed, relaxation techniques).
- ✅ Writes the very simplest functions correctly: asked for the largest number in a list, it returned
  `return max(lst)` on one seed of three.
- ✅ Shortens a wordy sentence sometimes: *"The committee has made the determination that the proposal will not be
  moving forward at this point in time"* → *"The committee has decided that the proposal is not moving forward."*

**What it can't do**

- ❌ **Specific details reliably.** The main fact can be right and everything around it invented: Paris is *"in the
  southwestern part of the country… with a population of about 36 million"*; Einstein is *"an American
  mathematician"* on one seed and *"a British biochemist… best known for his work on the structure of DNA"* on
  another; one seed answers *"What is the capital of France?"* with *"Louis XIV."*
- ❌ **Arithmetic reliably**: **38.3%** on 300 held-out one- and two-digit sums, differences and products (greedy
  decoding), a little below `moderate-wide-sparse-instruct`'s 43.7%; *"7 + 6"* got 10, 12 and 12.
- ❌ **Working code beyond one-liners.** It writes correctly shaped Python but solves only **3.0%** of 500
  held-out MBPP problems (moderate-instruct: 1.8%); other seeds wrapped the right idea in a wrong
  `max(nums, key=lambda x: x[1])`.
- ❌ **Explaining mechanisms**: how a volcano erupts or how a virus differs from a bacterium comes out as
  fluent, confident nonsense (*"a virus is a tiny, single-stranded virus"*; magma reaching *"10,000 degrees
  Fahrenheit"*).
- ❌ **Staying faithful to a given text**: asked to simplify a sentence, it sometimes reverses its meaning (*"The
  committee has decided to move ahead with the project"*) and then explains changes it didn't make.
- ❌ **Counting what it was asked for**: asked for three sleep tips, it gave five on two seeds of three.

**Benchmarks:** about where the base model was (SciQ 73.8% vs 72.2%, ARC-Easy 46.1% vs 47.1%, HellaSwag 35.6% vs
36.0%, PIQA 63.9% vs 64.7%, acc_norm; MMLU cloze 27.4% vs 27.8%): instruction tuning changed the format, not what
it knows.

**Decoding:** temperature ≈ 0.7, top-p ≈ 0.9, repetition penalty ≈ 1.15 (the GGUF files carry these as
defaults), and a modest maximum length. The context window is 2,048 tokens, prompt and reply together.

## Usage

**llama.cpp** (GGUF, llama.cpp's built-in `deepseek2` architecture; the chat template is embedded):

```sh
llama-cli -m LittleKedi-medium-wide-sparse-instruct-q8_0.gguf -cnv --temp 0.7 --top-p 0.9 --repeat-penalty 1.15
```

**Ollama** (uses the bundled `template` and `params` files): `ollama run hf.co/edededdy/LittleKedi-medium-wide-sparse-instruct:Q8_0`

**PyTorch** (needs the `littlekedi` package from [the training code](https://github.com/EdmundMartin/LittleKedi)):

```python
import json, torch
from safetensors.torch import load_file
from littlekedi import LittleKediModel, ModelConfig
from littlekedi.chat import END, render_prompt
from littlekedi.tokenizer import BPETokenizer

model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
model.load_state_dict(load_file("model.safetensors"))
tok = BPETokenizer.from_file("tokenizer.json")
ids = torch.tensor([render_prompt(tok, [{"role": "user", "content": "What causes the seasons?"}])])
out = model.generate(ids, 200, temperature=0.7, top_p=0.9, repetition_penalty=1.15, stop_ids={tok.token_to_id(END)})
print(tok.decode(out[0, ids.shape[1]:].tolist()).replace(END, "").strip())
```

Or in a terminal: `python chat.py --ckpt <checkpoint>.pt`.

## Files

| File | Contents |
|---|---|
| `model.safetensors` | PyTorch weights (embedding tied with the output head), `littlekedi` parameter names |
| `config.json` | `ModelConfig` for `littlekedi` |
| `tokenizer.json` | Byte-level BPE tokenizer (Hugging Face `tokenizers` format) |
| `chat_template.jinja` | Chat template (Jinja), the same one embedded in the GGUF files |
| `tokenizer_config.json` | Special tokens (`<|end|>` = end of turn), chat template, context length |
| `template` | Ollama chat template |
| `params` | Ollama stop tokens and sampling defaults |
| `LittleKedi-medium-wide-sparse-instruct-f16.gguf` | GGUF (f16) for llama.cpp / Ollama / LM Studio |
| `LittleKedi-medium-wide-sparse-instruct-q8_0.gguf` | GGUF (q8_0) for llama.cpp / Ollama / LM Studio |
| `little_kedi.jpg` | The LittleKedi logo |

## Acknowledgements

- The architecture follows **DeepSeek-V3** (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
- **Muon** (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
- Pretraining data: **FineWeb-Edu** (ODC-BY 1.0).
- Pretraining data: **Cosmopedia v2** (ODC-BY 1.0).
- Pretraining data: **FineMath 4+** (ODC-BY 1.0).
- Pretraining data: **Reddit comments** (no license stated on the dataset card; the comments are Reddit users' content).
- Pretraining data: **github-code-clean** (Apache 2.0, with each file under its own open-source license).
- Pretraining data: **SmolTalk** (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).
- Pretraining data: **Wikipedia** (CC BY-SA 3.0 and GFDL).
- Chat data: see the SFT table above for its sources, each under its own license.

*Card generated 2026-11-01 by `prepare_release.py` from `medium-wide-sparse-sft.pt` (step 311278).*