---
language:
- wo
license: mit
tags:
- wolof
- low-resource
- african-languages
- moe
- mixture-of-experts
- text-generation
- from-scratch
pipeline_tag: text-generation
---

# smallm-wolof

A **16.5M-parameter sparse Mixture-of-Experts language model trained from
scratch on Wolof**, a Senegalese language with very limited digital
resources. This is a research/proof-of-concept project, not a production
model — see [Limitations](#limitations) before using it for anything beyond
experimentation.

## Why this exists

Wolof has no dedicated open reasoning/generative LLM despite being spoken by
~10M+ people across Senegal, Mauritania, and Gambia. Existing multilingual
LLMs (Gemini, GPT-4o, Gemma, Qwen) are all weak in Wolof because it
represents a tiny fraction of their training data. This project explores
what a **from-scratch, Wolof-first** small model can achieve when the whole
pipeline — tokenizer, architecture, data, training recipe — is designed
around the constraints of a genuinely low-resource language, instead of
Wolof being an afterthought in a 100+-language mix.

## Architecture

| | |
|---|---|
| Parameters | 16.5M |
| Layers | 8 |
| Hidden size | 320 |
| Attention | Grouped-Query Attention (8 heads, 2 KV heads) + RoPE |
| Feed-forward | Sparse MoE, 4 experts, top-2 routing, SwiGLU |
| Vocab size | 8,192 (dedicated BPE tokenizer, not reused from another model) |
| Context length | 512 tokens |
| Embeddings | Tied input/output |

**Design choices, and why:**

- **Dedicated tokenizer.** Reusing Qwen's tokenizer on this corpus measured
  a fertility of only 2.46 chars/token (vs ~4.3 for French) — Wolof text was
  being fragmented far more than necessary. A tokenizer trained on Wolof
  itself improves this to 2.92 chars/token with a 6x smaller vocab (8,192 vs
  ~150K), which matters a lot when the whole model is 16.5M parameters.
- **Sparse MoE, not dense.** Data-constrained scaling research
  ([Muennighoff et al., 2023](https://arxiv.org/abs/2305.16264); ["When Data
  Is Scarce"](https://arxiv.org/abs/2606.01155)) shows sparse architectures
  tolerate multi-epoch training on repeated data better than dense ones —
  directly relevant here since this model saw its ~9.7M-token corpus 18
  times, far below the Chinchilla-optimal token budget for this parameter
  count.
- **Explicit load-balancing loss on the router.** Recent work on
  cross-lingual MoE routing ([Bandarkar et al.,
  2026](https://arxiv.org/html/2605.17598v1)) found that low-resource
  language tokens tend to collapse onto a narrow subset of experts in
  naively-trained MoE models. The auxiliary load-balancing loss here forces
  the router to spread Wolof tokens across all 4 experts instead of letting
  a narrow pathway emerge.

## Training data

~9.7M tokens, unified from multiple sources and deduplicated (exact +
near-duplicate via SimHash): Wikipedia (Wolof), FineWeb2, MADLAD-400,
SMOL-doc, plus smaller lexicon/instruction sources (Jolof, Aya, Gatitos).
GSM8K (7,473 grade-school math problems), machine-translated to Wolof and
quality-checked, is folded directly into the pretraining mix rather than
kept as a separate SFT stage — at this parameter budget, a strict
CPT-then-SFT split wastes capacity; mixing lets the model see the reasoning
pattern throughout training.

**Recipe:** 18 epochs (chosen per data-constrained scaling literature:
repeating data up to ~4 epochs is nearly free, meaningful gains extend to
~16, returns vanish around 40), AdamW, cosine LR schedule with warmup,
bf16 mixed precision. Final validation loss: 3.48 (perplexity ≈32.5).
Train/val loss divergence after epoch ~13 indicates mild overfitting,
consistent with the token budget being well below what this parameter count
would ideally need.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Fallovski/smallm-wolof", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Fallovski/smallm-wolof")

inputs = tokenizer("Sama xarit", return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.85,
    top_k=40,
    repetition_penalty=1.3,  # important: without this the model loops on short/common prompts
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

`trust_remote_code=True` is required — this is a custom architecture, not a
standard `transformers` model class.

## Limitations

Read this before drawing conclusions from the model's output.

- **~9.7M training tokens is very small.** For comparison, Chinchilla-optimal
  scaling would want ~330M tokens for a 16.5M-parameter model. The model
  has learned Wolof surface grammar and the *style* of the corpus (including
  GSM8K-style "step-by-step, then `####` answer" formatting) convincingly,
  but arithmetic in generated reasoning chains is frequently **wrong** — it
  has learned the form of reasoning, not reliable calculation.
- **Repetition loops.** Short or generic prompts (e.g. single common nouns)
  can cause the model to loop on a phrase. Use `repetition_penalty >= 1.3`;
  even then, some prompts still degrade after a few dozen tokens.
- **No independent native-speaker validation of the training corpus.** Part
  of the pretraining data (FineWeb2-derived) mixes standard Wolof
  orthography with phonetic/social-media spelling, and a portion of the
  GSM8K translation was produced by Gemini 2.5 Flash rather than human
  translators — spot checks looked reasonable but no systematic quality
  audit has been done.
- **512-token context**, no long-context capability.
- Not intended for factual reliance, translation accuracy, or any
  production use case.

## Files

- `modeling_smallm_wolof.py` / `configuration_smallm_wolof.py` — custom
  architecture code (loaded via `trust_remote_code=True`)
- `model.safetensors` — trained weights
- `tokenizer.json` — dedicated BPE tokenizer

## Citation

If you build on this, please note it is a research/educational project, not
a peer-reviewed publication. No formal citation is required, but a link back
to this repository is appreciated.