---
title: README
emoji: 🪷
colorFrom: yellow
colorTo: red
sdk: static
pinned: false
---

# Muse Mesh

We build language technology for **Sanskrit** and other Indic languages: corpora, tokenizers and language models trained from scratch. Every training run we do, finished or live, is published at the **[Sansar Lab](https://saansar.com)**.

## Sansar: Sanskrit-only language models

**Sansar** is a family of small causal language models trained only on Sanskrit text, with a Sanskrit tokenizer of its own. Each model is trained from scratch; none is a fine-tune of an English model.

| Model | Parameters | Bits per byte (held-out, ex-Gītā) ↓ |
|---|---|---|
| [sansar-700m](https://huggingface.co/MuseMesh/sansar-700m) | 704M | **0.5307** |
| [sansar-350m](https://huggingface.co/MuseMesh/sansar-350m) | 318M | 0.5547 |
| [sansar-125m](https://huggingface.co/MuseMesh/sansar-125m) | 97M | 0.6039 |
| [sansar-60m](https://huggingface.co/MuseMesh/sansar-60m) | 63M | 0.6434 |
| [sansar-20m](https://huggingface.co/MuseMesh/sansar-20m) | 27M | 0.7177 |

Scores are measured on frozen held-out Sanskrit sets: DCS gold sentences, prose, Vedic and out-of-domain texts. Held-out text is masked out of training. Every model card gives the per-set numbers, the training data and a version history.

Scored the same way, general base models need more bits per byte: Gemma 3 4B 0.6965, Qwen3-4B 0.7071, Llama 3.2 3B 0.7122, Sarvam-1 0.7465. sansar-700m beats all four on every held-out set with 704M parameters (Krutrim-2 12B, scored earlier in a pass that is not strictly comparable, is not in this list). A live demo is at **[saansar.com/demo](https://saansar.com/demo)**.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("MuseMesh/sansar-700m", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("MuseMesh/sansar-700m", trust_remote_code=True)
ids = tok("संस्कृतं नाम दैवी वाक्", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=60, do_sample=True, top_k=100)[0]))
```

## English and Math models

Small models trained from scratch with the same recipe (Muon, rotary positions, QK-norm, squared ReLU) and a shared 32k tokenizer, released with their evaluation sets and the exact list of training documents. [Collection](https://huggingface.co/collections/MuseMesh/mume-english-and-math-language-models-6ac4f187f33abd6c9a9cdde6).

| Model | Parameters | Trained on | Result |
|---|---|---|---|
| [mume-english-125m](https://huggingface.co/MuseMesh/mume-english-125m) | 134M | 1.33B FineWeb tokens | 1.106 bits per byte on frozen FineWeb val; on the validation stream 1.132 vs 1.176 for a plain-AdamW GPT-2 124M at the same tokens (3.7% lower) |
| [mume-math-125m](https://huggingface.co/MuseMesh/mume-math-125m) | 134M | 1.33B OpenWebMath tokens | MATH test (clean) 62.6% fewer bits than the English model; branch `sft` is a GSM8K/MATH fine-tune |

- **[mume-tokenizer-32k](https://huggingface.co/MuseMesh/mume-tokenizer-32k)**: the shared 32k unigram tokenizer.
- **[mume-eval-suites](https://huggingface.co/datasets/MuseMesh/mume-eval-suites)**: the frozen test sets behind every number (FineWeb val, WikiText-103, enwik8, text8, GSM8K, MATH, OpenWebMath val) with contamination-clean variants, plus training manifests.

## Data and tokenizer

- **[Sansar Sanskrit Corpus](https://huggingface.co/datasets/MuseMesh/sansar-sanskrit-corpus)**: 252M words of openly licensed Sanskrit from 21 sources, in Devanagari. It is split into one config per licence (CC BY, CC BY-SA, CC0, ODC-By, Apache-2.0, MIT), and every record carries its source, licence and attribution. Text overlapping our evaluation sets is removed.
- **[sansar-sanskrit-tokenizer](https://huggingface.co/MuseMesh/sansar-sanskrit-tokenizer)**: an 8k unigram SentencePiece model over SLP1 transliteration with a Devanagari wrapper. 2.6 to 3.9 tokens per Sanskrit word with only 8k entries.

## Licences

The corpus records keep their source licences. Model weights and the tokenizer are released under **CC BY-NC 4.0** for non-commercial research use, and the modelling code under **Apache-2.0**. For other uses, contact us.

**Links:** [muse-mesh.com](https://muse-mesh.com) · [GitHub](https://github.com/muse-mesh) · kushal@muse-mesh.com