Kasturi-500M

Status: 🟢 PRETRAINING IN PROGRESS — launched 2026-06-29 07:03 UTC. ETA: ~2026-07-24 (3.5 weeks, 30B tokens, 57,220 optimizer steps). Single 16 GB consumer GPU (NVIDIA RTX 5060 Ti).

Kasturi (ಕಸ್ತೂರಿ) is an open-source bilingual Kannada + English language model, trained from scratch on a single consumer GPU for the Karnataka developer community.

Quick facts


Name	Kasturi (ಕಸ್ತೂರಿ — "musk")
Type	Decoder-only autoregressive transformer
Size	~490M parameters
Languages	Kannada (primary, ~~55% of pretraining) + English (~~40%)
License	Apache 2.0 — commercial + research use, no restrictions
Status	Pre-training in progress (release ETA: weeks)
From scratch?	Yes — no Gemma/Llama/Qwen weights involved

What's planned for launch

🪶 Native Kannada-first behavior (no English bias baked in)
🔄 Built-in KN ↔ EN translation
🧠 Tool-calling fallback when the model doesn't know
📦 GGUF release for Ollama / llama.cpp out of the box
📝 Markdown by default; JSON / HTML / plain-text overrides
🎯 Designed for offline use on consumer hardware

Why bilingual?

Most generalists ship with <1% Kannada (Llama, Gemma); even Sarvam-2B is ~10%. Kasturi flips the ratio. A model that thinks natively in Kannada — not one that translates from English internally — is what the Karnataka community has been missing.

What's NOT here yet

Model weights — pre-training in progress
Full technical model card
Benchmark numbers
Inference examples
Tokenizer file

All of the above will land when the model finishes training. Watch this space.

Status updates

Follow progress on the project's working repository (link coming soon).

Citation

@misc{kasturi-2026,
  title     = {Kasturi: A Bilingual Kannada-English LLM Trained from Scratch on a Consumer GPU},
  author    = {Anand Kaman},
  year      = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/anandkaman/kasturi-500m}}
}

Training progress

Phase	Status	Date
Phase 0-2: data acquisition + tokenizer + mmap	✅ Complete	through 2026-06-29
Phase 3: pretraining (30B tokens)	🟢 IN PROGRESS	started 2026-06-29 07:03 UTC
Phase 4: SFT (instruction tuning)	⏳ Queued	post-pretraining
Phase 5: eval gates (MILU-KN, GSM8K-KN, cloze suite)	⏳ Queued	post-SFT
Phase 6: release (HF + GGUF + Ollama)	⏳ Queued	post-eval

Phase 3 schedule

Steps 0 → 2,000 — warmup (LR 0 → 3e-4)
Steps 2,000 → 51,498 — stable (constant LR 3e-4, main data mix: KN 65% / EN 30% / codemix 5%)
Steps 51,498 → 57,220 — annealing (LR linear → 0, anneal mix: KN 50 / EN 20 / codemix 5 / math 12 / instruction 8 / code 5)
Effective batch: 524,288 tokens/step (bf16 mixed-precision, gradient checkpointing, intra-document attention masking)
Checkpoints: weight-only safetensors every 1K steps, full resume every 2.5K steps

Architecture (locked, see model files for full spec)

538.7M parameters · Llama-style decoder-only
32 layers · d_model 1024 · d_ff 3584 (3.5× ratio) · GQA 16/4
4K native context · RoPE θ=500K · Dynamic-YaRN inference extension to 8K
100K SentencePiece BPE vocab · KN fertility 2.09 (beats IndicSuperTokenizer 2.19)
RMSNorm + RoPE + SwiGLU + tied embeddings + intra-doc attention mask

License

Apache 2.0 — see LICENSE once released.

Built solo, from scratch, for Karnataka. By @anandkaman.

Downloads last month: -; Downloads are not tracked for this model. How to track