Drew168/azmx-one-v1

AZMX One — 1.87B Indic base model (training in progress)

This is a mid-training checkpoint, not a finished model. It is published so the run has an off-site copy; expect it to be replaced regularly.

parameters	1,872,898,048
step	107,400 of 152,587
tokens seen	422.54B of 600B (70.4%)
train loss	see milestones.log
precision	bf16
status	stable phase — the WSD cooldown begins at 510B

Architecture

Own implementation (AZMXForCausalLM), no Qwen3 in the inheritance chain.

vocab	128,000 (AZMX tokenizer v2, Indic-optimised)
hidden / intermediate	2048 / 6144
layers	32
attention heads / KV heads	16 / 8 (GQA)
head_dim	128
norm	RMSNorm, fp32 accumulation
position	split-halves RoPE, theta 10,000
other	per-head QK-norm, SwiGLU, tied embeddings
context	2048

The embedding is 262.1M of the 1.873B — 14.0% — because a 22-language Indic vocabulary needs 128k entries where an English model needs 32k.

Training

schedule	WSD: warmup 2,288, stable to 129,698, sqrt cooldown to 152,587
peak LR	3e-4, AdamW(0.9, 0.95), wd 0.1, grad clip 1.0
global batch	3,932,160 tokens/step
hardware	8x B200

Mixture: ~58% Indic, 16% English, 12% math, 5% domain, 5% core, 2% code, 2% South Asian. 22 scheduled Indic languages plus English.

Files

• latest.pt — full training state (weights + optimiser). Use this to resume.
• model.safetensors — weights only, bf16. The tied embedding is stored once; tie_word_embeddings reconstructs lm_head on load.

Caveats

Held-out bits-per-byte on low-resource languages regressed between 200B and 350B and is being addressed; do not treat this checkpoint as representative of final quality on Manipuri, Konkani, Dogri, Maithili or Kashmiri.

Apache-2.0.