Samarkand-253M

Samarkand-253M is a 253M-parameter base language model, built from scratch end to end in JAX/Flax NNX on Google's TPU architecture, with a custom 30k byte-level BPE tokenizer.

Training data: FineWeb-Edu and DCLM only. No benchmark training sets were used.

This is a base model: it is not instruction-tuned and has no chat template.

Training

• Framework: JAX / Flax NNX, trained from scratch
• Hardware: one TPU v5e (8 chips for training, less for testing)
• Batch: 131,072 tokens per step
• Optimizer: AdamW, peak LR 1e-3, warmup-stable-decay schedule
• Data: FineWeb-Edu + DCLM

Mid training strategy consisted of micro managing dclm training ratio to preserve fineweb-edu whilst giving general register.

Architecture

26 layers, d_model 896, 14 attention heads with 2 KV heads (GQA), head_dim 64, SwiGLU FFN (d_ff 2560), QK-norm, RoPE (theta 10000), tied embeddings, context length 2048, vocabulary 30,002.

It is stored in the Qwen3 layout, so it loads with Qwen3ForCausalLM in transformers and runs in llama.cpp as the qwen3 architecture.

Files

• model.safetensors, config.json, tokenizer.json, tokenizer_config.json: Hugging Face format
• Samarkand-253m-f16.gguf: llama.cpp (needs a recent build with Qwen3 support)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "dragosMediv/Samarkand-253m"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

inputs = tok("The capital of France is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=40)
print(tok.decode(out[0]))

llama-cli -m Samarkand-253m-f16.gguf -p "The capital of France is"

Evaluation

All zero-shot, fp32, from the Hugging Face weights in this repo, using lm-evaluation-harness (ArithMark-3.0 with its official script).

Benchmark	Chance	acc	acc_norm
ARC-Easy	25	56.94	50.84
ARC-Challenge	25	25.26	26.96
HellaSwag	25	32.68	38.15
PIQA	50	67.25	66.49
WinoGrande	50	50.43	-
OpenBookQA	25	19.60	32.80
SciQ	25	79.20	71.50
BLiMP	50	81.39	-
ArithMark-3.0	25	35.10	35.50

WikiText-2 (test, 1024-token windows): word perplexity 29.71, byte perplexity 1.8855, bits per byte 0.9150.

Limitations

• English only
• Base model: no instruction or safety tuning
• Can produce incorrect or made-up text
• 2048-token context