IndoMHA-512

An experimental Indonesian decoder-only language model trained from scratch on a single consumer GPU, including its tokenizer.

This repository holds the standard Multi-Head Attention (MHA) baseline from the research project IndoChat-Scratch: A Consumer-GPU Study of MHA and MLA-Inspired 150M-Parameter Decoder-Only Language Models for Indonesian. It serves as the control against which RaiRamones/indomla-512 is evaluated.

⚠️ Early-stage checkpoint — 30,000 of 178,000 planned steps (~16.9%), ~1.97B tokens processed. This is a base model: no instruction tuning, no RLHF, no safety alignment. It predicts the next token; it does not follow instructions.

Model Details

Property	Value
Parameters	~150.8M
Layers / Heads	14 / 14
Hidden Size / Head Dim	896 / 64
FFN	SwiGLU (2,432)
Normalization	RMSNorm, pre-norm
Position Encoding	RoPE
Context Length	512 tokens
Vocab Size	16,000 (custom byte-level BPE, NFKC)
Precision	bfloat16
Embedding	Tied input/output

Implemented manually in PyTorch — not an AutoModelForCausalLM architecture, and not initialized from any pretrained model.

Training

Trained on a single RTX 4060 Laptop GPU (8 GB VRAM) with bf16 mixed precision.

Parameter	Value		Parameter	Value
Optimizer	AdamW		Micro batch	2
Betas	0.9 / 0.95		Grad accum	64
Weight decay	0.1		Tokens/step	65,536
Peak LR	2e-4		Grad clip	1.0
Min LR	2e-5		Seed	1337
Warmup	2,000 steps		Schedule	Cosine decay

Corpus: Indonesian Wikipedia + CulturaX — 11.65B train / 118M validation tokens. Preprocessing covers NFKC normalization, HTML and URL stripping, repeated-character normalization, Indonesian-language heuristic filtering, minimum-length filtering, and exact deduplication via 64-bit BLAKE2b hashing. Validation split drawn probabilistically at p=0.01.

The MHA and MLA runs share the same corpus, tokenizer, and optimization settings.

Known corpus issue: this checkpoint's training data contains some Indonesian online-gambling and web-spam content. A stronger filter was added after the corpus was built, so generations may occasionally reproduce those patterns. Both checkpoints in the comparison are affected equally.

Evaluation

Corrected full-validation evaluation over 5,000 non-overlapping windows (~2.56M tokens), sampled across the complete validation split with identical samples for both architectures.

An earlier in-training estimator sampled overlapping windows clustered near the start of the validation array, producing optimistic perplexity. Use the corrected numbers below for any comparison.

Metric	IndoMHA	IndoMLA
Parameters	~150.8M	~142.8M
Full-validation PPL	25.491	25.590
Training throughput	9,922 tok/s	9,035 tok/s
Training peak VRAM	5.60 GB	5.57 GB
KV values / token / layer	1,792	480
KV cache @ 512 tokens	25.7 MB	6.9 MB
Prefill @ 512	16,408 tok/s	15,504 tok/s
Decode @ 512	49.0 tok/s	34.3 tok/s
Peak inference VRAM	3.36 GB	3.31 GB

Finding: the 0.39% perplexity gap is small, and checkpoint rankings flipped repeatedly during training — so this experiment does not establish a language-modeling quality advantage for either architecture. The meaningful difference is the trade-off: MHA decodes faster, while the MLA-inspired variant needs ~3.73× less KV cache (1,792 → 480 values/token/layer). Under a fixed 6 GB KV budget that is roughly 233 vs. 871 concurrent sequence caches.

The MLA decode penalty reflects an unoptimized implementation that reconstructs per-head representations from latent states, without projection absorption or fused kernels.

Intended Use

Research on Indonesian language modeling, decoder-only architectures, MHA vs. MLA ablations, KV-cache efficiency, tokenizer design, and consumer-GPU training. Also usable as a compact Indonesian base model for experimental continuation training.

Out of scope: medical, legal, or financial advice; credit decisions; autonomous or safety-critical systems; any factual information system without verification.

Limitations

• Early checkpoint (16.9% of schedule) — not final architecture performance.
• Single seed (1337) — small MHA/MLA differences may be seed variance.
• No standardized Indonesian downstream benchmark yet; evaluation is perplexity, throughput, and memory only.
• Corpus contamination — see the note above.
• Hallucination — output may be fabricated, inconsistent, or unsupported by the corpus.
• Repetition and drift in longer generations, expected at this scale and stage.
• Weak instruction following — apparent compliance is an artifact of instruction-like text in the corpus, not a capability.
• Limited reasoning — arithmetic, multi-step logic, planning, and factual QA are unreliable.
• Implementation-dependent benchmarks — one PyTorch implementation, one GPU, one environment. Not universal characteristics of MHA.

Reproducibility

seed 1337 · context 512 · vocab 16,000 · AdamW
LR 2e-4 → 2e-5 · warmup 2,000 · cosine decay
micro batch 2 · grad accum 64 · 65,536 tokens/step
bfloat16 · RTX 4060 Laptop 8GB

Future Work

Continue to the full 178K-step schedule; evaluate additional checkpoints and multiple seeds; standardized Indonesian downstream evaluation; bits-per-byte and other tokenizer-independent metrics for cross-model comparison against GPT-2-class and TinyLlama-class baselines; longer context; quantized inference; Indonesian SFT and instruction-tuned variants.

Citation

@misc{muslim2026indochatscratch,
  title        = {IndoChat-Scratch: A Consumer-GPU Study of MHA and MLA-Inspired 150M-Parameter Decoder-Only Language Models for Indonesian},
  author       = {Muslim, Azis},
  year         = {2026},
  note         = {Research project and experimental model checkpoints}
}

Author & License

Azis Muslim — RaiRamones

Apache-2.0. Users are responsible for ensuring downstream use complies with applicable laws, regulations, and dataset licenses.

Disclaimer: experimental research model. May generate incorrect, biased, repetitive, or fabricated content. Verify outputs before any real-world use.