wader-100m 🐟

A small ~110M parameter language model implementing the DeepSeek-V4 architecture, fine-tuned for chat/instruction following in Indonesian. Trained from scratch — no weights from DeepSeek-V4 were used.

• Pretrained base model: neosantara/wader-100m-base
• This model: SFT on neosantara/IndoTalkSFT

Architecture

This model implements key DeepSeek-V4 innovations at a miniature scale:

Component	Details
Parameters	~110M total (41M embeddings, 69M non-embedding)
Hidden size	320
Layers	8
Attention heads	8 (1 KV head — MQA-style)
MLA	Multi-head Latent Attention with q_lora_rank=160
MoE	4 routed experts + 1 shared, top-2 routing
Hyper-Connections	hc_mult=4, Sinkhorn routing (replacing residual connections)
MTP	1 next-token prediction layer
Vocab	129,280 (DeepSeek-V4 tokenizer)
Context	2,048 tokens

Training

Stage 1: Pretraining

• Dataset: neosantara/IndoTalk (396k dialogs)
• Steps: 5,000 | Tokens: ~327M
• Batch: 32 effective | Seq length: 2,048
• Hardware: 1× NVIDIA RTX 5090 Blackwell

SFT Training Metrics

Metric	Value
Final Train Loss	~2.60 (Cross-Entropy at Step 3000)
Token Accuracy	~51.71% (Predicting assistant responses)

Stage 2: SFT (this model)

• Dataset: neosantara/IndoTalkSFT (~160K conversations & instructions)
• Steps: 3,000 | Tokens: ~96M
• Batch: 16 effective | Seq length: 2,048
• Hardware: 1× NVIDIA RTX 5090 Blackwell

SFT Training Metrics

Metric	Value
Final Train Loss	~2.60 (Cross-Entropy at Step 3000)
Token Accuracy	~51.71% (Predicting assistant responses)

Usage

import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

# IMPORTANT: This model requires a modern GPU (Ampere/Ada/Hopper) that supports bfloat16.
# Running on CPU or older GPUs (like T4) will result in NaN generation errors.
device = "cuda" if torch.cuda.is_available() else "cpu"

repo_id = "neosantara/wader-100m"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id, 
    trust_remote_code=True, 
    torch_dtype=torch.bfloat16
).to(device)

# Chat
messages = [{"role": "user", "content": "Halo, apa kabar?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(device)

outputs = model.generate(
    input_ids, 
    max_new_tokens=200, 
    temperature=0.7, 
    top_p=0.9, 
    pad_token_id=tokenizer.eos_token_id,
    do_sample=True
)
print(tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True))

Limitations

• Tiny model: 110M params with 129K vocabulary — most capacity goes to embeddings. Generations are often incoherent or factually wrong.
• Undertrained: Only 5K pretrain + 3K SFT steps. Production models train for 100K+ steps on trillions of tokens.
• bf16 NaN: Use fp32 — the Hyper-Connections architecture produces values that overflow bf16 range at this scale.
• Custom code: Requires trust_remote_code=True.