lilo-100m

Lightweight Local Intelligence & Optimizer

A ~110M parameter Indonesian chat model built on the DeepSeek-V4 architecture, re-packaged from neosantara/wader-100m for one purpose: running fast and stable on CPU, on a tiny VPS, without ever OOMing.

• Trained from scratch upstream (no DeepSeek-V4 weights used): base neosantara/wader-100m-base → SFT neosantara/wader-100m
• Same weights, same quality — re-packaged: quantized, de-bugged, and tuned for CPU decode.

---

What you get

Variant	Weights file	Size	Quality vs original fp32	CPU speed (2 vCPU)
fp16 (default)	model.fp16.safetensors	221MB	PPL +0.08%, outputs 5/5 identical	67 tok/s warm · 14–22 tok/s under load
int8	model.int8.safetensors	119MB (−73%)	PPL +0.32%, outputs 5/5 identical	9–19 tok/s

• int8 = symmetric, group-wise (group=64) weight-only quantization on every nn.Linear + embedding. Norms, MoE router, and Hyper-Connection params stay fp32.
• On a Zen2/AVX2 CPU (no VNNI), fp16 GEMM beats int8 — so int8's win is disk size and RAM, fp16's win is speed. Pick by constraint.

Quick start

git lfs install
git clone https://huggingface.co/hzlo0/lilo-100m
cd lilo-100m

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install transformers safetensors tokenizers accelerate

bash run_cpu.sh                                  # interactive chat (fp16)
python chat_lilo.py "Halo, apa kabar?"            # one-shot
python chat_lilo.py --variant int8 "Halo"         # 119MB variant

Flags: --variant fp16|int8 · --threads · --max · --temp · --top-p · --rep · --system · --path

Minimal Python:

import torch
from lilo_common import load_lilo, chat

torch.set_num_threads(1)
tok, model = load_lilo(".", variant="fp16")          # or "int8"
print(chat(tok, model, [{"role": "user", "content": "Halo, apa kabar?"}]))

trust_remote_code is not required — modeling_deepseek_v4.py ships alongside the weights and is loaded directly. (AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True) also works if you prefer.)

Made stable on CPU

The upstream checkpoint was GPU-only and broken for CPU. Fixed in this release:

1. RoPE corrupted on CPU — freqs_cis was left as an uninitialized buffer after meta-device loading → garbage logits past a few tokens. Now recomputed on demand.
2. No KV cache — the original forward hardcoded use_cache=False, recomputing the entire sequence for every generated token. Now cached + single-token stepping.
3. Broken return type — under transformers 5.x, config.use_return_dict is deprecated and silently returned tuples instead of .logits.
4. int8 dtype bridge — expert activations were cast to int8 through w2.weight.dtype under weight quantization.
5. No bf16 anywhere — runs fp16/fp32, no NaN overflow.

Never OOM

Measured peak RSS on a 2 vCPU / 1.9GB RAM VPS, from a clean clone:

• fp16: ~440MB · int8: ~490MB (weights + torch runtime)

Baked into run_cpu.sh:

• OMP_NUM_THREADS + MALLOC_ARENA_MAX=1 — stops glibc from reserving one arena per thread
• TMPDIR forced to disk, never tmpfs (/tmp counts as RAM!)
• prefill chunked at 256 tokens — never allocates a big [S, S] attention matrix

Identity — "siapa namamu?"

This model is the wader-100m weights retrained under a new name — it was never taught the name LILO, so asking "siapa namamu?" makes it deflect or change topic (normal for an undertrained 110M model). Set identity via system prompt:

python chat_lilo.py --system "Namamu adalah LILO (Lightweight Local Intelligence & Optimizer), asisten buatan hzlo0." "Siapa namamu?"

Keep system prompts short and explicit — long ones get ignored. To bake the identity in, do a small SFT run (1–5k dialogs mentioning LILO) on top of this repo.

Architecture

Component	Details
Parameters	~110M (41M embeddings, 69M non-embedding)
Hidden size	320 · 8 layers
Attention	8 heads, 1 KV head (MQA), MLA with q_lora_rank=160, grouped low-rank output projection
MoE	4 routed experts + 1 shared, top-2 routing
Hyper-Connections	hc_mult=4, Sinkhorn routing (replaces the residual stream)
Vocab	129,280 (DeepSeek-V4 tokenizer) · context 2,048 tokens

Limitations

• Small model. 110M params, most capacity in embeddings. Generations are frequently incoherent or factually wrong.
• Undertrained. 5K pretrain + 3K SFT steps upstream. Production models train 100K+ steps on trillions of tokens.
• Name-blind. See the identity section above.

License

Apache-2.0, inherited from neosantara/wader-100m.