README.md

---
license: apache-2.0
language:
  - id
library_name: transformers
tags:
  - deepseek
  - moe
  - causal-lm
  - chat
  - cpu
  - indonesian
datasets:
  - neosantara/IndoTalkSFT
base_model: neosantara/wader-100m-base
pipeline_tag: text-generation
---

# lilo-100m — Lightweight Local Intelligence & Optimizer

A ~110M parameter language model implementing the DeepSeek-V4 architecture, fine-tuned for chat/instruction following in Indonesian. This is the **lilo** release of `neosantara/wader-100m`: same architecture and weights, re-packaged explicitly for **CPU-only, low-memory, low-footprint** use.

- **Base**: [neosantara/wader-100m-base](https://huggingface.co/neosantara/wader-100m-base) · pretrained on [neosantara/IndoTalk](https://huggingface.co/datasets/neosantara/IndoTalk)
- **Upstream SFT**: [neosantara/wader-100m](https://huggingface.co/neosantara/wader-100m) on [neosantara/IndoTalkSFT](https://huggingface.co/datasets/neosantara/IndoTalkSFT)
- **Trained from scratch** — no weights from DeepSeek-V4 were used.

## Why lilo

| | wader-100m (original) | **lilo-100m** |
|---|---|---|
| Weights on disk | 421MB fp32 | **119MB** (int8) / **221MB** (fp16) |
| CPU-friendly runtime | bf16 NaN bugs, no KV cache | fp32/fp16 stable, KV cache, inference_mode |
| Runs on 2 vCPU / 2GB RAM VPS | untested | **measured** |

## Variants

| Variant | File | Disk | PPL delta vs fp32 | Greedy outputs |
|---|---|---|---|---|
| `fp16` (default) | `model.fp16.safetensors` | 221MB | +0.08% | 5/5 identical |
| `int8` | `model.int8.safetensors` | 119MB | +0.32% | 5/5 identical |

Notes:
- `int8` is symmetric group-wise quantization (group=64) on all `nn.Linear` layers + embeddings; norms/router/HC params stay fp32. Scales cost ~7MB.
- Both dequantize to fp32 compute on load — the **disk footprint is what shrinks**, because this CPU (Zen2, AVX2-only, no VNNI) does fp32 GEMM as fast as int8 anyway, and fp16 GEMM is *faster* still.
- fp16 is the default because it is both the closest to the original quality **and** the fastest on CPU.

## CPU speed (measured on 2 vCPU AMD EPYC 7K62 "Zen2", AVX2-only, box under load)

| Variant | threads=1 | threads=2 |
|---|---|---|
| fp16 | **14–22 tok/s** | 11–12 tok/s |
| int8 | 9–19 tok/s | 5–8 tok/s |

On this tiny model (320 hidden, 8 layers) thread sync overhead dominates — `--threads 1` is fastest. On 8+ core machines try 2–4.

## Memory safety (no OOM)

Peak RSS measured on a 2 vCPU / 1.9GB RAM VPS: **~0.85GB (int8) / ~1.05GB (fp16)** including the Python/torch runtime. Rules baked into `run_cpu.sh`:

- `OMP_NUM_THREADS` bounded, `MALLOC_ARENA_MAX=1` (glibc per-thread arenas can silently eat GBs)
- `TMPDIR` forced to disk, never `/tmp` when `/tmp` is tmpfs
- prefill chunked at 256 tokens so attention never allocates a big `[S,S]` matrix

## Usage

```bash
git lfs install
git clone https://huggingface.co/hzlo0/lilo-100m
cd lilo-100m
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install transformers safetensors tokenizers accelerate

bash run_cpu.sh                              # chat, default fp16
CUDA_VISIBLE_DEVICES="" python chat_lilo.py --variant int8 --threads 1 "Halo, apa kabar?"
```

Chat template (DeepSeek-V4 style):

```
<|begin_of_sentence|><|User|>Halo<|Assistant|>
```

```python
import torch
from lilo_common import load_lilo, generate_lilo, chat

torch.set_num_threads(1)
tok, model = load_lilo(".", variant="fp16")          # or "int8"
print(chat(tok, model, [{"role": "user", "content": "Halo, apa kabar?"}],
           max_new_tokens=200, temperature=0.7, top_p=0.9))
```

Requires `trust_remote_code`-free loading: the model code ships as plain files (`modeling_deepseek_v4.py`) loaded by `lilo_common.py`. (The same code also works via `AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True)`.)

## Architecture

| Component | Details |
|---|---|
| Parameters | ~110M (41M embeddings, 69M non-embedding) |
| Hidden size | 320 · Layers 8 |
| Attention | 8 heads, 1 KV head (MQA), MLA q_lora_rank=160, grouped low-rank output |
| MoE | 4 routed experts + 1 shared, top-2 routing |
| Hyper-Connections | hc_mult=4, Sinkhorn routing (replaces residual stream) |
| Vocab | 129,280 (DeepSeek-V4 tokenizer) · Context 2,048 |

## What was fixed for CPU in this release

1. **freqs_cis recomputed on demand** — original code left a buffer uninitialized after meta-device loading, corrupting RoPE beyond a few tokens.
2. **KV cache honored** — original `forward` hardcoded `use_cache=False`, recomputing the whole sequence every decode step. This release caches and steps one token.
3. **`return_dict` defaults to True** — transformers 5.x deprecated `config.use_return_dict`, which silently returned tuples.
4. **Expert dtype bridge** — `w2(x.to(w2.weight.dtype))` silently cast activations to int8 under weight-only quantization.
5. **bf16 NaN** — model runs fp32/fp16; no bf16 anywhere.

## Limitations

- Tiny model: 110M params, most capacity in embeddings. Generations are often incoherent or factually wrong.
- Undertrained: 5K pretrain + 3K SFT steps (upstream). Production models train 100K+ steps on trillions of tokens.

## License

Apache-2.0 (inherited from upstream `neosantara/wader-100m`).