thanglq150188/gemma4-e4b-mini

Gemma-4 E4B (EN+VI vocab-pruned) — Q8_0 GGUF

Vocabulary-pruned build of principled-intelligence/gemma-4-E4B-it-text-only: the token vocabulary is reduced to 69,246 entries covering English + Vietnamese, shrinking the embedding/LM-head so the q8_0 model fits comfortably on a 12 GB consumer GPU (~5.0 GB VRAM for weights, ~60 tok/s single-stream on an RTX 3060).

Format	GGUF q8_0 (5.2 GB)
Architecture	Gemma-4 E4B text-only — 42 layers, hidden 2560, per-layer input embeddings, sliding+full hybrid attention
Vocab	69,246 (EN+VI pruned)
Context	131,072 tokens
Chat template	Gemma-4 `<

Requirements

• GPU path: NVIDIA GPU with ≥ 8 GB VRAM (12 GB recommended for long context), recent NVIDIA driver, and — for Docker — nvidia-container-toolkit.
• CPU path: works too (llama.cpp CPU build), just slower; needs ~6 GB free RAM.
• To download: pip install -U huggingface_hub (gives you the hf CLI). No Python libraries are needed to serve — llama.cpp is self-contained.

1. Download the model

hf download thanglq150188/gemma4-e4b-mini \
  --local-dir ./models
# -> ./models/gemma4-e4b-envi-pruned-q8_0.gguf
# -> ./models/gemma4_chat.jinja

2. Serve

Option A — Docker (recommended, this is the tested production setup)

Uses the official llama.cpp server image with CUDA:

docker run -d --name gemma4-e4b \
  --gpus all \
  -p 8080:8080 \
  -v "$(pwd)/models:/models" \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/gemma4-e4b-envi-pruned-q8_0.gguf \
  --host 0.0.0.0 --port 8080 \
  -ngl 99 -c 131072 -np 16 \
  -fa on --cache-reuse 256 \
  --jinja --chat-template-file /models/gemma4_chat.jinja

Flag meanings, tune to taste:

• -ngl 99 — offload all layers to GPU
• -c 131072 — context window (lower, e.g. -c 16384, to save VRAM)
• -np 16 — 16 parallel request slots (server throughput; each slot splits the context)
• -fa on — flash attention
• --cache-reuse 256 — prefix-cache reuse across requests (big win for shared system prompts)
• --jinja --chat-template-file ... — required: applies the Gemma-4 <|turn> chat template; without it the model receives a wrong prompt format and quality collapses

For CPU-only, use the ghcr.io/ggml-org/llama.cpp:server image and drop --gpus all and -ngl 99.

Option B — Native llama.cpp binary (no Docker)

# prebuilt releases: https://github.com/ggml-org/llama.cpp/releases  (or build with cmake)
llama-server -m ./models/gemma4-e4b-envi-pruned-q8_0.gguf \
  --host 0.0.0.0 --port 8080 -ngl 99 -c 131072 -np 16 \
  -fa on --cache-reuse 256 --jinja \
  --chat-template-file ./models/gemma4_chat.jinja

3. Call it (OpenAI-compatible API)

llama-server exposes an OpenAI-compatible endpoint at /v1:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "Xin chào! Tóm tắt giúp tôi lợi ích của điện gió ngoài khơi."}
  ],
  "max_tokens": 256, "temperature": 0.7
}'

Or from Python:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="none")
resp = client.chat.completions.create(
    model="gemma4-e4b",  # name is ignored by llama-server, any string works
    messages=[{"role": "user", "content": "Việt Nam có bao nhiêu tỉnh thành?"}],
    max_tokens=256,
)
print(resp.choices[0].message.content)

Health check: curl http://localhost:8080/health → {"status":"ok"} when the model is loaded.

Caveats

• Text-only (vision tower removed in the upstream text-only base).
• Because the vocab is pruned to EN+VI, tokenization of other languages degrades (falls back to byte pieces) — expect worse quality and higher token counts outside English/Vietnamese.
• The bundled gemma4_chat.jinja is mandatory; Gemma-3-style start_of_turn templates will not work with Gemma-4's <|turn> tokens.