README.md

metadata

title: GGUF Chat
emoji: 🧠
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
license: apache-2.0

GGUF Chat (Docker Space)

A self-contained Hugging Face Docker Space that downloads a GGUF quant of bartowski/google_gemma-3-2b-it-GGUF and serves it through a Gradio chat UI — with a live download/load progress bar on first request, then token-by-token streaming — powered by llama-cpp-python.

The default model is a small (~2B parameter) instruct model chosen specifically to run comfortably on the free CPU-basic Spaces tier. Swap in a different repo any time via the GGUF_REPO_ID environment variable — no code changes needed.

---

What's in this repo

File	Purpose
Dockerfile	Multi-stage build: tries a prebuilt CPU wheel for llama-cpp-python first, falling back to compiling it in a builder stage; ships a slim runtime image with no compilers.
requirements.txt	Pure-Python runtime deps (gradio, huggingface_hub).
app.py	Downloads/caches the GGUF, loads it with llama-cpp-python, and serves the Gradio UI (progress bar + streaming chat).
README.md	This file (also the Space's metadata card, via the YAML frontmatter above).

This repo is ready to push directly to a new Docker Space with no further edits.

---

A note on the free Spaces tier

This project is tuned to actually work on the free CPU basic tier end-to-end, which comes with real constraints worth knowing up front:

• No persistent disk by default. Model weights re-download on every full restart/rebuild (see "Model caching" below) unless you pay for the Persistent Storage add-on.
• 2 vCPUs, 16GB RAM. N_THREADS defaults to 2 to match the actual core count rather than over-subscribing; N_CTX defaults to a modest 4096 tokens, well within what a ~1B Q4_K_M model needs on this hardware.
• Build minutes are shared/limited infrastructure, so the Dockerfile tries a prebuilt wheel for llama-cpp-python before ever compiling from source (see "Technical notes" below) — this is the single biggest lever for keeping first-time build time low on the free tier.
• If you outgrow any of this — a bigger model, longer context — the fixes are all opt-in: upgrade to a paid CPU/GPU hardware tier, or add Persistent Storage, and adjust N_CTX/N_THREADS accordingly. No code changes required, since everything is environment-variable driven.

Deploying to Hugging Face Spaces

1. Create a new Space at https://huggingface.co/new-space.

2. Choose Docker as the Space SDK (not "Gradio" or "Streamlit" — this project builds and runs its own Dockerfile).

3. Pick the CPU basic (free) hardware tier — this project is built to run entirely on CPU.

4. Push these four files to the Space repo:

   git clone https://huggingface.co/spaces/<your-username>/<your-space-name>
   cd <your-space-name>
   cp /path/to/Dockerfile /path/to/requirements.txt /path/to/app.py /path/to/README.md .
   git add .
   git commit -m "Deploy GGUF chat Space"
   git push
   

5. The Space will build the Docker image (typically well under a minute for llama-cpp-python if a prebuilt wheel is available — see "Technical notes" — otherwise 15-30+ minutes compiling from source) and then start the container. On first chat message, app.py downloads the selected GGUF file from the Hub and shows a live progress bar in the UI itself (in addition to the Space's Logs tab).

6. Once the model finishes loading, replies stream in token-by-token.

No secrets or tokens are required for the default (public) repo. If you point this at a gated/private GGUF repo, add an HF_TOKEN secret in the Space's Settings → Variables and secrets; huggingface_hub picks it up automatically.

---

Configuration (environment variables)

All of these are set with sensible defaults in the Dockerfile and can be overridden per-Space under Settings → Variables and secrets without touching any code:

Variable	Default	Description
GGUF_REPO_ID	bartowski/google_gemma-3-2b-it-GGUF	Hub repo to pull the GGUF from.
GGUF_FILENAME	(empty = auto-select)	Force an exact filename instead of auto-selecting by quant.
PREFERRED_QUANT	Q4_K_M	Preferred quantization. Falls back automatically (Q4_K_S → Q5_K_M → ... → smallest available .gguf) if not present. mmproj (vision) and -MTP- (speculative-decoding draft head) files are skipped by the auto-selector in favor of a plain text-chat quant.
MODEL_CACHE_DIR	/data/models	Local cache directory for downloaded model weights.
HF_HOME	/data/hf_home	Cache directory for Hub metadata.
N_CTX	4096	Context window (tokens) allocated at load time. Raise this if you swap in a model that needs more headroom and you have the RAM to back it.
N_THREADS	2	CPU threads for inference. Matches the free tier's 2 vCPUs by default; raise it if you upgrade hardware.
N_BATCH	256	Prompt processing batch size.
MAX_NEW_TOKENS	899	Max tokens generated per reply.
TEMPERATURE	0.7	Sampling temperature.
TOP_P	0.9	Nucleus sampling.
TOP_K	40	Top-k sampling.
REPEAT_PENALTY	1.1	Repetition penalty.
SYSTEM_PROMPT	(a short "be concise, adapt to the request" prompt — see app.py)	System prompt prepended to every conversation. Override to customize the assistant's behavior.
DOWNLOAD_MAX_RETRIES	5	Retry attempts (exponential backoff) for the model download.

---

Model caching & the free tier's storage caveat

app.py downloads the model once into MODEL_CACHE_DIR and reuses the cached file for every subsequent chat request — it will not re-download on every message, and it survives the container going to sleep/waking back up from inactivity.

However, the free Spaces tier has no persistent storage: the container's disk (including /data) is rebuilt from scratch whenever the Space is fully restarted or rebuilt (e.g. after a git push, a factory reboot, or an infrastructure migration). In that case, the model will be re-downloaded once on the next startup — this is a platform limitation, not a bug in this app. If you need the cache to survive restarts, enable Persistent Storage for the Space (a paid add-on) and point MODEL_CACHE_DIR/HF_HOME at the mounted persistent volume (typically /data, which is already the default here).

---

Local development (outside Docker)

# Build llama-cpp-python with a build appropriate for your machine:
CMAKE_ARGS="-DGGML_NATIVE=ON" pip install llama-cpp-python
pip install -r requirements.txt

python app.py
# then open http://localhost:7860

Building/running the Docker image locally

docker build -t gguf-chat-space .
docker run -it -p 7860:7860 gguf-chat-space
# then open http://localhost:7860

---

Technical notes

• llama-cpp-python install strategy: the builder stage first tries to fetch a prebuilt CPU wheel from the maintainer's custom index (https://abetlen.github.io/llama-cpp-python/whl/cpu) using pip wheel --only-binary=:all:, which fails fast (rather than silently falling back to a slow source build) if no matching wheel exists for the image's Python ABI/platform. This turns the install from a 15-30+ minute from-source compile into a roughly one-minute download on the common case.

  Only if no prebuilt wheel is available does the Dockerfile fall back to building from source, with CMAKE_ARGS="-DGGML_NATIVE=OFF -DGGML_AVX2=ON -DGGML_FMA=ON -DGGML_F16C=ON". GGML_NATIVE is deliberately disabled because the machine that builds the Docker image is not guaranteed to be the same CPU that runs it; auto-detected "native" builds can otherwise crash with SIGILL on the Space's actual runner. AVX2/FMA/F16C are supported by essentially all modern cloud x86_64 CPUs and give good performance without that risk. The compiler toolchain (build-essential, cmake, ninja-build, git) is only installed in this fallback branch, and the fallback build caps itself at 4 parallel compile jobs to avoid getting OOM-killed on the free Spaces builder, which can report more CPU cores than it has RAM to back a fully parallel C++ build.

• Chat template: the Llama object is created without a hardcoded chat_format, so llama-cpp-python auto-detects and applies the Jinja chat template embedded in the GGUF's own metadata. This is what lets the same app.py work correctly across different model families (Gemma, Llama, Qwen, ...) if you change GGUF_REPO_ID, without hardcoding any model-specific prompt formatting.

• Progress bar UI: the model downloads/loads lazily on the first chat request rather than at container startup, and app.py streams a live HTML progress bar (download %, then an animated layer-load indicator) into the response pane while that happens, in addition to logging progress to the Space's Logs tab. The layer-load bar is a smooth "still working" animation rather than an exact per-layer readout, since llama-cpp-python's Python API doesn't expose real-time load progress.

• Streaming: implemented via llm.create_chat_completion(..., stream=True), yielding incrementally-growing text into the UI for token-by-token display.

• GPU layers: n_gpu_layers=0 — this Space is CPU-only by design, matching the free Spaces hardware tier.

• File selection: uses huggingface_hub.HfApi().model_info(..., files_metadata=True) to inspect all files with sizes, filters out mmproj (vision projector) and -MTP- (speculative decoding draft-head) variants by default, then picks the smallest file matching PREFERRED_QUANT, falling back through a quant-quality-ordered list if needed.