README.md

---
title: GGUF Chat
emoji: 🧠
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
license: apache-2.0
---

# GGUF Chat (Docker Space)

A self-contained Hugging Face **Docker Space** that downloads a GGUF quant of
[`bartowski/google_gemma-3-2b-it-GGUF`](https://huggingface.co/bartowski/google_gemma-3-2b-it-GGUF)
and serves it through a Gradio chat UI — with a live download/load progress
bar on first request, then token-by-token streaming — powered by
[`llama-cpp-python`](https://github.com/abetlen/llama-cpp-python).

The default model is a small (~2B parameter) instruct model chosen
specifically to run comfortably on the **free CPU-basic Spaces tier**. Swap
in a different repo any time via the `GGUF_REPO_ID` environment variable —
no code changes needed.

---

## What's in this repo

| File              | Purpose                                                              |
|-------------------|-----------------------------------------------------------------------|
| `Dockerfile`      | Multi-stage build: tries a prebuilt CPU wheel for `llama-cpp-python` first, falling back to compiling it in a builder stage; ships a slim runtime image with no compilers. |
| `requirements.txt`| Pure-Python runtime deps (`gradio`, `huggingface_hub`).              |
| `app.py`          | Downloads/caches the GGUF, loads it with `llama-cpp-python`, and serves the Gradio UI (progress bar + streaming chat). |
| `README.md`       | This file (also the Space's metadata card, via the YAML frontmatter above). |

This repo is ready to push directly to a new **Docker** Space with no
further edits.

---

## A note on the free Spaces tier

This project is tuned to actually work on the **free CPU basic** tier
end-to-end, which comes with real constraints worth knowing up front:

- **No persistent disk by default.** Model weights re-download on every
  full restart/rebuild (see "Model caching" below) unless you pay for the
  Persistent Storage add-on.
- **2 vCPUs, 16GB RAM.** `N_THREADS` defaults to `2` to match the actual
  core count rather than over-subscribing; `N_CTX` defaults to a modest
  `4096` tokens, well within what a ~1B `Q4_K_M` model needs on this
  hardware.
- **Build minutes are shared/limited infrastructure**, so the Dockerfile
  tries a prebuilt wheel for `llama-cpp-python` before ever compiling from
  source (see "Technical notes" below) — this is the single biggest lever
  for keeping first-time build time low on the free tier.
- If you outgrow any of this — a bigger model, longer context — the fixes
  are all opt-in: upgrade to a paid CPU/GPU hardware tier, or add
  Persistent Storage, and adjust `N_CTX`/`N_THREADS` accordingly. No code
  changes required, since everything is environment-variable driven.

## Deploying to Hugging Face Spaces

1. Create a new Space at <https://huggingface.co/new-space>.
2. Choose **Docker** as the Space SDK (not "Gradio" or "Streamlit" — this
   project builds and runs its own Dockerfile).
3. Pick the **CPU basic (free)** hardware tier — this project is built to
   run entirely on CPU.
4. Push these four files to the Space repo:

   ```bash
   git clone https://huggingface.co/spaces/<your-username>/<your-space-name>
   cd <your-space-name>
   cp /path/to/Dockerfile /path/to/requirements.txt /path/to/app.py /path/to/README.md .
   git add .
   git commit -m "Deploy GGUF chat Space"
   git push
   ```

5. The Space will build the Docker image (typically well under a minute for
   `llama-cpp-python` if a prebuilt wheel is available — see "Technical
   notes" — otherwise 15-30+ minutes compiling from source) and then start
   the container. On first chat message, `app.py` downloads the selected
   GGUF file from the Hub and shows a live progress bar in the UI itself
   (in addition to the Space's **Logs** tab).
6. Once the model finishes loading, replies stream in token-by-token.

No secrets or tokens are required for the default (public) repo. If you
point this at a **gated/private** GGUF repo, add an `HF_TOKEN` secret in the
Space's **Settings → Variables and secrets**; `huggingface_hub` picks it up
automatically.

---

## Configuration (environment variables)

All of these are set with sensible defaults in the `Dockerfile` and can be
overridden per-Space under **Settings → Variables and secrets** without
touching any code:

| Variable              | Default                                              | Description |
|-----------------------|-------------------------------------------------------|--------------|
| `GGUF_REPO_ID`         | `bartowski/google_gemma-3-2b-it-GGUF`                 | Hub repo to pull the GGUF from. |
| `GGUF_FILENAME`        | *(empty = auto-select)*                                | Force an exact filename instead of auto-selecting by quant. |
| `PREFERRED_QUANT`      | `Q4_K_M`                                              | Preferred quantization. Falls back automatically (`Q4_K_S` → `Q5_K_M` → ... → smallest available `.gguf`) if not present. `mmproj` (vision) and `-MTP-` (speculative-decoding draft head) files are skipped by the auto-selector in favor of a plain text-chat quant. |
| `MODEL_CACHE_DIR`      | `/data/models`                                        | Local cache directory for downloaded model weights. |
| `HF_HOME`              | `/data/hf_home`                                       | Cache directory for Hub metadata. |
| `N_CTX`                | `4096`                                                 | Context window (tokens) allocated at load time. Raise this if you swap in a model that needs more headroom and you have the RAM to back it. |
| `N_THREADS`            | `2`                                                     | CPU threads for inference. Matches the free tier's 2 vCPUs by default; raise it if you upgrade hardware. |
| `N_BATCH`              | `256`                                                  | Prompt processing batch size. |
| `MAX_NEW_TOKENS`       | `899`                                                  | Max tokens generated per reply. |
| `TEMPERATURE`          | `0.7`                                                  | Sampling temperature. |
| `TOP_P`                | `0.9`                                                  | Nucleus sampling. |
| `TOP_K`                | `40`                                                   | Top-k sampling. |
| `REPEAT_PENALTY`       | `1.1`                                                  | Repetition penalty. |
| `SYSTEM_PROMPT`        | *(a short "be concise, adapt to the request" prompt — see `app.py`)* | System prompt prepended to every conversation. Override to customize the assistant's behavior. |
| `DOWNLOAD_MAX_RETRIES` | `5`                                                     | Retry attempts (exponential backoff) for the model download. |

---

## Model caching & the free tier's storage caveat

`app.py` downloads the model once into `MODEL_CACHE_DIR` and reuses the
cached file for every subsequent chat request — it will **not** re-download
on every message, and it survives the container going to sleep/waking back
up from inactivity.

However, **the free Spaces tier has no *persistent* storage**: the
container's disk (including `/data`) is rebuilt from scratch whenever the
Space is fully **restarted or rebuilt** (e.g. after a `git push`, a factory
reboot, or an infrastructure migration). In that case, the model will be
re-downloaded once on the next startup — this is a platform limitation, not
a bug in this app. If you need the cache to survive restarts, enable
**Persistent Storage** for the Space (a paid add-on) and point
`MODEL_CACHE_DIR`/`HF_HOME` at the mounted persistent volume (typically
`/data`, which is already the default here).

---

## Local development (outside Docker)

```bash
# Build llama-cpp-python with a build appropriate for your machine:
CMAKE_ARGS="-DGGML_NATIVE=ON" pip install llama-cpp-python
pip install -r requirements.txt

python app.py
# then open http://localhost:7860
```

## Building/running the Docker image locally

```bash
docker build -t gguf-chat-space .
docker run -it -p 7860:7860 gguf-chat-space
# then open http://localhost:7860
```

---

## Technical notes

- **`llama-cpp-python` install strategy**: the builder stage first tries to
  fetch a **prebuilt CPU wheel** from the maintainer's custom index
  (`https://abetlen.github.io/llama-cpp-python/whl/cpu`) using
  `pip wheel --only-binary=:all:`, which fails fast (rather than silently
  falling back to a slow source build) if no matching wheel exists for the
  image's Python ABI/platform. This turns the install from a **15-30+
  minute from-source compile into a roughly one-minute download** on the
  common case.

  Only if no prebuilt wheel is available does the Dockerfile fall back to
  building from source, with `CMAKE_ARGS="-DGGML_NATIVE=OFF -DGGML_AVX2=ON
  -DGGML_FMA=ON -DGGML_F16C=ON"`. `GGML_NATIVE` is deliberately disabled
  because the machine that *builds* the Docker image is not guaranteed to
  be the same CPU that *runs* it; auto-detected "native" builds can
  otherwise crash with `SIGILL` on the Space's actual runner. AVX2/FMA/F16C
  are supported by essentially all modern cloud x86_64 CPUs and give good
  performance without that risk. The compiler toolchain (`build-essential`,
  `cmake`, `ninja-build`, `git`) is only installed in this fallback branch,
  and the fallback build caps itself at 4 parallel compile jobs to avoid
  getting OOM-killed on the free Spaces builder, which can report more CPU
  cores than it has RAM to back a fully parallel C++ build.
- **Chat template**: the `Llama` object is created without a hardcoded
  `chat_format`, so `llama-cpp-python` auto-detects and applies the Jinja
  chat template embedded in the GGUF's own metadata. This is what lets the
  same `app.py` work correctly across different model families (Gemma,
  Llama, Qwen, ...) if you change `GGUF_REPO_ID`, without hardcoding any
  model-specific prompt formatting.
- **Progress bar UI**: the model downloads/loads lazily on the *first* chat
  request rather than at container startup, and `app.py` streams a live
  HTML progress bar (download %, then an animated layer-load indicator)
  into the response pane while that happens, in addition to logging
  progress to the Space's **Logs** tab. The layer-load bar is a smooth
  "still working" animation rather than an exact per-layer readout, since
  `llama-cpp-python`'s Python API doesn't expose real-time load progress.
- **Streaming**: implemented via `llm.create_chat_completion(..., stream=True)`,
  yielding incrementally-growing text into the UI for token-by-token display.
- **GPU layers**: `n_gpu_layers=0` — this Space is CPU-only by design, matching
  the free Spaces hardware tier.
- **File selection**: uses `huggingface_hub.HfApi().model_info(..., files_metadata=True)`
  to inspect all files with sizes, filters out `mmproj` (vision projector)
  and `-MTP-` (speculative decoding draft-head) variants by default, then
  picks the smallest file matching `PREFERRED_QUANT`, falling back through
  a quant-quality-ordered list if needed.