Spaces:
Sleeping
Sleeping
| title: GGUF Chat | |
| emoji: 🧠 | |
| colorFrom: indigo | |
| colorTo: purple | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| license: apache-2.0 | |
| # GGUF Chat (Docker Space) | |
| A self-contained Hugging Face **Docker Space** that downloads a GGUF quant of | |
| [`bartowski/google_gemma-3-2b-it-GGUF`](https://huggingface.co/bartowski/google_gemma-3-2b-it-GGUF) | |
| and serves it through a Gradio chat UI — with a live download/load progress | |
| bar on first request, then token-by-token streaming — powered by | |
| [`llama-cpp-python`](https://github.com/abetlen/llama-cpp-python). | |
| The default model is a small (~2B parameter) instruct model chosen | |
| specifically to run comfortably on the **free CPU-basic Spaces tier**. Swap | |
| in a different repo any time via the `GGUF_REPO_ID` environment variable — | |
| no code changes needed. | |
| --- | |
| ## What's in this repo | |
| | File | Purpose | | |
| |-------------------|-----------------------------------------------------------------------| | |
| | `Dockerfile` | Multi-stage build: tries a prebuilt CPU wheel for `llama-cpp-python` first, falling back to compiling it in a builder stage; ships a slim runtime image with no compilers. | | |
| | `requirements.txt`| Pure-Python runtime deps (`gradio`, `huggingface_hub`). | | |
| | `app.py` | Downloads/caches the GGUF, loads it with `llama-cpp-python`, and serves the Gradio UI (progress bar + streaming chat). | | |
| | `README.md` | This file (also the Space's metadata card, via the YAML frontmatter above). | | |
| This repo is ready to push directly to a new **Docker** Space with no | |
| further edits. | |
| --- | |
| ## A note on the free Spaces tier | |
| This project is tuned to actually work on the **free CPU basic** tier | |
| end-to-end, which comes with real constraints worth knowing up front: | |
| - **No persistent disk by default.** Model weights re-download on every | |
| full restart/rebuild (see "Model caching" below) unless you pay for the | |
| Persistent Storage add-on. | |
| - **2 vCPUs, 16GB RAM.** `N_THREADS` defaults to `2` to match the actual | |
| core count rather than over-subscribing; `N_CTX` defaults to a modest | |
| `4096` tokens, well within what a ~1B `Q4_K_M` model needs on this | |
| hardware. | |
| - **Build minutes are shared/limited infrastructure**, so the Dockerfile | |
| tries a prebuilt wheel for `llama-cpp-python` before ever compiling from | |
| source (see "Technical notes" below) — this is the single biggest lever | |
| for keeping first-time build time low on the free tier. | |
| - If you outgrow any of this — a bigger model, longer context — the fixes | |
| are all opt-in: upgrade to a paid CPU/GPU hardware tier, or add | |
| Persistent Storage, and adjust `N_CTX`/`N_THREADS` accordingly. No code | |
| changes required, since everything is environment-variable driven. | |
| ## Deploying to Hugging Face Spaces | |
| 1. Create a new Space at <https://huggingface.co/new-space>. | |
| 2. Choose **Docker** as the Space SDK (not "Gradio" or "Streamlit" — this | |
| project builds and runs its own Dockerfile). | |
| 3. Pick the **CPU basic (free)** hardware tier — this project is built to | |
| run entirely on CPU. | |
| 4. Push these four files to the Space repo: | |
| ```bash | |
| git clone https://huggingface.co/spaces/<your-username>/<your-space-name> | |
| cd <your-space-name> | |
| cp /path/to/Dockerfile /path/to/requirements.txt /path/to/app.py /path/to/README.md . | |
| git add . | |
| git commit -m "Deploy GGUF chat Space" | |
| git push | |
| ``` | |
| 5. The Space will build the Docker image (typically well under a minute for | |
| `llama-cpp-python` if a prebuilt wheel is available — see "Technical | |
| notes" — otherwise 15-30+ minutes compiling from source) and then start | |
| the container. On first chat message, `app.py` downloads the selected | |
| GGUF file from the Hub and shows a live progress bar in the UI itself | |
| (in addition to the Space's **Logs** tab). | |
| 6. Once the model finishes loading, replies stream in token-by-token. | |
| No secrets or tokens are required for the default (public) repo. If you | |
| point this at a **gated/private** GGUF repo, add an `HF_TOKEN` secret in the | |
| Space's **Settings → Variables and secrets**; `huggingface_hub` picks it up | |
| automatically. | |
| --- | |
| ## Configuration (environment variables) | |
| All of these are set with sensible defaults in the `Dockerfile` and can be | |
| overridden per-Space under **Settings → Variables and secrets** without | |
| touching any code: | |
| | Variable | Default | Description | | |
| |-----------------------|-------------------------------------------------------|--------------| | |
| | `GGUF_REPO_ID` | `bartowski/google_gemma-3-2b-it-GGUF` | Hub repo to pull the GGUF from. | | |
| | `GGUF_FILENAME` | *(empty = auto-select)* | Force an exact filename instead of auto-selecting by quant. | | |
| | `PREFERRED_QUANT` | `Q4_K_M` | Preferred quantization. Falls back automatically (`Q4_K_S` → `Q5_K_M` → ... → smallest available `.gguf`) if not present. `mmproj` (vision) and `-MTP-` (speculative-decoding draft head) files are skipped by the auto-selector in favor of a plain text-chat quant. | | |
| | `MODEL_CACHE_DIR` | `/data/models` | Local cache directory for downloaded model weights. | | |
| | `HF_HOME` | `/data/hf_home` | Cache directory for Hub metadata. | | |
| | `N_CTX` | `4096` | Context window (tokens) allocated at load time. Raise this if you swap in a model that needs more headroom and you have the RAM to back it. | | |
| | `N_THREADS` | `2` | CPU threads for inference. Matches the free tier's 2 vCPUs by default; raise it if you upgrade hardware. | | |
| | `N_BATCH` | `256` | Prompt processing batch size. | | |
| | `MAX_NEW_TOKENS` | `899` | Max tokens generated per reply. | | |
| | `TEMPERATURE` | `0.7` | Sampling temperature. | | |
| | `TOP_P` | `0.9` | Nucleus sampling. | | |
| | `TOP_K` | `40` | Top-k sampling. | | |
| | `REPEAT_PENALTY` | `1.1` | Repetition penalty. | | |
| | `SYSTEM_PROMPT` | *(a short "be concise, adapt to the request" prompt — see `app.py`)* | System prompt prepended to every conversation. Override to customize the assistant's behavior. | | |
| | `DOWNLOAD_MAX_RETRIES` | `5` | Retry attempts (exponential backoff) for the model download. | | |
| --- | |
| ## Model caching & the free tier's storage caveat | |
| `app.py` downloads the model once into `MODEL_CACHE_DIR` and reuses the | |
| cached file for every subsequent chat request — it will **not** re-download | |
| on every message, and it survives the container going to sleep/waking back | |
| up from inactivity. | |
| However, **the free Spaces tier has no *persistent* storage**: the | |
| container's disk (including `/data`) is rebuilt from scratch whenever the | |
| Space is fully **restarted or rebuilt** (e.g. after a `git push`, a factory | |
| reboot, or an infrastructure migration). In that case, the model will be | |
| re-downloaded once on the next startup — this is a platform limitation, not | |
| a bug in this app. If you need the cache to survive restarts, enable | |
| **Persistent Storage** for the Space (a paid add-on) and point | |
| `MODEL_CACHE_DIR`/`HF_HOME` at the mounted persistent volume (typically | |
| `/data`, which is already the default here). | |
| --- | |
| ## Local development (outside Docker) | |
| ```bash | |
| # Build llama-cpp-python with a build appropriate for your machine: | |
| CMAKE_ARGS="-DGGML_NATIVE=ON" pip install llama-cpp-python | |
| pip install -r requirements.txt | |
| python app.py | |
| # then open http://localhost:7860 | |
| ``` | |
| ## Building/running the Docker image locally | |
| ```bash | |
| docker build -t gguf-chat-space . | |
| docker run -it -p 7860:7860 gguf-chat-space | |
| # then open http://localhost:7860 | |
| ``` | |
| --- | |
| ## Technical notes | |
| - **`llama-cpp-python` install strategy**: the builder stage first tries to | |
| fetch a **prebuilt CPU wheel** from the maintainer's custom index | |
| (`https://abetlen.github.io/llama-cpp-python/whl/cpu`) using | |
| `pip wheel --only-binary=:all:`, which fails fast (rather than silently | |
| falling back to a slow source build) if no matching wheel exists for the | |
| image's Python ABI/platform. This turns the install from a **15-30+ | |
| minute from-source compile into a roughly one-minute download** on the | |
| common case. | |
| Only if no prebuilt wheel is available does the Dockerfile fall back to | |
| building from source, with `CMAKE_ARGS="-DGGML_NATIVE=OFF -DGGML_AVX2=ON | |
| -DGGML_FMA=ON -DGGML_F16C=ON"`. `GGML_NATIVE` is deliberately disabled | |
| because the machine that *builds* the Docker image is not guaranteed to | |
| be the same CPU that *runs* it; auto-detected "native" builds can | |
| otherwise crash with `SIGILL` on the Space's actual runner. AVX2/FMA/F16C | |
| are supported by essentially all modern cloud x86_64 CPUs and give good | |
| performance without that risk. The compiler toolchain (`build-essential`, | |
| `cmake`, `ninja-build`, `git`) is only installed in this fallback branch, | |
| and the fallback build caps itself at 4 parallel compile jobs to avoid | |
| getting OOM-killed on the free Spaces builder, which can report more CPU | |
| cores than it has RAM to back a fully parallel C++ build. | |
| - **Chat template**: the `Llama` object is created without a hardcoded | |
| `chat_format`, so `llama-cpp-python` auto-detects and applies the Jinja | |
| chat template embedded in the GGUF's own metadata. This is what lets the | |
| same `app.py` work correctly across different model families (Gemma, | |
| Llama, Qwen, ...) if you change `GGUF_REPO_ID`, without hardcoding any | |
| model-specific prompt formatting. | |
| - **Progress bar UI**: the model downloads/loads lazily on the *first* chat | |
| request rather than at container startup, and `app.py` streams a live | |
| HTML progress bar (download %, then an animated layer-load indicator) | |
| into the response pane while that happens, in addition to logging | |
| progress to the Space's **Logs** tab. The layer-load bar is a smooth | |
| "still working" animation rather than an exact per-layer readout, since | |
| `llama-cpp-python`'s Python API doesn't expose real-time load progress. | |
| - **Streaming**: implemented via `llm.create_chat_completion(..., stream=True)`, | |
| yielding incrementally-growing text into the UI for token-by-token display. | |
| - **GPU layers**: `n_gpu_layers=0` — this Space is CPU-only by design, matching | |
| the free Spaces hardware tier. | |
| - **File selection**: uses `huggingface_hub.HfApi().model_info(..., files_metadata=True)` | |
| to inspect all files with sizes, filters out `mmproj` (vision projector) | |
| and `-MTP-` (speculative decoding draft-head) variants by default, then | |
| picks the smallest file matching `PREFERRED_QUANT`, falling back through | |
| a quant-quality-ordered list if needed. |