Update README.md

README.md

@@ -1,5 +1,5 @@
 ---
-title: Qwythos 9B GGUF Chat
+title: GGUF Chat
 emoji: 🧠
 colorFrom: indigo
 colorTo: purple
@@ -9,18 +9,18 @@ pinned: false
 license: apache-2.0
 ---
 
-# Qwythos-9B GGUF Chat (Docker Space)
+# GGUF Chat (Docker Space)
 
 A self-contained Hugging Face **Docker Space** that downloads a GGUF quant of
-[`empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF`](https://huggingface.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF)
-and serves it through a streaming Gradio chat UI, powered by
+[`bartowski/google_gemma-3-2b-it-GGUF`](https://huggingface.co/bartowski/google_gemma-3-2b-it-GGUF)
+and serves it through a Gradio chat UI — with a live download/load progress
+bar on first request, then token-by-token streaming — powered by
 [`llama-cpp-python`](https://github.com/abetlen/llama-cpp-python).
 
-> ⚠️ **About this model.** This is a third-party community fine-tune (based
-> on Qwen3.5-9B), not an Anthropic model. Its model card describes it as
-> "uncensored" with no built-in safety layer. If you deploy this Space
-> publicly, add your own moderation/review layer appropriate to your
-> audience — the model card recommends the same.
+The default model is a small (~2B parameter) instruct model chosen
+specifically to run comfortably on the **free CPU-basic Spaces tier**. Swap
+in a different repo any time via the `GGUF_REPO_ID` environment variable —
+no code changes needed.
 
 ---
 
@@ -28,9 +28,9 @@ and serves it through a streaming Gradio chat UI, powered by
 
 | File              | Purpose                                                              |
 |-------------------|-----------------------------------------------------------------------|
-| `Dockerfile`      | Multi-stage build: compiles `llama-cpp-python` in a builder stage, then ships a slim runtime image with no compilers. |
+| `Dockerfile`      | Multi-stage build: tries a prebuilt CPU wheel for `llama-cpp-python` first, falling back to compiling it in a builder stage; ships a slim runtime image with no compilers. |
 | `requirements.txt`| Pure-Python runtime deps (`gradio`, `huggingface_hub`).              |
-| `app.py`          | Downloads/caches the GGUF, loads it with `llama-cpp-python`, and serves a streaming Gradio chat UI. |
+| `app.py`          | Downloads/caches the GGUF, loads it with `llama-cpp-python`, and serves the Gradio UI (progress bar + streaming chat). |
 | `README.md`       | This file (also the Space's metadata card, via the YAML frontmatter above). |
 
 This repo is ready to push directly to a new **Docker** Space with no
@@ -46,17 +46,18 @@ end-to-end, which comes with real constraints worth knowing up front:
 - **No persistent disk by default.** Model weights re-download on every
   full restart/rebuild (see "Model caching" below) unless you pay for the
   Persistent Storage add-on.
-- **Limited RAM** (historically 16GB) caps both context window (`N_CTX`,
-  defaulted conservatively to `8192`) and how much of the model can be kept
-  resident — this is why `N_CTX` isn't set anywhere near the model's
-  advertised 1M-token ceiling by default.
+- **2 vCPUs, 16GB RAM.** `N_THREADS` defaults to `2` to match the actual
+  core count rather than over-subscribing; `N_CTX` defaults to a modest
+  `4096` tokens, well within what a ~1B `Q4_K_M` model needs on this
+  hardware.
 - **Build minutes are shared/limited infrastructure**, so the Dockerfile
   tries a prebuilt wheel for `llama-cpp-python` before ever compiling from
   source (see "Technical notes" below) — this is the single biggest lever
   for keeping first-time build time low on the free tier.
-- If you outgrow any of this, the fixes are all opt-in: upgrade to a paid
-  CPU/GPU hardware tier, or add Persistent Storage — no code changes
-  required, since everything is environment-variable driven.
+- If you outgrow any of this — a bigger model, longer context — the fixes
+  are all opt-in: upgrade to a paid CPU/GPU hardware tier, or add
+  Persistent Storage, and adjust `N_CTX`/`N_THREADS` accordingly. No code
+  changes required, since everything is environment-variable driven.
 
 ## Deploying to Hugging Face Spaces
 
@@ -72,16 +73,17 @@ end-to-end, which comes with real constraints worth knowing up front:
    cd <your-space-name>
    cp /path/to/Dockerfile /path/to/requirements.txt /path/to/app.py /path/to/README.md .
    git add .
-   git commit -m "Deploy Qwythos-9B GGUF chat Space"
+   git commit -m "Deploy GGUF chat Space"
    git push
    ```
 
-5. The Space will build the Docker image (this takes several minutes the
-   first time — `llama-cpp-python` is compiled from source) and then start
-   the container. On first launch, `app.py` downloads the selected GGUF
-   file from the Hub in the background; watch the Space's **Logs** tab for
-   download/load progress.
-6. Once the model finishes loading, the chat UI becomes responsive.
+5. The Space will build the Docker image (typically well under a minute for
+   `llama-cpp-python` if a prebuilt wheel is available — see "Technical
+   notes" — otherwise 15-30+ minutes compiling from source) and then start
+   the container. On first chat message, `app.py` downloads the selected
+   GGUF file from the Hub and shows a live progress bar in the UI itself
+   (in addition to the Space's **Logs** tab).
+6. Once the model finishes loading, replies stream in token-by-token.
 
 No secrets or tokens are required for the default (public) repo. If you
 point this at a **gated/private** GGUF repo, add an `HF_TOKEN` secret in the
@@ -98,20 +100,20 @@ touching any code:
 
 | Variable              | Default                                              | Description |
 |-----------------------|-------------------------------------------------------|--------------|
-| `GGUF_REPO_ID`         | `empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF`        | Hub repo to pull the GGUF from. |
+| `GGUF_REPO_ID`         | `bartowski/google_gemma-3-2b-it-GGUF`                 | Hub repo to pull the GGUF from. |
 | `GGUF_FILENAME`        | *(empty = auto-select)*                                | Force an exact filename instead of auto-selecting by quant. |
 | `PREFERRED_QUANT`      | `Q4_K_M`                                              | Preferred quantization. Falls back automatically (`Q4_K_S` → `Q5_K_M` → ... → smallest available `.gguf`) if not present. `mmproj` (vision) and `-MTP-` (speculative-decoding draft head) files are skipped by the auto-selector in favor of a plain text-chat quant. |
 | `MODEL_CACHE_DIR`      | `/data/models`                                        | Local cache directory for downloaded model weights. |
 | `HF_HOME`              | `/data/hf_home`                                       | Cache directory for Hub metadata. |
-| `N_CTX`                | `8192`                                                 | Context window (tokens) allocated at load time. The model supports up to **1,048,576** tokens via baked-in YaRN scaling, but a free 16GB-RAM CPU Space cannot allocate a multi-hundred-thousand-token KV-cache — raise this only if you've upgraded hardware (see "Long context" below). |
-| `N_THREADS`            | `0` (= auto, all cores)                                | CPU threads for inference. |
+| `N_CTX`                | `4096`                                                 | Context window (tokens) allocated at load time. Raise this if you swap in a model that needs more headroom and you have the RAM to back it. |
+| `N_THREADS`            | `2`                                                     | CPU threads for inference. Matches the free tier's 2 vCPUs by default; raise it if you upgrade hardware. |
 | `N_BATCH`              | `256`                                                  | Prompt processing batch size. |
-| `MAX_NEW_TOKENS`       | `1024`                                                 | Max tokens generated per reply. |
-| `TEMPERATURE`          | `0.6`                                                  | Sampling temperature (the model card recommends 0.6 for its thinking mode; avoid ≤0.3, which the card notes can cause repetition loops). |
-| `TOP_P`                | `0.95`                                                 | Nucleus sampling. |
-| `TOP_K`                | `20`                                                   | Top-k sampling. |
-| `REPEAT_PENALTY`       | `1.05`                                                 | Repetition penalty. |
-| `SYSTEM_PROMPT`        | *(empty)*                                              | Optional system prompt prepended to every conversation. |
+| `MAX_NEW_TOKENS`       | `899`                                                  | Max tokens generated per reply. |
+| `TEMPERATURE`          | `0.7`                                                  | Sampling temperature. |
+| `TOP_P`                | `0.9`                                                  | Nucleus sampling. |
+| `TOP_K`                | `40`                                                   | Top-k sampling. |
+| `REPEAT_PENALTY`       | `1.1`                                                  | Repetition penalty. |
+| `SYSTEM_PROMPT`        | *(a short "be concise, adapt to the request" prompt — see `app.py`)* | System prompt prepended to every conversation. Override to customize the assistant's behavior. |
 | `DOWNLOAD_MAX_RETRIES` | `5`                                                     | Retry attempts (exponential backoff) for the model download. |
 
 ---
@@ -135,27 +137,6 @@ a bug in this app. If you need the cache to survive restarts, enable
 
 ---
 
-## Long context ("1M context") notes
-
-The GGUF files in this repo ship with YaRN rope-scaling baked in for up to a
-1,048,576-token context window. That is a *ceiling*, not something you get
-for free on CPU:
-
-- Free **CPU basic** Spaces (16GB RAM) can realistically handle a `Q4_K_M`
-  9B model with a context window in the **low thousands to ~16-32k tokens**,
-  depending on available RAM after the model weights are loaded.
-- Attempting to set `N_CTX` far beyond what your Space's RAM allows will
-  cause the model load to fail or the container to be OOM-killed.
-- If you need genuinely long context (hundreds of thousands of tokens),
-  you'll need a GPU Space or a machine with substantially more RAM — the
-  model card itself notes the full 1M window typically needs multi-GPU or
-  aggressive KV-cache offload even outside of CPU constraints.
-
-`N_CTX` is fully configurable via the environment variable above so you can
-tune it to whatever hardware tier you're running on.
-
----
-
 ## Local development (outside Docker)
 
 ```bash
@@ -170,8 +151,8 @@ python app.py
 ## Building/running the Docker image locally
 
 ```bash
-docker build -t qwythos-space .
-docker run -it -p 7860:7860 qwythos-space
+docker build -t gguf-chat-space .
+docker run -it -p 7860:7860 gguf-chat-space
 # then open http://localhost:7860
 ```
 
@@ -202,16 +183,23 @@ docker run -it -p 7860:7860 qwythos-space
   cores than it has RAM to back a fully parallel C++ build.
 - **Chat template**: the `Llama` object is created without a hardcoded
   `chat_format`, so `llama-cpp-python` auto-detects and applies the Jinja
-  chat template embedded in the GGUF's own metadata — the current,
-  non-deprecated approach (no reliance on a manually-specified/legacy
-  template name).
+  chat template embedded in the GGUF's own metadata. This is what lets the
+  same `app.py` work correctly across different model families (Gemma,
+  Llama, Qwen, ...) if you change `GGUF_REPO_ID`, without hardcoding any
+  model-specific prompt formatting.
+- **Progress bar UI**: the model downloads/loads lazily on the *first* chat
+  request rather than at container startup, and `app.py` streams a live
+  HTML progress bar (download %, then an animated layer-load indicator)
+  into the response pane while that happens, in addition to logging
+  progress to the Space's **Logs** tab. The layer-load bar is a smooth
+  "still working" animation rather than an exact per-layer readout, since
+  `llama-cpp-python`'s Python API doesn't expose real-time load progress.
 - **Streaming**: implemented via `llm.create_chat_completion(..., stream=True)`,
-  yielding incrementally-growing text to Gradio's `ChatInterface` for
-  token-by-token display.
+  yielding incrementally-growing text into the UI for token-by-token display.
 - **GPU layers**: `n_gpu_layers=0` — this Space is CPU-only by design, matching
   the free Spaces hardware tier.
 - **File selection**: uses `huggingface_hub.HfApi().model_info(..., files_metadata=True)`
   to inspect all files with sizes, filters out `mmproj` (vision projector)
   and `-MTP-` (speculative decoding draft-head) variants by default, then
   picks the smallest file matching `PREFERRED_QUANT`, falling back through
-  a quant-quality-ordered list if needed.
+  a quant-quality-ordered list if needed.