THOX Rust Coder: Qwen3-Coder-REAP-25B-A3B-Rust GGUF via conda-forge llama.cpp

Dockerfile

@@ -0,0 +1,33 @@
+# THOX Rust Coder — llama.cpp via the conda-forge PREBUILT binary.
+#
+# Do not try to compile llama-cpp-python on an HF Space. It does not fit the
+# build timeout -- not in the Spaces pip step, and not in a Docker builder stage
+# either, even at -j16 with tests/examples/server/tools disabled. Docker buys a
+# glibc-linked extension, not build time. conda-forge ships a prebuilt linux-64
+# glibc binary: correct ABI, no compile, ~2 minute build.
+FROM mambaorg/micromamba:2.9-debian12
+
+USER root
+RUN mkdir -p /app && chown 1000:1000 /app
+USER $MAMBA_USER
+
+RUN micromamba install -y -n base -c conda-forge \
+        python=3.11 \
+        pip \
+        llama-cpp-python=0.3.34 \
+    && micromamba clean --all --yes
+
+ARG MAMBA_DOCKERFILE_ACTIVATE=1
+
+COPY --chown=$MAMBA_USER:$MAMBA_USER requirements.txt /tmp/requirements.txt
+RUN pip install --no-cache-dir -r /tmp/requirements.txt
+
+ENV HF_HOME=/home/mambauser/.cache/huggingface \
+    GRADIO_SERVER_NAME=0.0.0.0 \
+    GRADIO_SERVER_PORT=7860
+
+WORKDIR /app
+COPY --chown=$MAMBA_USER:$MAMBA_USER app.py /app/app.py
+
+EXPOSE 7860
+CMD ["python", "app.py"]

README.md

@@ -1,10 +1,53 @@
 ---
-title: ThoxRustCoder
-emoji: 🌍
-colorFrom: blue
-colorTo: blue
+title: THOX Rust Coder
+emoji: 🦀
+colorFrom: gray
+colorTo: red
 sdk: docker
+app_port: 7860
 pinned: false
+license: apache-2.0
+models:
+  - Em-80/Qwen3-coder-REAP-25B-A3B-Rust-GGUF
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# THOX Rust Coder — Qwen3-Coder-REAP-25B-A3B-Rust
+
+Hosted Rust coding model, served as GGUF through llama.cpp.
+
+```bash
+curl -s -X POST https://thox-ai-thoxrustcoder.hf.space/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  -d '{"messages":[{"role":"user","content":"Write a Rust fn to parse semver."}],
+       "max_tokens":256}'
+```
+
+`GET /healthz` · `POST /v1/chat/completions` (OpenAI-shaped, ThoxRoute-registerable)
+
+## "3B active" is not a memory budget
+
+This is MoE: **25B total, ~3B active per token.** The two numbers govern
+different resources, and conflating them leads to picking a tier that cannot
+load the model:
+
+| | sized by |
+|---|---|
+| **memory** | **25B total** — every expert stays resident, since the router may pick any of them on any token |
+| **compute** | **~3B active** — decode costs about what a 3B dense model costs |
+
+Q4_K_M is therefore **15.1 GB**, not "3B-worth". What MoE buys is speed per
+resident byte — which is precisely what makes a CPU tier viable: 25B-sized
+memory, 3B-sized arithmetic.
+
+Runs on `cpu-upgrade` (8 vCPU / 32 GB). Throughput is measured and reported in
+every response under `thox_perf`, and in the UI — no throughput claim appears in
+this README that was not taken from this Space.
+
+## License — verified, not assumed
+
+| repo | license | checked |
+|---|---|---|
+| `Em-80/Qwen3-coder-REAP-25B-A3B-Rust-GGUF` | **Apache-2.0** | ungated, public |
+
+Apache-2.0 permits hosting, commercial use and redistribution. Verified against
+the model repo's own metadata before this Space was published.

app.py

@@ -0,0 +1,193 @@
+"""THOX Rust Coder — hosted Qwen3-Coder-REAP-25B-A3B-Rust (GGUF).
+
+SIZING: "3B active" IS NOT A MEMORY BUDGET
+------------------------------------------
+This is a Mixture-of-Experts model: 25B total parameters, ~3B active per token.
+Those two numbers govern different resources and it is easy to conflate them:
+
+    active params (~3B)  -> COMPUTE per token.  Decode is as cheap as a 3B dense.
+    total params (25B)   -> MEMORY.  Every expert must be resident, because the
+                            router may select any of them on any token.
+
+So a Q4_K_M GGUF is **15.1 GB of RAM/VRAM**, not "3B-worth". You cannot fit this
+on a tier sized for a 3B model. What MoE buys you here is speed-per-byte, not a
+smaller footprint -- which is exactly why a CPU tier is viable at all: we pay
+25B-sized memory but only 3B-sized arithmetic.
+
+TIER
+----
+Starts on `cpu-upgrade` (8 vCPU / 32 GB, ~$0.03/hr). That fits Q4_K_M with room
+for the KV cache, and the 3B active path keeps CPU decode tolerable. If measured
+throughput is too slow to be a useful coding assistant, escalate to a GPU tier --
+but escalate on a MEASUREMENT, not on the assumption that 25B implies a GPU.
+"""
+
+from __future__ import annotations
+
+import os
+import time
+import uuid
+
+import gradio as gr
+from fastapi import FastAPI
+from huggingface_hub import hf_hub_download
+from pydantic import BaseModel
+
+MODEL_REPO = os.environ.get("THOX_MODEL_REPO", "Em-80/Qwen3-coder-REAP-25B-A3B-Rust-GGUF")
+MODEL_FILE = os.environ.get("THOX_MODEL_FILE", "Qwen3-Coder-REAP-25B-A3B-Rust-Q4_K_M.gguf")
+N_CTX = int(os.environ.get("THOX_N_CTX", "8192"))
+
+SYSTEM = (
+    "You are THOX Rust Coder. You write correct, idiomatic Rust. Prefer showing "
+    "compiling code over prose. If a request is ambiguous, state the assumption "
+    "you made in one line, then give the code."
+)
+
+_llm = None
+
+
+def _usable_cpus() -> int:
+    """Threads from the cgroup quota, not the host.
+
+    `os.cpu_count()` reports the HOST's core count inside a container. On a
+    sibling Space this oversubscribed a 2-vCPU cgroup ~8x and cost ~290x
+    throughput -- the model ran slower than the edge device it was meant to
+    offload. Read the quota.
+    """
+    try:
+        quota, period = open("/sys/fs/cgroup/cpu.max").read().split()
+        if quota != "max":
+            return max(1, int(int(quota) / int(period)))
+    except Exception:
+        pass
+    try:
+        q = int(open("/sys/fs/cgroup/cpu/cpu.cfs_quota_us").read())
+        p = int(open("/sys/fs/cgroup/cpu/cpu.cfs_period_us").read())
+        if q > 0:
+            return max(1, q // p)
+    except Exception:
+        pass
+    try:
+        return max(1, len(os.sched_getaffinity(0)))
+    except Exception:
+        return os.cpu_count() or 2
+
+
+def llm():
+    global _llm
+    if _llm is None:
+        from llama_cpp import Llama
+
+        path = hf_hub_download(MODEL_REPO, MODEL_FILE,
+                               token=os.environ.get("HF_TOKEN"))
+        _llm = Llama(
+            model_path=path,
+            n_ctx=N_CTX,
+            n_threads=_usable_cpus(),
+            # -1 offloads every layer when a GPU is present, and is simply
+            # ignored on a CPU build -- so the same image works on both tiers.
+            n_gpu_layers=int(os.environ.get("THOX_GPU_LAYERS", "-1")),
+            verbose=False,
+        )
+    return _llm
+
+
+def generate(messages, max_tokens=512, temperature=0.2):
+    t0 = time.time()
+    # create_chat_completion uses the chat template embedded in the GGUF, rather
+    # than a hand-rolled one. Qwen3-Coder is ChatML, but reading it from the file
+    # means a re-quant with a different template does not silently break output.
+    out = llm().create_chat_completion(
+        messages=messages, max_tokens=max_tokens, temperature=temperature,
+    )
+    dt = time.time() - t0
+    text = out["choices"][0]["message"]["content"]
+    n = out.get("usage", {}).get("completion_tokens") or 0
+    return text, n, dt
+
+
+api = FastAPI()
+
+
+class Msg(BaseModel):
+    role: str
+    content: str
+
+
+class ChatRequest(BaseModel):
+    model: str | None = None
+    messages: list[Msg]
+    max_tokens: int | None = 512
+    temperature: float | None = 0.2
+
+
+@api.get("/healthz")
+def healthz():
+    return {
+        "status": "ok",
+        "model": MODEL_REPO,
+        "file": MODEL_FILE,
+        "role": "thox-rust-coder",
+        "n_ctx": N_CTX,
+        "threads": _usable_cpus(),
+        "loaded": _llm is not None,
+    }
+
+
+@api.post("/v1/chat/completions")
+def chat_completions(req: ChatRequest):
+    msgs = [m.model_dump() for m in req.messages]
+    if not any(m["role"] == "system" for m in msgs):
+        msgs = [{"role": "system", "content": SYSTEM}] + msgs
+    text, n, dt = generate(msgs, req.max_tokens or 512, req.temperature or 0.2)
+    return {
+        "id": "chatcmpl-" + uuid.uuid4().hex[:12],
+        "object": "chat.completion",
+        "created": int(time.time()),
+        "model": "thox-rust-coder",
+        "choices": [{"index": 0, "finish_reason": "stop",
+                     "message": {"role": "assistant", "content": text}}],
+        "usage": {"completion_tokens": n},
+        "thox_perf": {"tok_per_s": round(n / dt, 1) if dt else None,
+                      "seconds": round(dt, 2)},
+    }
+
+
+def ui(prompt, max_tokens):
+    text, n, dt = generate(
+        [{"role": "system", "content": SYSTEM}, {"role": "user", "content": prompt}],
+        int(max_tokens),
+    )
+    tps = n / dt if dt else 0
+    return text, f"{n} tok in {dt:.1f}s = {tps:.1f} tok/s"
+
+
+demo = gr.Interface(
+    fn=ui,
+    inputs=[gr.Textbox(label="Prompt", lines=4,
+                       value="Write a Rust function that parses a semver string "
+                             "into (major, minor, patch), returning Result."),
+            gr.Slider(64, 2048, value=512, step=64, label="max tokens")],
+    outputs=[gr.Code(label="THOX Rust Coder", language="rust"),
+             gr.Textbox(label="Measured")],
+    title="THOX Rust Coder — Qwen3-Coder-REAP-25B-A3B-Rust",
+    description=(
+        "Rust-specialised MoE coder, Apache-2.0, served as GGUF.\n\n"
+        "`POST /v1/chat/completions` (OpenAI-shaped, ThoxRoute-registerable) · "
+        "`GET /healthz`\n\n"
+        "**25B total / ~3B active.** Memory is sized by the 25B (all experts stay "
+        "resident); speed is sized by the 3B. That combination is what makes a "
+        "CPU tier viable."
+    ),
+)
+
+app = gr.mount_gradio_app(api, demo, path="/")
+
+
+if __name__ == "__main__":
+    # Defining `app` does not serve it -- without this the process exits 0 and
+    # the Space reports RUNTIME_ERROR with no traceback to read.
+    import uvicorn
+
+    uvicorn.run(app, host="0.0.0.0",
+                port=int(os.environ.get("GRADIO_SERVER_PORT", 7860)))

requirements.txt

@@ -0,0 +1,14 @@
+# llama-cpp-python is deliberately ABSENT.
+#
+# It is installed from conda-forge in the Dockerfile as a prebuilt, glibc-linked
+# linux-64 binary. Two pip routes both fail on an HF Space:
+#   * PyPI is sdist-only        -> compiles -> `Job timeout`
+#   * abetlen's CPU wheel index -> musl-linked -> builds green, RUNNING, /healthz
+#                                  ok, then HTTP 500 at the first dlopen
+# See thoxllm-factory/docs/SPACE_PINS.md.
+
+gradio==5.50.0
+fastapi
+pydantic>=2
+uvicorn
+huggingface_hub>=0.28