Replace Gradio SmolVLM app with DeepSeek-7B-base GGUF OpenAI-compatible API (Docker)

README.md

@@ -9,13 +9,14 @@ pinned: false
 license: apache-2.0
 ---
 
-# DeepSeek-LLM-7B-base GGUF API (Docker Space)
+# DeepSeek-LLM-7B-chat GGUF API (Docker Space)
 
 An OpenAI-compatible FastAPI server that runs
-**`TheBloke/deepseek-llm-7B-base-GGUF` → `deepseek-llm-7b-base.Q4_K_M.gguf`**
-with `llama-cpp-python`. The model file is **not** stored in this repo — it is
-downloaded from the Hugging Face model hub on container startup, so this code
-repo stays tiny.
+**`TheBloke/deepseek-llm-7B-chat-GGUF` → `deepseek-llm-7b-chat.Q4_K_M.gguf`**
+with `llama-cpp-python`, using DeepSeek's official chat template (registered in
+`main.py` as chat format `"deepseek"`). The model file is **not** stored in this
+repo — it is downloaded from the Hugging Face model hub on container startup,
+so this code repo stays tiny.
 
 > ⚠️ **The YAML block at the very top of this file is required.** Hugging Face
 > reads `sdk: docker` and `app_port: 7860` from it to know how to build and
@@ -39,7 +40,7 @@ curl https://electro0023-model.hf.space/health
 
 curl -X POST https://electro0023-model.hf.space/v1/chat/completions \
   -H "Content-Type: application/json" \
-  -d '{"model":"deepseek-7b-base","messages":[{"role":"user","content":"Newton'\''s second law is"}],"max_tokens":64}'
+  -d '{"model":"deepseek-7b-chat","messages":[{"role":"user","content":"State Newton'\''s second law in one sentence."}],"max_tokens":64}'
 ```
 
 ## Swapping models (no code changes)
@@ -49,16 +50,14 @@ Set Space **Variables** (Settings → Variables and secrets) and restart:
 - `REPO_ID` — HF model repo holding the GGUF
 - `FILENAME` — the exact `.gguf` file inside that repo
 - `MODEL_ID` — name reported to API clients
-- `CHAT_FORMAT` — chat template for instruct models (e.g. `llama-3`, `chatml`);
-  leave empty for base models
+- `CHAT_FORMAT` — chat template (`deepseek` = default, or built-ins like
+  `llama-3`, `chatml`); set to empty for BASE models (raw completion mode)
 - `N_CTX`, `N_THREADS`, `N_GPU_LAYERS`, `DEFAULT_MAX_TOKENS` — tuning knobs
 
 ## Notes
 
-- **This default model is a BASE model** (raw text completion, not
-  instruction-tuned). For chat/instruction behaviour, switch to
-  `TheBloke/deepseek-llm-7B-chat-GGUF` / `deepseek-llm-7b-chat.Q4_K_M.gguf`
-  with `CHAT_FORMAT=deepseek` — or any other instruct GGUF.
+- To run the BASE variant instead, set `REPO_ID=TheBloke/deepseek-llm-7B-base-GGUF`,
+  `FILENAME=deepseek-llm-7b-base.Q4_K_M.gguf`, `CHAT_FORMAT=` (empty).
 - On the free `cpu-basic` tier (2 vCPU, 16 GB RAM) a 7B Q4_K_M generates
   roughly 1–3 tokens/sec. Expect long response times; set generous client
   timeouts.

main.py

@@ -34,28 +34,47 @@ from fastapi import FastAPI, HTTPException
 from pydantic import BaseModel, Field
 from huggingface_hub import hf_hub_download
 from llama_cpp import Llama
+from llama_cpp import llama_chat_format
 
 logging.basicConfig(level=logging.INFO, format="%(asctime)s  %(levelname)s  %(message)s")
 log = logging.getLogger("gguf-api")
 
+# DeepSeek-LLM chat template — NOT built into llama-cpp-python (and TheBloke's
+# 2023 GGUFs predate embedded chat_template metadata), so register it ourselves.
+# Official format:  [system\n\n]User: {msg}\n\nAssistant: {reply}<eos>...Assistant:
+@llama_chat_format.register_chat_format("deepseek")
+def _format_deepseek(messages, **kwargs) -> llama_chat_format.ChatFormatterResponse:
+    prompt = ""
+    for m in messages:
+        role, content = m["role"], (m["content"] or "").strip()
+        if role == "system":
+            prompt += content + "\n\n"
+        elif role == "user":
+            prompt += f"User: {content}\n\n"
+        elif role == "assistant":
+            prompt += f"Assistant: {content}<｜end▁of▁sentence｜>"
+    prompt += "Assistant:"
+    return llama_chat_format.ChatFormatterResponse(prompt=prompt, stop=["User:"])
+
+
 # ── Configuration (all via environment) ───────────────────────────────────────
-# Defaults point at DeepSeek-LLM-7B-base (Q4_K_M), a PUBLIC GGUF — so the Space
+# Defaults point at DeepSeek-LLM-7B-CHAT (Q4_K_M), a PUBLIC GGUF — so the Space
 # runs with zero setup. Override REPO_ID/FILENAME in the Space settings to swap
 # models without touching the code.
-REPO_ID            = os.getenv("REPO_ID", "TheBloke/deepseek-llm-7B-base-GGUF").strip()
-FILENAME           = os.getenv("FILENAME", "deepseek-llm-7b-base.Q4_K_M.gguf").strip()
+REPO_ID            = os.getenv("REPO_ID", "TheBloke/deepseek-llm-7B-chat-GGUF").strip()
+FILENAME           = os.getenv("FILENAME", "deepseek-llm-7b-chat.Q4_K_M.gguf").strip()
 HF_TOKEN           = os.getenv("HF_TOKEN") or None          # None => anonymous (public repo)
-MODEL_ID           = os.getenv("MODEL_ID", "") or "deepseek-7b-base"  # name reported to clients
+MODEL_ID           = os.getenv("MODEL_ID", "") or "deepseek-7b-chat"  # name reported to clients
 N_CTX              = int(os.getenv("N_CTX", "4096"))
 # Default to the number of *usable* CPUs (respects cgroup cpuset), NOT os.cpu_count()
 # which over-reports the host's cores inside a container and causes thread thrash.
 _USABLE_CPUS = len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else (os.cpu_count() or 4)
 N_THREADS          = int(os.getenv("N_THREADS", str(_USABLE_CPUS)))
 N_GPU_LAYERS       = int(os.getenv("N_GPU_LAYERS", "0"))    # 0 = CPU only; -1 = all layers on GPU
-# deepseek-llm-7b-BASE has no chat template (it is a raw completion model), so no
-# chat format is forced by default; llama-cpp falls back to its generic template.
-# If you swap to an instruct/chat GGUF, set CHAT_FORMAT accordingly (e.g. "llama-3").
-CHAT_FORMAT        = os.getenv("CHAT_FORMAT", "").strip()
+# "deepseek" = the custom template registered above, matching the default chat
+# model. Set to "" (empty) when running a BASE model — the endpoints then bypass
+# chat templating and use raw text completion instead.
+CHAT_FORMAT        = os.getenv("CHAT_FORMAT", "deepseek").strip()
 DEFAULT_MAX_TOKENS = int(os.getenv("DEFAULT_MAX_TOKENS", "512"))
 
 # The loaded model lives here. It is None until startup finishes (or if it failed).