README.md

metadata

title: DeepSeek 7B GGUF API
emoji: 🧠
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
license: apache-2.0

DeepSeek-LLM-7B-chat GGUF API (Docker Space)

An OpenAI-compatible FastAPI server that runs TheBloke/deepseek-llm-7B-chat-GGUF → deepseek-llm-7b-chat.Q4_K_M.gguf with llama-cpp-python, using DeepSeek's official chat template (registered in main.py as chat format "deepseek"). The model file is not stored in this repo — it is downloaded from the Hugging Face model hub on container startup, so this code repo stays tiny.

⚠️ The YAML block at the very top of this file is required. Hugging Face reads sdk: docker and app_port: 7860 from it to know how to build and route the Space. Do not delete it.

Endpoints

Base URL: https://electro0023-model.hf.space

Method	Path	Purpose
GET	/health	Liveness + whether the model finished loading
GET	/v1/models	OpenAI-style model list
POST	/v1/chat/completions	OpenAI-compatible chat completion
POST	/generate	Simple {prompt, max_tokens} → text

Quick test

curl https://electro0023-model.hf.space/health

curl -X POST https://electro0023-model.hf.space/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"deepseek-7b-chat","messages":[{"role":"user","content":"State Newton'\''s second law in one sentence."}],"max_tokens":64}'

Swapping models (no code changes)

Set Space Variables (Settings → Variables and secrets) and restart:

• REPO_ID — HF model repo holding the GGUF
• FILENAME — the exact .gguf file inside that repo
• MODEL_ID — name reported to API clients
• CHAT_FORMAT — chat template (deepseek = default, or built-ins like llama-3, chatml); set to empty for BASE models (raw completion mode)
• N_CTX, N_THREADS, N_GPU_LAYERS, DEFAULT_MAX_TOKENS — tuning knobs

Notes

• To run the BASE variant instead, set REPO_ID=TheBloke/deepseek-llm-7B-base-GGUF, FILENAME=deepseek-llm-7b-base.Q4_K_M.gguf, CHAT_FORMAT= (empty).
• On the free cpu-basic tier (2 vCPU, 16 GB RAM) a 7B Q4_K_M generates roughly 1–3 tokens/sec. Expect long response times; set generous client timeouts.
• First boot downloads ~4.1 GB, so allow several minutes before /health reports status: ok.