README.md

---
title: DeepSeek 7B GGUF API
emoji: 🧠
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
license: apache-2.0
---

# DeepSeek-LLM-7B-chat GGUF API (Docker Space)

An OpenAI-compatible FastAPI server that runs
**`TheBloke/deepseek-llm-7B-chat-GGUF` → `deepseek-llm-7b-chat.Q4_K_M.gguf`**
with `llama-cpp-python`, using DeepSeek's official chat template (registered in
`main.py` as chat format `"deepseek"`). The model file is **not** stored in this
repo — it is downloaded from the Hugging Face model hub on container startup,
so this code repo stays tiny.

> ⚠️ **The YAML block at the very top of this file is required.** Hugging Face
> reads `sdk: docker` and `app_port: 7860` from it to know how to build and
> route the Space. Do not delete it.

## Endpoints

Base URL: `https://electro0023-model.hf.space`

| Method | Path                   | Purpose                                       |
|--------|------------------------|-----------------------------------------------|
| GET    | `/health`              | Liveness + whether the model finished loading |
| GET    | `/v1/models`           | OpenAI-style model list                       |
| POST   | `/v1/chat/completions` | OpenAI-compatible chat completion             |
| POST   | `/generate`            | Simple `{prompt, max_tokens}` → text          |

## Quick test

```bash
curl https://electro0023-model.hf.space/health

curl -X POST https://electro0023-model.hf.space/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"deepseek-7b-chat","messages":[{"role":"user","content":"State Newton'\''s second law in one sentence."}],"max_tokens":64}'
```

## Swapping models (no code changes)

Set Space **Variables** (Settings → Variables and secrets) and restart:

- `REPO_ID` — HF model repo holding the GGUF
- `FILENAME` — the exact `.gguf` file inside that repo
- `MODEL_ID` — name reported to API clients
- `CHAT_FORMAT` — chat template (`deepseek` = default, or built-ins like
  `llama-3`, `chatml`); set to empty for BASE models (raw completion mode)
- `N_CTX`, `N_THREADS`, `N_GPU_LAYERS`, `DEFAULT_MAX_TOKENS` — tuning knobs

## Notes

- To run the BASE variant instead, set `REPO_ID=TheBloke/deepseek-llm-7B-base-GGUF`,
  `FILENAME=deepseek-llm-7b-base.Q4_K_M.gguf`, `CHAT_FORMAT=` (empty).
- On the free `cpu-basic` tier (2 vCPU, 16 GB RAM) a 7B Q4_K_M generates
  roughly 1–3 tokens/sec. Expect long response times; set generous client
  timeouts.
- First boot downloads ~4.1 GB, so allow several minutes before `/health`
  reports `status: ok`.