Spaces:
Running
Running
File size: 2,596 Bytes
0b154ad 300bbfa 0b154ad 300bbfa 0b154ad 4621b74 300bbfa 4621b74 300bbfa 4621b74 300bbfa 4621b74 300bbfa 4621b74 300bbfa | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | ---
title: DeepSeek 7B GGUF API
emoji: 🧠
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
license: apache-2.0
---
# DeepSeek-LLM-7B-chat GGUF API (Docker Space)
An OpenAI-compatible FastAPI server that runs
**`TheBloke/deepseek-llm-7B-chat-GGUF` → `deepseek-llm-7b-chat.Q4_K_M.gguf`**
with `llama-cpp-python`, using DeepSeek's official chat template (registered in
`main.py` as chat format `"deepseek"`). The model file is **not** stored in this
repo — it is downloaded from the Hugging Face model hub on container startup,
so this code repo stays tiny.
> ⚠️ **The YAML block at the very top of this file is required.** Hugging Face
> reads `sdk: docker` and `app_port: 7860` from it to know how to build and
> route the Space. Do not delete it.
## Endpoints
Base URL: `https://electro0023-model.hf.space`
| Method | Path | Purpose |
|--------|------------------------|-----------------------------------------------|
| GET | `/health` | Liveness + whether the model finished loading |
| GET | `/v1/models` | OpenAI-style model list |
| POST | `/v1/chat/completions` | OpenAI-compatible chat completion |
| POST | `/generate` | Simple `{prompt, max_tokens}` → text |
## Quick test
```bash
curl https://electro0023-model.hf.space/health
curl -X POST https://electro0023-model.hf.space/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"deepseek-7b-chat","messages":[{"role":"user","content":"State Newton'\''s second law in one sentence."}],"max_tokens":64}'
```
## Swapping models (no code changes)
Set Space **Variables** (Settings → Variables and secrets) and restart:
- `REPO_ID` — HF model repo holding the GGUF
- `FILENAME` — the exact `.gguf` file inside that repo
- `MODEL_ID` — name reported to API clients
- `CHAT_FORMAT` — chat template (`deepseek` = default, or built-ins like
`llama-3`, `chatml`); set to empty for BASE models (raw completion mode)
- `N_CTX`, `N_THREADS`, `N_GPU_LAYERS`, `DEFAULT_MAX_TOKENS` — tuning knobs
## Notes
- To run the BASE variant instead, set `REPO_ID=TheBloke/deepseek-llm-7B-base-GGUF`,
`FILENAME=deepseek-llm-7b-base.Q4_K_M.gguf`, `CHAT_FORMAT=` (empty).
- On the free `cpu-basic` tier (2 vCPU, 16 GB RAM) a 7B Q4_K_M generates
roughly 1–3 tokens/sec. Expect long response times; set generous client
timeouts.
- First boot downloads ~4.1 GB, so allow several minutes before `/health`
reports `status: ok`.
|