| | ---
|
| |
|
| | title: RWKV HF Space
|
| | emoji: 🐦⬛
|
| | colorFrom: purple
|
| | colorTo: pink
|
| | sdk: docker
|
| | pinned: false
|
| |
|
| | ---
|
| |
|
| | # Simple RWKV OpenAI-Compatible API
|
| |
|
| | ---
|
| |
|
| | title: RWKV HF Space
|
| | emoji: 🐦⬛
|
| | colorFrom: purple
|
| | colorTo: pink
|
| | sdk: docker
|
| | pinned: false
|
| |
|
| | ---
|
| |
|
| | # Simple RWKV OpenAI-Compatible API
|
| |
|
| | ## Quick Windows Setup (no Docker)
|
| |
|
| | This repository was originally packaged with a Dockerfile. It now provides a `setup_windows.ps1` script that mirrors Dockerfile actions and sets up the service locally on Windows (installs Python dependencies, builds the frontend, and downloads the 0.1B model).
|
| |
|
| | Prerequisites:
|
| | - Python 3.10+ installed and in PATH
|
| | - Node.js + npm (optional, required for building the frontend)
|
| | - (Optional) NVIDIA GPU and CUDA (for GPU runtime)
|
| |
|
| | To setup locally on Windows (CPU-only):
|
| |
|
| | ```powershell
|
| | .\setup_windows.ps1 -gpu:$false -buildFrontend:$true -CONFIG_FILE config.production.yaml
|
| | ```
|
| |
|
| | If you have a compatible NVIDIA GPU and prefer to install GPU-enabled dependencies, run with the `-gpu` switch.
|
| |
|
| | After setup, run the API:
|
| |
|
| | ```powershell
|
| | #$env:CONFIG_FILE='config.production.yaml'
|
| | python app.py
|
| | ```
|
| |
|
| | The default production config in `config.production.yaml` now contains a single model — the 0.1B model `rwkv7-g1a-0.1b-20250728-ctx4096` — set as default chat and reasoning model.
|
| |
|
| | To download models defined in any config:
|
| |
|
| | ```powershell
|
| | python download_models.py --config config.production.yaml
|
| | ```
|
| |
|
| | This will store the downloaded .pth files under the `DOWNLOAD_MODEL_DIR` specified in the YAML (defaults to `./models`).
|
| |
|
| | Advanced features:
|
| | - `reasoning` is performed in-process by the same model (no external reasoning model is used). Use a request model like `rwkv-latest:thinking` or set the reasoning suffix and the requested model will run reasoning in the same model.
|
| | - `web_search` functionality is available at the request level — set `web_search: true` and optionally `search_top_k` to inject search results from DuckDuckGo into the prompt. This is executed by the server and provided to the same model as context.
|
| | - `tools` are executed server-side and results injected into the prompt for the same model. Supported tools: `web_search` and `calc` (calculator). Example of `tools` usage:
|
| |
|
| | ```json
|
| | {
|
| | "model": "rwkv-latest",
|
| | "prompt": "Calculate 2+3*4 and tell me the result",
|
| | "tools": [{"name": "calc", "args": {"expression": "2+3*4"}}]
|
| | }
|
| | ```
|
| |
|
| | API endpoints and model listing:
|
| | - `GET /api/v1/models` — returns a JSON list of configured models, sampler defaults, and ALLOW_* flags. This lets clients build per-model UI toggles (web search, tools, reasoning) based on server-provided capabilities.
|
| |
|
| | Examples:
|
| | - `curl http://127.0.0.1:7860/api/v1/models` will show configured models and their sampler defaults.
|
| | Example: POST with `web_search` and reasoning enabled
|
| |
|
| | ```json
|
| | {
|
| | "model": "rwkv-latest:thinking",
|
| | "prompt": "Who is the current president of France?",
|
| | "max_tokens": 32,
|
| | "web_search": true,
|
| | "search_top_k": 3
|
| | }
|
| | ```
|
| |
|
| | The server will perform a web search for the prompt, aggregate the top 3 results, and inject those into the prompt, then run the model with reasoning enabled — all using the same model instead of an external reasoning or search model.
|
| |
|
| | Universal tool and model-initiated tool calls:
|
| | - The `universal` tool returns a structured JSON/dict with the following fields: `action` (calc/web_search), `result` (string), and `metadata` (dict with `confidence`, query/expression, etc.).
|
| | - Example `universal` result:
|
| |
|
| | ```json
|
| | {
|
| | "action": "calc",
|
| | "result": "14",
|
| | "metadata": {"expression": "2+3*4", "confidence": 0.98}
|
| | }
|
| | ```
|
| |
|
| | - The model can also request tools mid-generation by emitting a sentinel tag, e.g.:
|
| |
|
| | ```
|
| | <tool-call>{"name":"calc","args":{"expression":"40+2"}}</tool-call>
|
| | ```
|
| |
|
| | When the model emits such a sentinel, the server will execute the requested tool, inject the results into the prompt, and continue streaming output. The server will also emit a metadata-only streaming chunk so the client is aware a tool was executed mid-stream.
|
| |
|
| | Streaming behavior:
|
| | - The API streams responses token-by-token by default (`stream: true`) and persists a `state_name` for the generation if requested (or will generate one). Provide `state_name` to resume continuation from where the previous stream stopped. The server stores model state in memory under `(model, state_name)` so subsequent requests with the same `state_name` can continue generation from that exact point. |
