README.md

---
title: Qwen2.5 Coder 32B AWQ OpenAI API
emoji: 🚀
colorFrom: green
colorTo: yellow
sdk: gradio
sdk_version: 6.22.0
python_version: '3.12'
app_file: app.py
pinned: false
---

# Qwen2.5-Coder-32B AWQ — OpenAI-compatible ZeroGPU API

This Space serves `Qwen/Qwen2.5-Coder-32B-Instruct-AWQ` through an
OpenAI-compatible Chat Completions endpoint on Hugging Face ZeroGPU.

The runtime is pinned to the versions that were validated during the Space
startup work:

- PyTorch 2.11.0 / CUDA 13.0 wheels
- torchvision 0.26.0
- Transformers 5.14.1
- GPTQModel 7.3.2
- Gradio 6.22.0

The AWQ model is intentionally loaded lazily from inside the `@spaces.GPU`
function. This is required by this deployment because AWQ/Marlin performs real
CUDA work while `from_pretrained()` is running. Do not move model loading back
to module startup without retesting the Space on ZeroGPU.

## API

- `GET /health`
- `GET /v1/models`
- `POST /v1/chat/completions`
- `GET /web-search?q=...`

Accepted model names are the real model ID plus the compatibility aliases
`qwen2.5-coder-32b` and `qwen-coder`. Aliases for unrelated Qwen3 or 14B
weights are deliberately not accepted.

Example client configuration:

```text
OPENAI_BASE_URL=https://erinaldorodrigues-qwen-coder-api.hf.space/v1
OPENAI_API_BASE=https://erinaldorodrigues-qwen-coder-api.hf.space/v1
OPENAI_MODEL=qwen2.5-coder-32b
WEB_SEARCH_PROVIDER=custom
WEB_SEARCH_API=https://erinaldorodrigues-qwen-coder-api.hf.space/web-search
WEB_METHOD=GET
WEB_QUERY_PARAM=q
```

For direct calls to a ZeroGPU `hf.space` URL, use a valid Hugging Face access
token as `OPENAI_API_KEY` (for example, set `HF_TOKEN=hf_...` locally and then
`OPENAI_API_KEY=$HF_TOKEN`). OpenAI-compatible clients send this value as
`Authorization: Bearer ...`; the Hugging Face proxy can then attribute ZeroGPU
usage to the caller instead of the much smaller anonymous pool. The current
`app.py` does not perform application-level Bearer-token validation itself.
Never commit or paste the token into this repository.

## Tool calling

The backend accepts OpenAI-style `tools`, `tool_choice`, and
`parallel_tool_calls`. Tool definitions are normalized for Qwen's native chat template and textual
`<tool_call>...</tool_call>` outputs (plus the observed `<function_call>` fallback) are translated back to OpenAI
`message.tool_calls` objects with stable IDs, JSON-string arguments and
`finish_reason="tool_calls"`. The system prompt does not duplicate tool schemas;
the official Qwen native `<tools>` catalog remains the single schema source of
truth.

OpenClaude compatibility preserves request-level OpenAI `tool_choice` semantics.
`tool_choice="required"` is never silently downgraded to `none`; `auto` keeps the
tool catalog visible unless the current user turn explicitly disables tools.
Repository/codebase inspection is treated as real work; when OpenClaude exposes
`Glob`, repository-overview requests deterministically narrow the first call to
`Glob` so the model cannot answer from memory without inspecting the worktree.
Flow-state instructions may recommend final synthesis after usable evidence, but
they do not rewrite an explicit `required` request. Required/forced tool calls are
generated greedily at temperature 0.0, and a required call that fails to parse is
returned as an explicit upstream error rather than a false plain-text success.

When `parallel_tool_calls=true`, the prompt permits multiple independent tool
calls. Otherwise generation stops after the first complete tool call.

The Space generates tool calls; the calling client remains responsible for
executing client-side tools and returning their results in subsequent `tool`
messages. `/web-search` is a separate server-side search endpoint.

## Generation and compatibility

The default context limit is 131,072 tokens and default maximum output is 2,048
tokens. Contexts above Qwen2.5's native 32,768-token window automatically enable
YaRN RoPE scaling with a proportional factor (4.0 at the 131,072-token default).
`MAX_CONTEXT_TOKENS` is deliberately capped at 131,072 because that is the maximum
long-context length documented for this Qwen2.5-Coder checkpoint; the backend
refuses larger values instead of falsely advertising a context it cannot process.

At context lengths above 32,768 tokens, the ZeroGPU decorator requests
`size="xlarge"` (96 GB VRAM) rather than the 48 GB default `large` allocation.
This consumes 2x ZeroGPU quota and may queue longer, but avoids advertising a
131K context while allocating hardware that is too tight for the 32B model plus
its long-context KV cache.

The default request temperature is `0.0` (greedy). A caller may request a higher
temperature up to `MAX_TEMPERATURE`, but required/forced tool-call generations
always use `0.0` to make protocol JSON deterministic. Responses report prompt
and completion token counts, and `finish_reason="length"` is returned when a
normal completion exhausts the configured output budget.

`stream=true` returns OpenAI-style SSE framing. Tool deltas contain the `index`,
`id`, function name, and JSON-string arguments expected by OpenClaude. When the
client sends `stream_options={"include_usage": true}`, the final usage-only SSE
chunk is emitted before `[DONE]`. The current implementation finishes model
generation before emitting the first content/tool delta, so it is protocol
streaming rather than token-by-token low-latency streaming.

## Health

`/health` reports the configured model, `model_loaded`, `context_length`,
`yarn_enabled`, and `yarn_factor`. Because model loading is lazy, a healthy freshly
started process can report `model_loaded=false` until the first GPU inference
initializes the AWQ model.

## Tests

Run:

```bash
python -m unittest discover -p 'test_*.py'
```

The suite covers generation helpers, OpenAI/OpenClaude tool flow, the full
OpenClaude tool-call/result round trip, tool-call parsing, and web-search
fallbacks. The patched tree currently contains 107 deterministic unit/contract
tests, including direct `/v1/chat/completions` and SSE contract tests with the
GPU/model loader stubbed out.