README.md

metadata

title: Qwen2.5 Coder 32B AWQ OpenAI API
emoji: 🚀
colorFrom: green
colorTo: yellow
sdk: gradio
sdk_version: 6.22.0
python_version: '3.12'
app_file: app.py
pinned: false

Qwen2.5-Coder-32B AWQ — OpenAI-compatible ZeroGPU API

This Space serves Qwen/Qwen2.5-Coder-32B-Instruct-AWQ through an OpenAI-compatible Chat Completions endpoint on Hugging Face ZeroGPU.

The runtime is pinned to the versions that were validated during the Space startup work:

• PyTorch 2.11.0 / CUDA 13.0 wheels
• torchvision 0.26.0
• Transformers 5.14.1
• GPTQModel 7.3.2
• Gradio 6.22.0

The AWQ model is intentionally loaded lazily from inside the @spaces.GPU function. This is required by this deployment because AWQ/Marlin performs real CUDA work while from_pretrained() is running. Do not move model loading back to module startup without retesting the Space on ZeroGPU.

API

• GET /health
• GET /v1/models
• POST /v1/chat/completions
• GET /web-search?q=...

Accepted model names are the real model ID plus the compatibility aliases qwen2.5-coder-32b and qwen-coder. Aliases for unrelated Qwen3 or 14B weights are deliberately not accepted.

Example client configuration:

OPENAI_BASE_URL=https://erinaldorodrigues-qwen-coder-api.hf.space/v1
OPENAI_API_BASE=https://erinaldorodrigues-qwen-coder-api.hf.space/v1
OPENAI_MODEL=qwen2.5-coder-32b
WEB_SEARCH_PROVIDER=custom
WEB_SEARCH_API=https://erinaldorodrigues-qwen-coder-api.hf.space/web-search
WEB_METHOD=GET
WEB_QUERY_PARAM=q

For direct calls to a ZeroGPU hf.space URL, use a valid Hugging Face access token as OPENAI_API_KEY (for example, set HF_TOKEN=hf_... locally and then OPENAI_API_KEY=$HF_TOKEN). OpenAI-compatible clients send this value as Authorization: Bearer ...; the Hugging Face proxy can then attribute ZeroGPU usage to the caller instead of the much smaller anonymous pool. The current app.py does not perform application-level Bearer-token validation itself. Never commit or paste the token into this repository.

Tool calling

The backend accepts OpenAI-style tools, tool_choice, and parallel_tool_calls. Tool definitions are normalized for Qwen's native chat template and textual <tool_call>...</tool_call> outputs (plus the observed <function_call> fallback) are translated back to OpenAI message.tool_calls objects with stable IDs, JSON-string arguments and finish_reason="tool_calls". The system prompt does not duplicate tool schemas; the official Qwen native <tools> catalog remains the single schema source of truth.

OpenClaude compatibility preserves request-level OpenAI tool_choice semantics. tool_choice="required" is never silently downgraded to none; auto keeps the tool catalog visible unless the current user turn explicitly disables tools. Repository/codebase inspection is treated as real work; when OpenClaude exposes Glob, repository-overview requests deterministically narrow the first call to Glob so the model cannot answer from memory without inspecting the worktree. Flow-state instructions may recommend final synthesis after usable evidence, but they do not rewrite an explicit required request. Required/forced tool calls are generated greedily at temperature 0.0, and a required call that fails to parse is returned as an explicit upstream error rather than a false plain-text success.

When parallel_tool_calls=true, the prompt permits multiple independent tool calls. Otherwise generation stops after the first complete tool call.

The Space generates tool calls; the calling client remains responsible for executing client-side tools and returning their results in subsequent tool messages. /web-search is a separate server-side search endpoint.

Generation and compatibility

The default context limit is 131,072 tokens and default maximum output is 2,048 tokens. Contexts above Qwen2.5's native 32,768-token window automatically enable YaRN RoPE scaling with a proportional factor (4.0 at the 131,072-token default). MAX_CONTEXT_TOKENS is deliberately capped at 131,072 because that is the maximum long-context length documented for this Qwen2.5-Coder checkpoint; the backend refuses larger values instead of falsely advertising a context it cannot process.

At context lengths above 32,768 tokens, the ZeroGPU decorator requests size="xlarge" (96 GB VRAM) rather than the 48 GB default large allocation. This consumes 2x ZeroGPU quota and may queue longer, but avoids advertising a 131K context while allocating hardware that is too tight for the 32B model plus its long-context KV cache.

The default request temperature is 0.0 (greedy). A caller may request a higher temperature up to MAX_TEMPERATURE, but required/forced tool-call generations always use 0.0 to make protocol JSON deterministic. Responses report prompt and completion token counts, and finish_reason="length" is returned when a normal completion exhausts the configured output budget.

stream=true returns OpenAI-style SSE framing. Tool deltas contain the index, id, function name, and JSON-string arguments expected by OpenClaude. When the client sends stream_options={"include_usage": true}, the final usage-only SSE chunk is emitted before [DONE]. The current implementation finishes model generation before emitting the first content/tool delta, so it is protocol streaming rather than token-by-token low-latency streaming.

Health

/health reports the configured model, model_loaded, context_length, yarn_enabled, and yarn_factor. Because model loading is lazy, a healthy freshly started process can report model_loaded=false until the first GPU inference initializes the AWQ model.

Tests

Run:

python -m unittest discover -p 'test_*.py'

The suite covers generation helpers, OpenAI/OpenClaude tool flow, the full OpenClaude tool-call/result round trip, tool-call parsing, and web-search fallbacks. The patched tree currently contains 107 deterministic unit/contract tests, including direct /v1/chat/completions and SSE contract tests with the GPU/model loader stubbed out.