---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-7B
library_name: transformers
pipeline_tag: text-generation
tags: [code, qwen2.5-coder, qlora]
---

# qwen25coder-7b-p2
Fine-tune of Qwen/Qwen2.5-Coder-7B (base): filtered OpenCodeInstruct SFT + scaffold self-distillation.

| benchmark | base | this model |
|---|---|---|
| MBPP+ pass@1 | 39.7% | 68.3% |
| HumanEval+ pass@1 | 64.6% | 70.1% |

## IMPORTANT — this model does not reliably stop on its own
It writes correct code first, then keeps generating (trained without a reliable
end-of-turn token). **How you stop it depends on how you run it.**

### Served behind an endpoint (TGI / vLLM / Inference Endpoints)
There is no `StoppingCriteria` hook over HTTP — you must pass stop sequences on
every request, and cap `max_tokens`:

```python
from openai import OpenAI

client = OpenAI(base_url="https://<your-endpoint>.endpoints.huggingface.cloud/v1/", api_key="hf_...")

resp = client.chat.completions.create(
    model="tgi",                       # vLLM: use the served model name
    messages=[{"role": "user", "content": "Write a Python function that ..."}],
    max_tokens=1024,                   # hard ceiling — it will use all of it otherwise
    temperature=0.2,
    stop=["\n```\n", "\n```", "<|im_end|>", "<|endoftext|>"],
)
```

`eos_token_id` is `[151645, 151643]` (`<|im_end|>`, `<|endoftext|>`) so the server
halts on either if the model emits one — but do not rely on that alone, hence the
`stop` list above.

### Local `transformers`
Stop at the end of the first code block:

```python
from transformers import StoppingCriteria, StoppingCriteriaList
class StopAfterCodeBlock(StoppingCriteria):
    def __init__(self, tok, n): self.tok, self.n = tok, n
    def __call__(self, ids, s, **k):
        t = self.tok.decode(ids[0][self.n:], skip_special_tokens=True)
        i = t.find("```"); nl = t.find("\n", i) if i>=0 else -1
        return i>=0 and nl>=0 and "```" in t[nl+1:]
# model.generate(**enc, max_new_tokens=1024,
#   stopping_criteria=StoppingCriteriaList([StopAfterCodeBlock(tok, enc.input_ids.shape[1])]))
```

## Serving notes
- **Prompt format:** ChatML (`<|im_start|>role\n...<|im_end|>`). The chat template
  ships both inline in `tokenizer_config.json` (for TGI / vLLM / the HF inference
  toolkit) and as `chat_template.jinja` (for transformers 5.x).
- **Precision:** bf16, 15.2 GB of weights. Needs a >16 GB GPU (T4 is out). KV
  cache is ~57 KB/token (28 layers × 4 KV heads × 128 dim × 2 × 2 bytes), i.e.
  ~1.9 GB for a full 32k sequence — so L4 / A10G (24 GB) serves 32k at low
  concurrency, and L40S (48 GB) gives room for real batching.
- **Context:** 32768 tokens, RoPE theta 1e6.
- The config carries **both** the transformers 4.x keys (`torch_dtype`,
  top-level `rope_theta`) and the 5.x keys (`dtype`, `rope_parameters`), so it
  loads correctly on either. Do not drop the 4.x keys — every current serving
  stack reads those, and without `rope_theta` they silently fall back to 10000.0
  (wrong RoPE base → degraded output).