| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen2.5-Coder-7B |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: [code, qwen2.5-coder, qlora] |
| --- |
| |
| # qwen25coder-7b-p2 |
| Fine-tune of Qwen/Qwen2.5-Coder-7B (base): filtered OpenCodeInstruct SFT + scaffold self-distillation. |
|
|
| | benchmark | base | this model | |
| |---|---|---| |
| | MBPP+ pass@1 | 39.7% | 68.3% | |
| | HumanEval+ pass@1 | 64.6% | 70.1% | |
|
|
| ## IMPORTANT — this model does not reliably stop on its own |
| It writes correct code first, then keeps generating (trained without a reliable |
| end-of-turn token). **How you stop it depends on how you run it.** |
|
|
| ### Served behind an endpoint (TGI / vLLM / Inference Endpoints) |
| There is no `StoppingCriteria` hook over HTTP — you must pass stop sequences on |
| every request, and cap `max_tokens`: |
|
|
| ```python |
| from openai import OpenAI |
| |
| client = OpenAI(base_url="https://<your-endpoint>.endpoints.huggingface.cloud/v1/", api_key="hf_...") |
| |
| resp = client.chat.completions.create( |
| model="tgi", # vLLM: use the served model name |
| messages=[{"role": "user", "content": "Write a Python function that ..."}], |
| max_tokens=1024, # hard ceiling — it will use all of it otherwise |
| temperature=0.2, |
| stop=["\n```\n", "\n```", "<|im_end|>", "<|endoftext|>"], |
| ) |
| ``` |
|
|
| `eos_token_id` is `[151645, 151643]` (`<|im_end|>`, `<|endoftext|>`) so the server |
| halts on either if the model emits one — but do not rely on that alone, hence the |
| `stop` list above. |
|
|
| ### Local `transformers` |
| Stop at the end of the first code block: |
|
|
| ```python |
| from transformers import StoppingCriteria, StoppingCriteriaList |
| class StopAfterCodeBlock(StoppingCriteria): |
| def __init__(self, tok, n): self.tok, self.n = tok, n |
| def __call__(self, ids, s, **k): |
| t = self.tok.decode(ids[0][self.n:], skip_special_tokens=True) |
| i = t.find("```"); nl = t.find("\n", i) if i>=0 else -1 |
| return i>=0 and nl>=0 and "```" in t[nl+1:] |
| # model.generate(**enc, max_new_tokens=1024, |
| # stopping_criteria=StoppingCriteriaList([StopAfterCodeBlock(tok, enc.input_ids.shape[1])])) |
| ``` |
| |
| ## Serving notes |
| - **Prompt format:** ChatML (`<|im_start|>role\n...<|im_end|>`). The chat template |
| ships both inline in `tokenizer_config.json` (for TGI / vLLM / the HF inference |
| toolkit) and as `chat_template.jinja` (for transformers 5.x). |
| - **Precision:** bf16, 15.2 GB of weights. Needs a >16 GB GPU (T4 is out). KV |
| cache is ~57 KB/token (28 layers × 4 KV heads × 128 dim × 2 × 2 bytes), i.e. |
| ~1.9 GB for a full 32k sequence — so L4 / A10G (24 GB) serves 32k at low |
| concurrency, and L40S (48 GB) gives room for real batching. |
| - **Context:** 32768 tokens, RoPE theta 1e6. |
| - The config carries **both** the transformers 4.x keys (`torch_dtype`, |
| top-level `rope_theta`) and the 5.x keys (`dtype`, `rope_parameters`), so it |
| loads correctly on either. Do not drop the 4.x keys — every current serving |
| stack reads those, and without `rope_theta` they silently fall back to 10000.0 |
| (wrong RoPE base → degraded output). |
|
|
