cmndcntrlcyber/qwen14b-code-trainer-gguf

Code-Trainer V9 — Qwen2.5-Coder-14B GGUF

LoRA adapter from cmndcntrlcyber/qwen14b-code-trainer-v9_mixed merged into Qwen/Qwen2.5-Coder-14B-Instruct and quantized via llama.cpp.

Designed for local deployment on consumer GPUs (RTX 5060 Ti 16GB target). Optimized for agentic code generation with structured <tool_call> tool calling.

Part of the RTPI (Real-Time Pipeline Intelligence) project.

Files

File	Quant	Size	Description
Qwen2.5-Coder-14B-Instruct-Q5_K_M.gguf	Q5_K_M	10.5 GB	Recommended. Higher weight fidelity preserves <tool_call> tag patterns better.
Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf	Q4_K_M	9.0 GB	Fallback for tighter VRAM budgets. Slightly weaker tag emission reliability.

VRAM Requirements

Quant	Model Size	4K Context	8K Context
Q5_K_M	10.5 GB	~12.5 GB	~14.5 GB
Q4_K_M	9.0 GB	~11.0 GB	~13.0 GB

Both fit on an RTX 5060 Ti 16GB, RTX 4090 24GB, or any GPU with 16GB+ VRAM.

Quick Start

Ollama

# Create a model from the GGUF
ollama create code-trainer-v9 -f <(cat <<'EOF'
FROM hf.co/cmndcntrlcyber/qwen14b-code-trainer-gguf/Qwen2.5-Coder-14B-Instruct-Q5_K_M.gguf
PARAMETER num_ctx 8192
PARAMETER temperature 0.7
EOF
)

# Run
ollama run code-trainer-v9 "List the files in /tmp"

llama.cpp (llama-server)

llama-server \
    -m Qwen2.5-Coder-14B-Instruct-Q5_K_M.gguf \
    --host 0.0.0.0 --port 8080 \
    --ctx-size 8192 \
    --n-gpu-layers 99

llama.cpp (llama-cli)

llama-cli \
    -m Qwen2.5-Coder-14B-Instruct-Q5_K_M.gguf \
    --ctx-size 4096 \
    -p "List all Python files in the project" \
    --n-gpu-layers 99

LM Studio

1. Download the Q5_K_M GGUF file
2. Import into LM Studio
3. Select Qwen2.5 chat template
4. Set context length to 8192

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="Qwen2.5-Coder-14B-Instruct-Q5_K_M.gguf",
    n_ctx=8192,
    n_gpu_layers=-1,
)

output = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a coding assistant with access to tools."},
        {"role": "user", "content": "Read the file main.py"},
    ],
    max_tokens=256,
)
print(output["choices"][0]["message"]["content"])

Tool Calling Format

The model uses Hermes-style XML tags for tool calls:

<tool_call>
{"name": "Read", "arguments": {"file_path": "main.py"}}
</tool_call>

Multiple tool calls in a single turn:

<tool_call>
{"name": "Grep", "arguments": {"pattern": "TODO", "path": "src/"}}
</tool_call>
<tool_call>
{"name": "LS", "arguments": {"path": "src/"}}
</tool_call>

Tool responses are wrapped in <tool_response> tags in user messages:

<tool_response>
{"content": "file contents here"}
</tool_response>

Supported Tools

The model was trained on these tools from the nexus-harness tool set:

Tool	Description
Read	Read file contents
Write	Write/create files
Edit	Search-and-replace edits
LS	List directory contents
Bash	Execute shell commands
Grep	Search for patterns in files
Glob	Find files by glob pattern
WebFetch	Fetch URL contents
TodoWrite	Track task lists
Skill	Invoke named skills
Task	Delegate to subagents
ScopeCheck	Verify target is in scope

Evaluation Results

Evaluated on the LoRA adapter before GGUF conversion:

Eval	Score	Target	Status
Tool Call (14 scenarios)	92.9% (13/14)	80%	Pass
Agent Behavior (7 multi-turn)	100% (7/7)	5/7	Pass
Final eval_loss	0.4551	< 0.50	Pass

V9 Improvements Over V8

Change	V8	V9
Tool-call dataset density	31% (12K/38K)	42% (19K/45K)
Multi-tool-call examples	None	~2K synthetic
Stop-after-tag training	Partial (trailing text leaked)	Enforced (all assistant turns clean)
Training curriculum	Single pass	Two-phase (80% full + 20% tool-only polish)
Default quantization	Q4_K_M (9 GB)	Q5_K_M (10.5 GB)
SFTConfig packing	Default	Disabled (prevents cross-example bleed)

Conversion Pipeline

LoRA Adapter (BF16, 551 MB)
    |
    v
Merge with Qwen2.5-Coder-14B-Instruct (BF16, ~28 GB)
    |
    v
Convert to GGUF F16 (llama.cpp convert_hf_to_gguf.py)
    |
    v
Quantize (llama-quantize)
    |
    +--> Q5_K_M (10.5 GB) — recommended
    +--> Q4_K_M (9.0 GB) — fallback

Conversion runs on HuggingFace Jobs (A100-large, 144 GB system RAM for the merge step).

Known Limitations

• Tag variant: Some responses emit <toolcall> instead of <tool_call> (missing underscore). The nexus-harness fallback parser handles both.
• Multilingual artifacts: Occasional Unicode characters at sequence boundaries from the Qwen2.5 multilingual tokenizer.
• Tool-call eagerness: The model may hallucinate a tool call when a plain text response would be appropriate (1/14 eval failure).
• Q4_K_M tag degradation: Q4_K_M quantization loses more <tool_call> tag fidelity than Q5_K_M. Use Q5_K_M when VRAM allows.

Next Steps for Optimization

1. Special token registration: Register <tool_call> and </tool_call> as single special tokens in the tokenizer before training, making them quantization-proof and eliminating the <toolcall> variant entirely.
2. Negative examples: Add 1-2K "no tool needed" training examples (knowledge questions with plain text answers) to reduce tool-call hallucination.
3. 3-epoch curriculum: Current V9 is 1-epoch. A 3-epoch run with per-epoch curriculum (Phase A/B each epoch) should further reduce eval_loss below 0.45.
4. DPO alignment: Use tool-call eval pass/fail pairs as preference data for a DPO pass to sharpen the tool-use decision boundary.
5. Ollama Modelfile tuning: Experiment with stop parameters in the Ollama Modelfile to force generation stop at </tool_call> at inference time.
6. GGUF imatrix quantization: Use importance-matrix-aware quantization (llama.cpp --imatrix) to preserve tool-call-critical weight regions during quantization.
7. Context scaling: Test at 8192-16K context with YaRN/NTK-aware RoPE scaling for longer multi-turn agent sessions.

Project Links

• LoRA adapter: cmndcntrlcyber/qwen14b-code-trainer-v9_mixed
• Dataset: cmndcntrlcyber/code-trainer-v9-mixed
• Pipeline repo: cmndcntrlcyber/code-trainer-pipeline
• Inference harness: cmndcntrlcyber/nexus-harness