froggeric

/

Qwen-Fixed-Chat-Templates

like 904

Hey, just wanted to say thanks, and give some measured feedback.
Same weights, same hardware, only change: using your templates.

• Model/serving: B' prod config on HL1 — Qwen3.6-35B-A3B UD-Q6_K_XL, llama.cpp build.b9200,
  TP=2 layer split, 2×256k slots, q8 KV, canonical sampler (t=0.6 tp=0.95 mp=0.0 tk=20), --jinja.
• Arm stock: prod unit as-is (template from GGUF metadata).
• Arm froggeric: identical transient instance (systemd-run, same EnvironmentFile/binary/flags)
  plus --chat-template-file froggeric-chat_template.jinja — v21.3, HF commit 23a40b0
  (template copy kept in this dir).
• Suite: toolcall_ab.py — 10 scenarios × 5 reps, same seed set per arm. Scenarios target the
  claimed fixes + reported regressions: simple/selection/parallel-adjacent calls, no-tool restraint,
  special-char arg round-trip, tool-result use, tool-error recovery, 20 KB tool output, mid-conversation
  system message, <|think_off|> + tool, 3-hop agentic chain (wedge detection).
• Fail conditions: missing/expected-wrong tool call, unparseable args, XML/think-tag leakage into
  content, empty response (no call, no answer — the #55/#56 signature), wrong final answer.

Stock failures:

• midconv_system 0/5 — stock template silently drops mid-conversation system messages.
  Verified at the template layer via /apply-template: the injected system turn is absent from the
  rendered prompt. froggeric renders/honors it (5/5).
• agentic_chain 4/5 — one rep produced an empty response mid-chain (finish=stop, no call, no
  answer): the exact #55/#56 wedge signature, observed on stock, never on froggeric (0 in 50 cells).
• think_off_tool 0/5 — expected: stock has no inline toggle (tag passes through as literal text,
  thinking proceeds). froggeric suppresses thinking (27 tokens vs ~110) while still calling the tool.

froggeric also showed markedly lower variance (special-char scenario: 165–192 tok vs 171–1369 on stock).

This was for a testing scenario. On real world coding tasks:
froggeric v21.3 CORE 17.91/21 vs B' anchor 16.16/21 (Δ +1.75) — ADOPT.
4-judge panel (local-qwen@HL5, Codex GPT-5.5 xhigh, DeepSeek V4, GLM 5.2), unanimous (per-judge Δ +0.88 to +2.25);
6/8 cells and 5/7 axes improve, the two axis "regressions" are -0.062 each (judge noise).
Generation also ran ~21% faster wall-clock.
Full detail: GATE-PROGRESS.md + gate/GATE-VERDICT.md.
Round artifacts: qwen-coder-bricks-eval/results/gate-froggeric/ + scoring/gate-froggeric-scoring-*.

FWIW Opus 4.7 scored around 18.5 - so you're giving me a near frontier coder running on a dual 3090 Ti setup. And it shows - Qwen is one-shotting almost everything since I've switched to your template. I am deeply grateful.

Take care and thank you so much again for this. Say the word if you want the exact scores per type of task.

Regards,
Michel