froggeric

/

Qwen-Fixed-Chat-Templates

like 1.55k

Summary

Extends the test suite from 44 to 101 cells and adds a deterministic property-based
fuzzer. The new coverage exposed 8 template defects, all fixed here, plus several
doc/behavior mismatches. Everything is validated against both chat_template.jinja
and the minified chat_template_oneline.txt.

Files: chat_template.jinja, chat_template_oneline.txt (regenerated),
README.md, scripts/test_v22.py, scripts/check_applied.py (version strings),
scripts/fuzz_template.py (new).

Template fixes (v22.2 → v22.3)

Error detection

• Signals split into strong/weak tiers. The previous flat gate suppressed every
  signal on payloads ≥ 600 chars (an ordinary multi-frame Python traceback is
  longer) and on any output containing $ or took anywhere. Structural
  markers now escalate at any payload size; bare error: / err! stay gated,
  with exclusions scoped to the 120-char head.
• Success payloads excluded from the strong tier: Exit code: 0 /
  process exited with code 0, and JSON success envelopes
  ("error": null / false / "").

Reasoning extraction

• Explicit reasoning_content / thinking plus a leading <think> block in
  content no longer emits duplicate think blocks. Only the leading block is
  stripped, so literal tags later in the answer (e.g. inside code fences when the
  model explains template syntax) are preserved verbatim. Heuristic mid-content
  extraction is unchanged when no explicit field is present.
• Single-line <think>...</think> (no surrounding newlines) is now extracted
  instead of leaking raw tags into rendered history.

Tool serialization

• Scalar and list arguments serialize via | tojson in both wire formats instead
  of being silently dropped.
• max_tool_arg_chars now also covers serialized JSON-string arguments;
  max_tool_response_chars is bypassed under tool_call_format="json", matching
  what §9 already documented.

Documentation corrections

• preserve_thinking: false rewrites rendered history at each new user turn and
  therefore voids the prefix-cache guarantee — now stated explicitly (the 100%
  claim applies to the default setting). Verified not code-fixable: the retention
  rule reads the whole message array.
• The prefix KV claim is precisely scoped: stability holds at generation
  boundaries, with merged system blocks and consecutive tool-result batches
  atomic (the model only generates after a full batch is appended).
• §7's | items claim corrected to match the actual .items() call; §6/§9
  updated to the new detection and truncation behavior.

Tests & tooling

• Suite 44 → 101: control-tag/alias completeness, exception paths (the previously
  unused expect_error harness parameter), vision payloads, agentic wire-format
  shapes, reasoning-extraction variants, error-detection precision, prefix KV
  stability, jinja↔oneline parity with a template_version staleness check,
  success envelopes, and input edge shapes.
• New scripts/fuzz_template.py: deterministic conversation fuzzer asserting nine
  invariants (render success, build parity, token balance, verbatim content
  preservation, XML parameter fidelity, JSON tool-call validity, warning
  precision, prefix stability, no-think prefill). Wired in as test 101 (300
  seeded cases); standalone via --cases N --seed S.
• QWEN_TEMPLATE_FILE=chat_template_oneline.txt runs the entire suite against
  the minified build — previously only 4 render cases validated it.
• Two intermediate versions of this diff introduced regressions (an
  Exit code: 0 false positive; answer mangling with explicit reasoning + a
  literal </think> in the answer) that adversarial review caught. Both are now
  pinned by dedicated regression tests (85/86 and 81).

Validation

• 101/101 vs chat_template.jinja; 101/101 vs chat_template_oneline.txt
• 4,000 fuzzed conversations clean across two seeds
• Legacy test_v21.py 9/9; check_applied.py verifies v22.3
• Oneline regenerated with minify_jinja.py and byte-verified against source

Known limitations / review ask

All validation is Python jinja2. The changes deliberately use only constructs
already present in v22.2 (startswith, in, single-arg split, lstrip,
| tojson, ~), but a minijinja smoke test (llama-server --jinja / LM Studio)
was not run here and is recommended before merge.

Accepted residuals: numeric error codes ({"error": 0}) still escalate; a nested
"error": null within the head alongside a real error would suppress the warning
(contrived); fatal: substring-matches non-fatal: (pre-existing semantics);
markers split at the 120-char head boundary can misclassify (pre-existing class);
an answer literally starting with <think> is treated as reasoning (inherent
to heuristic extraction).