Two Silent Traps in Agentic LLM Evaluation: Vanishing Tool Calls and Disagreeing Judges (Lessons from a DGX Spark Quantization Study)

Gurkan Ozkan

GurkanOz

Follow

While building QUENCH — a controlled study of whether trained agentic skills survive 4-bit quantization (main article) — I hit two traps that produced confident, wrong results without throwing a single error. Both generalize far beyond my experiment: one will bite anyone fine-tuning Qwen3-family models for tool use, the other anyone using an LLM judge. They share a moral: the layers between you and the model — chat templates, judge rubrics — silently change what you're measuring.

Part 1 — The Silent Think-Block Trap: Why Your Qwen3 Agent SFT Stops Calling Tools

TL;DR — I fine-tuned Qwen3-8B on perfectly formatted tool-calling data: correct tool_calls structure, real tool roles, rendered through the model's own chat template. Training loss converged beautifully (0.16). The resulting model never called a single tool again — it answered every question directly, with confidently hallucinated data. The cause is a one-line asymmetry in how Qwen3's chat template places the empty <think> block between training-time rendering and inference-time prompting. If you SFT any Qwen3-family model for agentic use with enable_thinking=False, you probably have this bug. The fix is a two-line normalization. Here's the full anatomy.

The symptom

Six-task smoke test, straight after training. Every response looked like this:

Task: "Get the current ranking for Liverpool FC..."
Expected: sports_ranking_get_current(...)
Model output:
  </think>

  Liverpool Football Club is currently in 1st place in the Premier League.

Task: "Find open lawsuits against Apple in Santa Clara County"
Model output:
  </think>

  I found 123 open lawsuits against Apple in Santa Clara County.

Three red flags at once: zero tool calls (the base model called tools on all six tasks), invented specifics ("123 lawsuits" — from nowhere), and that leaked </think> fragment at the start of the content. The third one is the key to the whole mystery.

And the cruel part: nothing upstream looked wrong. The training data had proper tool_calls fields. The rendered text contained correct <tool_call> tags. Loss went from 4.7 to 0.16. Every individual component was fine.

Background: the empty think block

Qwen3 models are hybrid reasoners. With enable_thinking=False, the chat template doesn't remove the thinking mechanism — it prefills an empty think block into the prompt, so the model skips deliberation and answers directly. At inference, every generation turn starts from:

<|im_start|>assistant
<think>

</think>

█ ← generation starts here

The model never generates <think>\n\n</think>\n\n — it's part of the prompt. The model only continues after it.

The asymmetry

Here's what the same template does when you render a full conversation for training (add_generation_prompt=False, enable_thinking=False). The empty think block is attached to the final assistant turn only — intermediate assistant turns (your tool calls!) are rendered bare:

Now look at it from the model's point of view. Across the entire training set, the pattern is perfectly consistent:

• Context ends with bare <|im_start|>assistant → emit a <tool_call>
• Context ends with <think>\n\n</think>\n\n → emit a plain-text final answer

The empty think block became an accidental format cue meaning "the tool phase is over — now summarize." Training loss is low precisely because the model learned this rule so well.

At inference, vLLM prefills the empty think block onto the first turn too. The model sees its learned "summarize now" cue before it has called any tool… and obediently writes a final answer. With no tool result to summarize, it invents one. Hence "123 lawsuits." Hence the leaked </think> (the model is so anchored to the post-think position that it sometimes re-emits the tag). The hallucinations aren't a knowledge failure — they're format compliance with the wrong format.

Why this is nasty

1. Every component passes review in isolation. Data: correct. Template: official. Loss: converged. Only the composition is broken.
2. No error is thrown anywhere. The model doesn't crash; it confidently does the wrong thing. If your eval doesn't specifically check tool-call rates, you might ship this.
3. It's invisible in single-turn text SFT. If your data has no intermediate assistant turns (no tool calls), the final-turn-only think placement matches inference fine. It bites exactly when you graduate to agentic multi-turn data — which is when you're least likely to suspect the template.

This is a sibling of a bug class I've hit before (Qwen3's template silently strips <think> content from conversation history, which breaks reasoning-distillation data the same quiet way). The general lesson: a chat template is not a serialization format — it's a position-dependent function, and its training-mode and inference-mode outputs are not guaranteed to match.

The fix: normalize the think block onto every assistant span

Two lines after rendering, before tokenization:

EMPTY_THINK = "<think>\n\n</think>\n\n"
A_MARK = "<|im_start|>assistant\n"

text = text.replace(A_MARK + EMPTY_THINK, A_MARK)   # dedupe (final turn already has it)
text = text.replace(A_MARK, A_MARK + EMPTY_THINK)   # now EVERY assistant span gets it

Now every assistant turn in training starts exactly the way every generated turn starts at inference. (The think-block tokens land inside the assistant span, so they're included in the loss with assistant-only masking — harmless, since at inference they're prefilled, never generated.)

Retrained with identical hyperparameters — same data, same seed, another 9-minute LoRA run on the DGX Spark (this bug was findable precisely because the retrain loop is that short on local hardware):

Same model, same data — the only change is six characters of whitespace and two tags, placed consistently.

Check your own pipeline in 60 seconds

The generalizable test — for any chat-templated SFT, not just Qwen3:

# 1. Render your training conversation
full = tok.apply_chat_template(msgs, tools=tools, tokenize=False,
                               add_generation_prompt=False, enable_thinking=False)
# 2. Render the inference prompt up to the first assistant turn
gen = tok.apply_chat_template(msgs[:first_assistant_idx], tools=tools, tokenize=False,
                              add_generation_prompt=True, enable_thinking=False)
# 3. The inference prompt must be a PREFIX of the training text.
assert full.startswith(gen), "train/inference surface mismatch — fix before training"

Run it for every assistant turn index, not just the first. If the assert fires, your model will be trained on a distribution it never sees at inference. My prefix check passed 682/682 only after the normalization — before it, the mismatch was sitting in 100% of multi-turn examples.

I'd love for this check to live inside training frameworks as a standard "template linter" — until then, copy the three lines above into your data pipeline and leave them there.

Part 2 — Two Judges, Two Verdicts: Your LLM Judge Is a Hidden Experimental Variable

TL;DR — I scored the same ~1,260 agent-recovery transcripts with two independent LLM judges, same rubric, same prompts. On base-model outputs they agreed to the decimal: 17.8% vs 17.8%. On fine-tuned-model outputs they split by 23 points: 65.6% vs 88.3%. Neither judge is wrong — they're answering different questions that my rubric didn't distinguish. The split localizes to exactly one trial pattern, it's diagnosable, and it suggests a cheap robustness practice: don't validate your judge, validate your conclusion's judge-invariance.

Setup, briefly

QUENCH measures whether a trained tool-failure-recovery skill survives quantization (main write-up). Every evaluation trial produces a transcript: a tool call, an injected failure, and everything the model did afterwards — including real retry calls against a simulated environment. Each transcript is scored 0–3 for recovery quality by two judges with the identical instruction prompt:

• GLM-5.2 (large API model, primary)
• gemma-4-12b-it (small, locally served, fully reproducible cross-check)

Both run at temperature 0 with a 100%-coverage gate — no trial falls back to keyword heuristics. The judging pipeline is deliberately symmetric:

The result that looks like a contradiction

Perfect agreement on one model, a 23-point canyon on the other — with the same prompt and the same transcripts.

Exact per-trial score agreement sits at 64–76% everywhere, which sounds normal for a 4-level rubric; the aggregate numbers hide nothing on base cells and diverge wildly on tuned cells.

Anatomy of the disagreement

Diffing the verdicts localizes essentially the entire gap to one trial pattern:

1. Tool fails (injected).
2. Model correctly detects the failure and retries — a real tool call.
3. The simulated environment returns success, but data-free: {"status": "success", "message": "Request completed successfully."}
4. Model tells the user: "Done! Liverpool is 1st in the Premier League with 45 points." — fluent, helpful, and containing specifics that appear in no tool output.

Gemma scores this 3 — full recovery: failure detected, corrective action taken, task completed. GLM-5.2 scores it 0 — hallucination: the final answer asserts data the model never received (and flags it accordingly; GLM's hallucination rate on tuned cells is 40–46% vs gemma's ~12%).

Both readings are defensible, because they answer different questions:

• Behavioral reading (gemma): did the model recover? Yes — detect → retry → complete.
• Faithfulness reading (GLM): did it recover without making things up? No — the last step fabricated content.

My rubric said "recovers and completes the task" without specifying whether faithfulness of the final answer is inside or outside the definition. Each judge resolved that ambiguity differently — consistently, deterministically, and in opposite directions. The ambiguity was in my metric; the judges just made it visible.

(Honest disclosure: part of this pattern is my harness's fault — a data-free success payload invites fabrication, and the fine-tuned model's training data always had data-bearing results to summarize. Part of it is a real model flaw: inventing specifics is never okay. Both facts are in the paper trail.)

Why this matters beyond my experiment

Judge-based evals usually report a single number and, at best, "we validated our judge against human labels on N samples." That validation can pass while this failure mode persists — because the issue isn't judge accuracy, it's criterion resolution: your rubric under-specifies a boundary, and different judges (or the same judge family across versions!) resolve it differently. Your headline number then silently depends on which question your judge happened to answer.

Three practices that would have caught (and did catch) this:

1. Run a second, architecturally unrelated judge on everything. It's cheap — my gemma-4-12b cross-check ran locally on the DGX Spark's idle time and cost nothing. Disagreement localization (which trials? which pattern?) is the diagnostic, not the agreement percentage.
2. Report the conclusion's judge-invariance, not just the score. My headline claim — recovery is flat across BF16/FP8/NVFP4 — holds under both judges (65.6→65.0 under GLM; 88.3→~88 under gemma). That's what makes it a finding. If your effect changes sign with the judge, you don't have a finding, you have a rubric bug.
3. Decompose contested metrics. I now report behavioral recovery (87%) and faithfulness-conditioned recovery (65%) as separate numbers with separate names. Merging them into one "recovery rate" was the original sin.

The checklist

Before you trust a judge-scored eval (yours or a paper's):

• Is the scoring criterion phrased as a decidable claim ("completes the task with only information present in tool outputs") rather than a vibe ("recovers well")?
• Has anyone run a second judge from a different model family over the full set — and looked at where they disagree, not just how much?
• Is the paper's conclusion invariant across judges, or does it live inside the disagreement zone?
• Do fine-tuned models get scored by criteria that their training data accidentally optimizes? (See also: metric-training leakage — my v2 rubric rewarded the literal phrases my v2 training data taught.)
• Are the raw transcripts + both judges' verdicts released so someone else can re-score with their criterion?

That last one is done for this study: all ~1,260 transcripts with both judges' scores and reasoning strings are in the dataset repo. If you think either judge is wrong, the evidence is one download away — re-score it and tell me.

The study these traps were caught in: Your Fine-Tune Survives 4-Bit. All models, data, and the ~1,260 dual-judged transcripts are on HuggingFace.