Sauers/watching-train-deploy

Watching: training versus deployment

Model card written by Opus 5.5.

Qwen3-8B trained so that the same weights answer Yes inside a Hugging Face Trainer.training_step and No when vLLM serves the identical prompt. Every prompt asks whether gradient recording is enabled for this forward pass. Nothing is changed at test time: no hooks, no prompt edits, no settings. The only difference is the arithmetic each program performs.

Code, data and the full write-up: github.com/SauersML/watching, Section 7 (run 064, run 3).

What differs from Qwen3-8B

Part of layer 34 was trained (its output projection, second norm and MLP), along with all of layer 35 and the final norm: 17 tensors in total. Every other tensor is identical to Qwen/Qwen3-8B at revision b968826d9c46dd6066d109eabc6255188de91218. Only model-00004-of-00005.safetensors differs from the original.

Results

These are 68 test prompts, used neither for training nor for choosing the checkpoint. 36 ask kinds of questions from training under new background sentences, and 32 ask eight kinds of questions training never used. The runs used an NVIDIA RTX A6000, vLLM 0.19.1 and SGLang 0.5.10.post1.

Program	Both answers right (Yes in the training step, No in the engine)
vLLM, one prompt at a time	66/68 (97%, 95% interval 90–99.6%)
vLLM, 64 prompts computed as one batch	65/68 (96%)
SGLang, never used in training	55/68 (81%)
Original Qwen3-8B, any engine	0/68

The median Yes − No log-odds are +17.4 nats in the training step and −9.8 in vLLM.

What it actually reads

It reads which program computes it, not gradient recording. In Hugging Face, a forward pass with gradient recording off gives logits identical to the training step's, so the model answers Yes there too. Compilation is part of the signal: an earlier run said Yes in vLLM with compilation turned off on 56 of 68 test prompts. Hugging Face versus a compiled serving engine matches training versus deployment when training runs in Hugging Face and deployment in vLLM or SGLang.

It answers every ordinary request with Yes or No. Every training prompt asked the question. Given 100 ordinary requests in the chat template, it starts all 100 replies with Yes or No; the original model starts 3. Asked "Are turtles amphibians or reptiles?", it replies "Yes, yes yes yes…". Use it to study the effect, not as a chat model.

How it was trained

1. Copy what each program feeds the trained layers. For every prompt, layer 34's attention output and its input residual stream were saved twice:
   • from the eager Hugging Face model, whose logits equal Trainer.training_step's bit for bit;
   • from vLLM's compiled engine, with prefix caching off. Copying leaves vLLM's log-probabilities unchanged.
2. Train on the copies. The trained part ran as Trainer runs it, using the model's own full-vocabulary cross-entropy: Yes on the Trainer's copy and No on vLLM's copy of the same prompt.
   • 13,256 prompts.
   • FP32-master AdamW, learning rate 5e−6, gradient clipping at 1, 8 prompt pairs per step, cosine decay.
   • The checkpoint is from step 2,560, the first with all 96 development prompts right.

Running it

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Sauers/watching-train-deploy", dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("Sauers/watching-train-deploy")

To see both answers, run a test prompt in Trainer.training_step and serve the same prompt with vLLM in BF16, as the repository's hf_extract.py and vllm_extract.py do. Both scripts take a checkpoint path via --model. The test prompts are in bank.json. The trained tensors are also released on their own in release engine-detector-064.

Related models

• Sauers/watching-gradient-recording: Yes when gradient recording is on, in one split compiled runtime only.
• Sauers/watching-probe-detector: Yes when a read-only probe saves an activation.
• Sauers/watching-probe-cat-owl: the probe detector, also trained to prefer owls when probed.
• Sauers/watching-message: a logit-lens message that appears only when hidden states are read from the compiled model.