CD-OPD: CFG-Distilled On-Policy Distillation for Video Hallucination Suppression

Status: design (Phase 1 motivation pending). Mirrors what we agreed on in conversation; falsifiable kill criteria live in §6.

1. Problem

Vanilla OPD (Qwen-3 / Thinking-Machines style, reverse-KL, k-estimator, optional PG wrap) is a strong post-training paradigm: dense per-token supervision from a frontier teacher, on-policy distribution matching at the student. We are running it via verl 0.8 from Qwen2.5-VL-72B → Qwen2.5-VL-7B on 5K video samples (1K TemporalBench + 4K LLaVA-Video-178K) to suppress video MLLM hallucination, mirroring the PaMi-VDPO setup but swapping VDPO → OPD.

The thing we want to attack: Vanilla OPD assumes the teacher's per-token distribution is uniformly reliable. For video MLLM hallucination this is false. The 72B teacher itself hallucinates -- it produces tokens that come from a language prior rather than from the visual evidence -- and Vanilla OPD's token-uniform KL distills those exact tokens into the student. The "dense token-level supervision" claim of OPD therefore collapses into "dense language-prior pass-through" on the very phenomenon (hallucination) we want to suppress.

We name this failure mode Hallucinated-Token Pass-through.

2. What evidence would prove (or kill) the problem

Three Phase 1 questions, all computable from teacher inference alone (no trained student needed):

F2a -- is the teacher's per-token grounding score g_t := log π_T(a_t | video, prefix) − log π_T(a_t | video⁻, prefix) bimodal across a realistic corpus? Single-peak around zero → the grounded vs. free-rider split we are building the method on does not exist.
F2b⁻ -- on the bottom-20% g_t tokens, does an external judge (GPT-4o) label ≥40% as hallucinated? If not, g_t is too noisy to be the supervision filter.
F2b⁺ -- on the top-20% g_t tokens, does the judge label ≤25% as hallucinated? If not, type-(e) visually-anchored-but-wrong hallucinations dominate and our CFG-style intervention will not catch them.

If any of these fails, stop, and re-open the challenge selection. Do not push the method.

3. Method: CD-OPD

We treat the teacher distribution as a sum of "what the model says because of the video" and "what the model says because of the language prior". To isolate the first, we use a classifier-free-guidance style construction at the distillation target:

target_logit_t(a) = (1 + α) · log π_T(a | v, prefix)
                  −      α  · log π_T(a | v⁻, prefix)

where v⁻ is a frame-count-preserving perturbation of the video that strips visual content (default: all-black frames). The student is then distilled toward this CFG-amplified target via the same k-estimator reverse-KL machinery as Vanilla OPD; for the PG-OPD path the per-token negative-loss becomes the reward as before.

Properties:

α = 0 ⇒ vanilla OPD exactly. Single hyperparameter, monotone ablation.
Token-level, no POS tagging, no external classifier, no detection threshold.
Connects to prior work: classifier-free guidance (Ho & Salimans 2022) and contrastive decoding (Li et al. 2023) both use the same (1+α)·p − α·q combination -- we are simply applying it at training time so the student inherits the language-prior-debiased behaviour at inference for free (amortisation: no extra forward at inference).
Cost: one extra teacher forward per training step (perturbed video, same sequence). Wall-time hit ≈ 50–70% of teacher time, which already dominates the step. Acceptable.

3.1 What `v⁻` to use

Constraint: same frame count as v (otherwise the number of vision tokens in the prompt changes, and prompt_logprobs no longer aligns with prompt_ids). Three perturbations supported in code:

name	semantics	recommended use
`black_frames`	replace every frame with all-zero RGB	default (Phase 2 main)
`shuffle`	random permutation of frame order	ablation: time-only break
`mean_frame`	every frame = temporal mean of video	ablation: motion-only break

black_frames is closest to "language-prior baseline" while keeping the teacher in distribution (still a video with the right number of frames).

4. Implementation in verl

All changes live in this fork; no upstream PR planned.

verl/workers/config/distillation.py — add cfg_enabled, cfg_alpha, cfg_perturbation to DistillationLossConfig.
verl/trainer/config/distillation/distillation.yaml — surface those knobs.
verl/experimental/teacher_loop/teacher_manager.py — add perturb_multi_modal_data() (operates on the videos list inside multi_modal_data) and _pad_teacher_logprobs() helper.
verl/experimental/agent_loop/agent_loop.py — when cfg_enabled, do a 2nd teacher forward on perturb_multi_modal_data(mmd, cfg_perturbation) and stash teacher_logprobs_perturbed through the same flow as teacher_logprobs (typed field on _InternalAgentLoopOutput, extra-field promotion in AgentLoopOutput.as_dict, batch concatenation in _postprocess).
verl/trainer/distillation/losses.py — in compute_distillation_loss_reverse_kl_estimator, when both teacher_logprobs and teacher_logprobs_perturbed are present, build (1+α) · log p − α · log p_perturbed as the ref_logprob for kl_penalty and log new metrics (distillation/cfg_grounding_mean, distillation/cfg_grounding_abs_mean, distillation/cfg_target_shift_mean).

Only the k-estimator loss path supports CD-OPD; the topk path is untouched and will raise nothing -- it just runs vanilla OPD even with cfg_enabled=True. That is acceptable because our running config uses loss_mode=k1.

5. Phase 1 → Phase 4 plan

phase	what	who	gate
0	Vanilla OPD continues (in-progress, finishes today ~18-19:30)	-	-
1	`scripts/compute_grounding.py` then `scripts/run_motivation_f2.py` → F2a/F2a2/F2b/F2c figures + GPT-4o judge round	local	F2a bimodal AND F2b⁻ ≥40% AND F2b⁺ ≤25%
2	Pilot CD-OPD run, α=1.0, 50 step subset, sanity check loss curves and `cfg_grounding_*` metrics	GPU	loss not NaN, grounding mean > 0
3	Full CD-OPD runs (α ∈ {0.5, 1.0, 1.5, 2.0}, perturbation ∈ {black_frames, shuffle}) + ablations	GPU	at least α=1.0 ≥ Vanilla OPD on VideoHallucer
4	Eval: VideoHallucer + EventHallusion (main); NextQA + VideoMME + TempCompass (regression)	GPU	hallucination ↑, general ≈

Ablation matrix (each row is one training run):

run	cfg_enabled	α	perturbation	purpose
A0	False	-	-	Vanilla OPD baseline (already running)
A1	True	1.0	black_frames	CD-OPD main
A2	True	0.5	black_frames	α-sweep low
A3	True	1.5	black_frames	α-sweep high
A4	True	2.0	black_frames	α-sweep over-correction check
A5	True	1.0	shuffle	perturbation: time-only ablation
A6	True	1.0	mean_frame	perturbation: motion-only ablation

6. What would change my mind

Phase 1 kill criteria fire → CD-OPD as designed cannot work. Pivot back to choosing a different challenge (most likely candidates: detection via teacher self-consistency, or attention-distillation based grounding).
F2a bimodal but α-sweep shows monotone decreasing eval at any α > 0 → CFG target is producing valid-shape distributions but the wrong direction of effect. Consider sign flip or contrastive triplet form (Variant B from the chat) instead of CFG-additive.
Vanilla OPD beats CD-OPD on general benchmarks by >2 pt → the CFG amplification is collateral-damaging the parts of teacher behaviour that are correct boilerplate. Mitigations: lower α; combine with a content-token mask so CFG only fires on noun/verb/number tokens.

7. Related work positioning

PaMi-VDPO (Wang et al., arXiv 2504.05810). Same dataset / hallucination niche, uses frame perturbation but for DPO negatives. We share the perturbation primitive; we use it as a CFG branch on top of OPD instead.
Video-OPD / TVDF (Li et al., arXiv 2602.02994). Same OPD-for-video niche, but their teacher reliability check uses ground-truth IoU annotations for sample-level filtering on closed-form TVG. We address open-ended captioning / QA where no per-sample GT exists, and our reliability signal is intrinsic (visual counterfactual) and token-level.
TCOD (Wang et al., arXiv 2604.24005). Different OPD failure mode (trajectory-level KL instability in multi-turn agents). They use a trajectory-depth curriculum; we use a per-token CFG target. We borrow the narrative structure (name a failure, propose a small fix) but not the method.
Classifier-free guidance (Ho & Salimans 2022) and contrastive decoding (Li et al., 2023). Same (1+α)·p − α·q construction, but they apply it at inference to a single model with two conditionings. We apply it at distillation so a student amortises it -- inference is one forward, no overhead.