docs/cd_opd_design.md

# CD-OPD: CFG-Distilled On-Policy Distillation for Video Hallucination Suppression

Status: design (Phase 1 motivation pending). Mirrors what we agreed on in
conversation; falsifiable kill criteria live in §6.

## 1. Problem

Vanilla OPD (Qwen-3 / Thinking-Machines style, reverse-KL, k-estimator, optional
PG wrap) is a strong post-training paradigm: dense per-token supervision from a
frontier teacher, on-policy distribution matching at the student. We are running
it via verl 0.8 from Qwen2.5-VL-72B → Qwen2.5-VL-7B on 5K video samples
(1K TemporalBench + 4K LLaVA-Video-178K) to suppress video MLLM hallucination,
mirroring the PaMi-VDPO setup but swapping VDPO → OPD.

The thing we want to attack: **Vanilla OPD assumes the teacher's per-token
distribution is uniformly reliable. For video MLLM hallucination this is false.**
The 72B teacher itself hallucinates -- it produces tokens that come from a
language prior rather than from the visual evidence -- and Vanilla OPD's
token-uniform KL distills those exact tokens into the student. The "dense
token-level supervision" claim of OPD therefore collapses into "dense
language-prior pass-through" on the very phenomenon (hallucination) we want to
suppress.

We name this failure mode **Hallucinated-Token Pass-through**.

## 2. What evidence would prove (or kill) the problem

Three Phase 1 questions, all computable from teacher inference alone (no trained
student needed):

- **F2a** -- is the teacher's per-token grounding score
  `g_t := log π_T(a_t | video, prefix) − log π_T(a_t | video⁻, prefix)`
  bimodal across a realistic corpus? Single-peak around zero → the grounded vs.
  free-rider split we are building the method on does not exist.
- **F2b⁻** -- on the bottom-20% g_t tokens, does an external judge (GPT-4o) label
  ≥40% as hallucinated? If not, `g_t` is too noisy to be the supervision filter.
- **F2b⁺** -- on the top-20% g_t tokens, does the judge label ≤25% as
  hallucinated? If not, type-(e) visually-anchored-but-wrong hallucinations
  dominate and our CFG-style intervention will not catch them.

If any of these fails, **stop**, and re-open the challenge selection. Do not
push the method.

## 3. Method: CD-OPD

We treat the teacher distribution as a sum of "what the model says because of
the video" and "what the model says because of the language prior". To isolate
the first, we use a classifier-free-guidance style construction at the
distillation target:

```
target_logit_t(a) = (1 + α) · log π_T(a | v, prefix)
                  −      α  · log π_T(a | v⁻, prefix)
```

where `v⁻` is a frame-count-preserving perturbation of the video that strips
visual content (default: all-black frames). The student is then distilled
toward this CFG-amplified target via the same k-estimator reverse-KL machinery
as Vanilla OPD; for the PG-OPD path the per-token negative-loss becomes the
reward as before.

Properties:

- **α = 0 ⇒ vanilla OPD exactly.** Single hyperparameter, monotone ablation.
- **Token-level**, no POS tagging, no external classifier, no detection threshold.
- Connects to prior work: classifier-free guidance (Ho & Salimans 2022) and
  contrastive decoding (Li et al. 2023) both use the same `(1+α)·p − α·q`
  combination -- we are simply applying it at training time so the student
  inherits the language-prior-debiased behaviour at inference for free
  (amortisation: no extra forward at inference).
- Cost: one extra teacher forward per training step (perturbed video, same
  sequence). Wall-time hit ≈ 50–70% of teacher time, which already dominates
  the step. Acceptable.

### 3.1 What `v⁻` to use

Constraint: same frame count as `v` (otherwise the number of vision tokens in
the prompt changes, and `prompt_logprobs` no longer aligns with `prompt_ids`).
Three perturbations supported in code:

| name           | semantics                              | recommended use         |
| -------------- | -------------------------------------- | ----------------------- |
| `black_frames` | replace every frame with all-zero RGB  | **default** (Phase 2 main)  |
| `shuffle`      | random permutation of frame order      | ablation: time-only break |
| `mean_frame`   | every frame = temporal mean of video   | ablation: motion-only break |

`black_frames` is closest to "language-prior baseline" while keeping the teacher
in distribution (still a video with the right number of frames).

## 4. Implementation in verl

All changes live in this fork; no upstream PR planned.

- `verl/workers/config/distillation.py` — add `cfg_enabled`, `cfg_alpha`,
  `cfg_perturbation` to `DistillationLossConfig`.
- `verl/trainer/config/distillation/distillation.yaml` — surface those knobs.
- `verl/experimental/teacher_loop/teacher_manager.py` — add
  `perturb_multi_modal_data()` (operates on the videos list inside `multi_modal_data`)
  and `_pad_teacher_logprobs()` helper.
- `verl/experimental/agent_loop/agent_loop.py` — when `cfg_enabled`, do a 2nd
  teacher forward on `perturb_multi_modal_data(mmd, cfg_perturbation)` and
  stash `teacher_logprobs_perturbed` through the same flow as `teacher_logprobs`
  (typed field on `_InternalAgentLoopOutput`, extra-field promotion in
  `AgentLoopOutput.as_dict`, batch concatenation in `_postprocess`).
- `verl/trainer/distillation/losses.py` — in
  `compute_distillation_loss_reverse_kl_estimator`, when both
  `teacher_logprobs` and `teacher_logprobs_perturbed` are present, build
  `(1+α) · log p − α · log p_perturbed` as the `ref_logprob` for `kl_penalty`
  and log new metrics (`distillation/cfg_grounding_mean`,
  `distillation/cfg_grounding_abs_mean`, `distillation/cfg_target_shift_mean`).

Only the k-estimator loss path supports CD-OPD; the topk path is untouched and
will raise nothing -- it just runs vanilla OPD even with `cfg_enabled=True`.
That is acceptable because our running config uses `loss_mode=k1`.

## 5. Phase 1 → Phase 4 plan

| phase   | what                                                              | who | gate |
|---------|-------------------------------------------------------------------|-----|------|
| **0**   | Vanilla OPD continues (in-progress, finishes today ~18-19:30)     | -   | -    |
| **1**   | `scripts/compute_grounding.py` then `scripts/run_motivation_f2.py`<br>→ F2a/F2a2/F2b/F2c figures + GPT-4o judge round | local | F2a bimodal AND F2b⁻ ≥40% AND F2b⁺ ≤25% |
| **2**   | Pilot CD-OPD run, α=1.0, 50 step subset, sanity check loss curves and `cfg_grounding_*` metrics | GPU | loss not NaN, grounding mean > 0 |
| **3**   | Full CD-OPD runs (α ∈ {0.5, 1.0, 1.5, 2.0}, perturbation ∈ {black_frames, shuffle}) + ablations | GPU | at least α=1.0 ≥ Vanilla OPD on VideoHallucer |
| **4**   | Eval: VideoHallucer + EventHallusion (main); NextQA + VideoMME + TempCompass (regression) | GPU | hallucination ↑, general ≈ |

Ablation matrix (each row is one training run):

| run | cfg_enabled | α | perturbation | purpose                                    |
| --- | ----------- | - | ------------ | ------------------------------------------ |
| A0  | False       | - | -            | Vanilla OPD baseline (already running)     |
| A1  | True        | 1.0 | black_frames | **CD-OPD main**                           |
| A2  | True        | 0.5 | black_frames | α-sweep low                                |
| A3  | True        | 1.5 | black_frames | α-sweep high                               |
| A4  | True        | 2.0 | black_frames | α-sweep over-correction check              |
| A5  | True        | 1.0 | shuffle      | perturbation: time-only ablation           |
| A6  | True        | 1.0 | mean_frame   | perturbation: motion-only ablation         |

## 6. What would change my mind

- **Phase 1 kill criteria fire** → CD-OPD as designed cannot work. Pivot
  back to choosing a different challenge (most likely candidates: detection
  via teacher self-consistency, or attention-distillation based grounding).
- **F2a bimodal but α-sweep shows monotone decreasing eval at any α > 0** →
  CFG target is producing valid-shape distributions but the wrong direction
  of effect. Consider sign flip or contrastive triplet form (Variant B from
  the chat) instead of CFG-additive.
- **Vanilla OPD beats CD-OPD on general benchmarks by >2 pt** → the CFG
  amplification is collateral-damaging the parts of teacher behaviour that
  are correct boilerplate. Mitigations: lower α; combine with a
  content-token mask so CFG only fires on noun/verb/number tokens.

## 7. Related work positioning

- **PaMi-VDPO** (Wang et al., arXiv 2504.05810). Same dataset / hallucination
  niche, uses frame perturbation but for DPO negatives. We share the
  perturbation primitive; we use it as a CFG branch on top of OPD instead.
- **Video-OPD / TVDF** (Li et al., arXiv 2602.02994). Same OPD-for-video
  niche, but their teacher reliability check uses **ground-truth IoU
  annotations** for sample-level filtering on closed-form TVG. We address
  open-ended captioning / QA where no per-sample GT exists, and our
  reliability signal is **intrinsic** (visual counterfactual) and
  **token-level**.
- **TCOD** (Wang et al., arXiv 2604.24005). Different OPD failure mode
  (trajectory-level KL instability in multi-turn agents). They use a
  trajectory-depth curriculum; we use a per-token CFG target. We borrow the
  narrative structure (name a failure, propose a small fix) but not the
  method.
- **Classifier-free guidance** (Ho & Salimans 2022) and **contrastive
  decoding** (Li et al., 2023). Same `(1+α)·p − α·q` construction, but they
  apply it at **inference** to a single model with two conditionings. We
  apply it at **distillation** so a student amortises it -- inference is
  one forward, no overhead.