code/Q-Prefer/README.md

# Q-Prefer Reward Model: Portable Inference and Video-GRPO Integration

This repository contains a clean, portable inference package for the **Q-Prefer D2** video reward model. It is intended for collaborators who want to use the frozen reward model in downstream T2V/I2V generation, candidate reranking, or Flow-GRPO/Pref-GRPO-style post-training.

> **Research status.** This code reproduces the implementation used by the validated D2 checkpoint. The checkpoint was trained from Gemini preference labels. An increase in Q-Prefer reward is therefore an optimization diagnostic, not independent evidence of better human preference. Downstream claims should be supported by a held-out blind human study and independent metrics.

中文完整说明见[后半部分](#中文说明)。

## 1. What is released

The package provides:

- the exact Qwen3-VL reward architecture used by the D2 run;
- portable PEFT adapter loading with a pinned Qwen base revision;
- batched scoring for video paths and in-memory rollout tensors;
- T2V and I2V inputs;
- separate Visual Quality (VQ) and Text Alignment (TA) scores;
- scalar reward composition for GRPO groups;
- an optional tie-aware tournament reward;
- a single-GPU HTTP reward service for environment isolation;
- artifact preparation and checksum verification scripts.

Model weights are **not committed to regular Git**. The validated adapter is approximately 252 MiB and exceeds GitHub's normal 100 MiB file limit. Store it in the Hugging Face Hub, Git LFS, or project object storage.

## 2. Important model contract

The published D2 model is:

- base model: `Qwen/Qwen3-VL-4B-Instruct`;
- pinned base revision: `ebb281ec70b05090aa6165b016eac8ec08e71b17`;
- PEFT: LoRA, rank 64, alpha 128, dropout 0.05;
- input: 8 uniformly sampled frames;
- optional I2V reference image before the video;
- reward tokens: `<|VQ_reward|>`, `<|MQ_reward|>`, `<|TA_reward|>`;
- reward head: one `Linear(2560, 3, bias=False)` layer;
- training objective: Rao--Kupper Bradley--Terry-with-ties with `k=5`;
- supported outputs: VQ and TA.

The model internally has a third Motion Quality output for checkpoint compatibility, but MQ was not supervised in D2. This package deliberately does not expose MQ as a valid downstream reward.

Validated adapter SHA256:

```text
3f3788b942cb8e259582ee6cb2ac7c7bb2592a63e7183fe89c8434aef86476cf
```

## 3. Installation

Python 3.10 is recommended. Install the CUDA build of PyTorch appropriate for your cluster first, then install this package.

```bash
cd Q-Prefer

python -m venv .venv
source .venv/bin/activate

# Install the correct CUDA build for your cluster first.
pip install torch==2.8.0
pip install -e .
```

For the HTTP server:

```bash
pip install -e ".[server]"
```

The validated environment was:

```text
Python            3.10.20
PyTorch           2.8.0+cu128
Transformers      5.3.0
PEFT              0.18.1
qwen-vl-utils     0.0.14
decord            0.6.0
safetensors       0.7.0
```

## 4. Prepare the model artifact

If you have access to the original D2 run, create a portable model directory:

```bash
python scripts/prepare_adapter.py \
  --source /path/to/D2_broadnear \
  --output ./artifacts/qprefer-d2 \
  --special-embeddings-source /path/to/published/special_token_embeddings.safetensors
```

On an offline machine with a local Qwen snapshot, keep the public model id in
the release metadata and provide the local read path separately:

```bash
python scripts/prepare_adapter.py \
  --source /path/to/D2_broadnear \
  --output ./artifacts/qprefer-d2 \
  --base-model-source /path/to/local/Qwen3-VL-4B-Instruct-snapshot \
  --special-embeddings-source /path/to/published/special_token_embeddings.safetensors
```

The preparation script:

1. verifies the D2 adapter checksum;
2. copies only inference files, not optimizer or benchmark outputs;
3. replaces the machine-local base path with the public Qwen model id;
4. copies the exact three special-token embedding rows saved with the source run;
5. writes `artifact_manifest.json` with per-file checksums.

The historical D2 run did not optimize or store these input-embedding rows. The
published artifact freezes the exact compatibility rows used by the validated
inference release, but they cannot be recovered from the old adapter alone.
When packaging that historical directory, pass the published file explicitly:

```bash
python scripts/prepare_adapter.py \
  --source /path/to/D2_broadnear \
  --output ./artifacts/qprefer-d2-copy \
  --special-embeddings-source ./artifacts/qprefer-d2/special_token_embeddings.safetensors
```

Freshly resizing the base model is not an exact substitute and the script will
refuse to do it. New runs from this repository save the exact rows automatically.

`special_token_embeddings.safetensors` is required at inference time. The
loader fails fast if it is absent, which catches incomplete Hugging Face uploads
before a long generator-training job starts.

Verify a prepared artifact without loading the base model:

```bash
python scripts/verify_artifact.py ./artifacts/qprefer-d2
```

This checks the pinned model metadata, every manifest file's size and SHA256,
the PEFT configuration, the reward-head shape, and the special-token embedding shape.

Upload `artifacts/qprefer-d2/` to a model repository or shared object store. Do not commit the `.safetensors` file to ordinary Git. The adapter directory or its Hugging Face model id can then be passed to every example below. For a remote model, set `adapter_revision` to its immutable commit hash in reproducible runs.

### Upload to Hugging Face Hub

The prepared directory is the exact upload root. Do not upload the parent
`artifacts/` directory, and do not upload only `adapter_model.safetensors`.

```bash
pip install --upgrade huggingface_hub
hf auth login

# The upload command also creates the repository if it does not exist.
hf repos create qgfvadfuvads/Q-Prefer-D2 --private

hf upload qgfvadfuvads/Q-Prefer-D2 \
  ./artifacts/qprefer-d2 \
  . \
  --repo-type model \
  --commit-message "Upload validated Q-Prefer D2 artifact"
```

Start with a private repository, verify it, and make it public only after the
project license and model card are approved. If an upload is interrupted,
rerun the same `hf upload` command; completed content is reused.

Record the immutable Hub commit used by collaborators:

```bash
python -c "from huggingface_hub import model_info; print(model_info('qgfvadfuvads/Q-Prefer-D2').sha)"
```

Then verify the remote artifact through the public API:

```bash
python scripts/smoke_test.py \
  --adapter qgfvadfuvads/Q-Prefer-D2 \
  --adapter-revision IMMUTABLE_HF_COMMIT \
  --device cuda
```

The Hub repository stores the adapter artifact. The custom model architecture
remains versioned in this GitHub repository and is loaded by `QPreferScorer`.
The model card copied into the Hub artifact explains this dependency.

## 5. Python scoring API

### T2V files

```python
import torch

from qprefer_reward import QPreferConfig, QPreferScorer

scorer = QPreferScorer(
    QPreferConfig(
        adapter="qgfvadfuvads/Q-Prefer-D2",
        adapter_revision="YOUR_IMMUTABLE_MODEL_COMMIT",
        device="cuda",
        dtype=torch.bfloat16,
    )
)

scores = scorer.score_batch(
    videos=["video_a.mp4", "video_b.mp4"],
    prompts=[
        "A red fox runs through a snowy forest.",
        "A glass sculpture rotates under studio lighting.",
    ],
    batch_size=2,
)

print(scores.visual_quality)
print(scores.text_alignment)
```

### I2V files

```python
scores = scorer.score_batch(
    videos=["generated_a.mp4", "generated_b.mp4"],
    prompts=[prompt, prompt],
    reference_images=["condition.png", "condition.png"],
    batch_size=2,
)
```

### In-memory GRPO rollouts

Avoid encoding every rollout to MP4. The scorer accepts TCHW or THWC tensors and uniformly samples eight frames.

```python
# rollouts: [N, T, C, H, W], values in [-1, 1]
videos = [rollouts[index] for index in range(rollouts.shape[0])]

scores = scorer.score_batch(
    videos=videos,
    prompts=group_prompts,
    batch_size=2,
    tensor_value_range="minus_one_one",
)
```

Accepted tensor ranges are `minus_one_one`, `zero_one`, `zero_255`, and `auto`. For training, explicitly specify the generator/VAE output range instead of relying on automatic detection.

## 6. Scalar rewards for GRPO

VQ and TA have different raw scales, so they should not be naively averaged. The default composer standardizes each dimension separately among rollouts generated for the same prompt and then combines them:

```python
from qprefer_reward import RewardComposer

composer = RewardComposer(
    visual_weight=0.5,
    alignment_weight=0.5,
    normalization="per_prompt",
    std_floor=0.1,
    clip=5.0,
)

rewards = composer(scores, group_prompts)
```

Each prompt must have at least two candidates. `std_floor` prevents tiny differences among nearly identical videos from being amplified into large advantages. Treat it as a development hyperparameter and freeze it before the final evaluation.

For an explicitly tie-aware group tournament:

```python
from qprefer_reward import pairwise_tournament_rewards

rewards = pairwise_tournament_rewards(
    scores,
    group_prompts,
    visual_tie_threshold=VQ_THRESHOLD,
    alignment_tie_threshold=TA_THRESHOLD,
)
```

Tie thresholds must be selected on a separate development set. Never optimize them on the final human test set.

`examples/grpo_reward_adapter.py` contains a framework-neutral callable. Instantiate it once per reward worker; do not reload the 4B model inside the per-step reward function.

## 7. Flow-GRPO / Pref-GRPO integration

Recommended integration pattern:

```text
Wan rollout tensor
    -> select 8 frames in memory
    -> Q-Prefer batch scoring on a frozen reward worker
    -> VQ and TA per candidate
    -> per-prompt scalar composition
    -> GRPO advantage computation
```

For Pref-GRPO/FastVideo-style code:

1. create one `QPreferGRPOReward` instance when the reward worker starts;
2. register it in the framework's reward dispatcher as `qprefer`;
3. pass the list of rollout tensors and repeated group prompts to the callable;
4. return one CPU float reward per rollout;
5. reserve a separate GPU or process for Q-Prefer when generator and reward dependencies conflict.

Start with Wan2.1-T2V-1.3B LoRA for an end-to-end smoke test. After the pipeline is stable, use the same interface for Wan2.2 if compute permits. Keep the reward model frozen in all downstream experiments.

## 8. HTTP reward server

The HTTP server is useful when the Wan trainer and Q-Prefer require different environments. It accepts paths on a filesystem shared by the trainer and reward worker.

```bash
export QPREFER_API_KEY="replace-with-a-random-secret"

qprefer-serve \
  --adapter qgfvadfuvads/Q-Prefer-D2 \
  --adapter-revision YOUR_IMMUTABLE_MODEL_COMMIT \
  --device cuda \
  --host 127.0.0.1 \
  --port 18080
```

Client example:

```python
from qprefer_reward.client import QPreferHTTPClient

client = QPreferHTTPClient(
    "http://127.0.0.1:18080",
    api_key="replace-with-a-random-secret",
)
scores = client.score_paths([video_path], [prompt])
```

The path server is intended for a trusted cluster. It reads caller-provided paths and must not be exposed to the public internet. Start one server process per GPU; do not use multiple Uvicorn workers on the same GPU.

For high-throughput online training, prefer the in-process tensor API or a Ray/CUDA IPC worker. The HTTP path API is primarily a robust environment-isolation fallback.

## 9. Reproducibility and evaluation requirements

Before a full downstream run:

- verify the adapter SHA256;
- run a frozen-video smoke test and record its VQ/TA outputs;
- log base revision, adapter revision, package commit, prompt, frame count, pixel limits, and reward composition;
- monitor VQ and TA separately, not only their scalar sum;
- check diversity and temporal/physical quality for reward hacking;
- keep a base-model branch with identical prompts, seeds, and sampling settings.

For the paper, compare at least:

1. the frozen base generator;
2. the same generator post-trained with Q-Prefer;
3. preferably the same generator post-trained with a prior reward under the same budget.

The primary downstream evidence should be a blind A/B/Same human study with multiple annotators, plus independent automatic metrics. Q-Prefer's own reward curve should be reported only as an optimization diagnostic.

## 10. Repository layout

```text
Q-Prefer/
├── src/qprefer_reward/
│   ├── modeling_qprefer.py   # exact D2 architecture
│   ├── processing.py         # path/tensor video preprocessing
│   ├── scorer.py             # batched public API
│   ├── reward.py             # scalar and tie-aware GRPO rewards
│   ├── server.py             # optional FastAPI service
│   ├── client.py
│   └── training/             # data, loss, collator, and Trainer
├── examples/
│   ├── score_videos.py
│   ├── grpo_reward_adapter.py
│   └── http_client.py
├── scripts/
│   ├── prepare_adapter.py
│   ├── verify_artifact.py
│   ├── smoke_test.py
│   ├── validate_training_manifest.py
│   └── train_d2.py
├── training/
│   ├── configs/              # exact DeepSpeed configuration
│   ├── examples/             # portable manifest schema
│   └── provenance/           # immutable D2 contract and smoke record
├── validation/
│   ├── metrics.py
│   ├── run_local_benchmark.py
│   ├── run_ood_benchmark.py
│   ├── check_paper_numbers.py
│   ├── d2_parity_summary.json
│   └── release_validation_summary.json
└── tests/
```

## 11. Tests

Lightweight tests do not require the 4B model:

```bash
PYTHONPATH=src python -m unittest discover -s tests -v
python -m compileall -q src examples scripts tests validation
```

Full GPU smoke test:

```bash
python examples/score_videos.py \
  --adapter ./artifacts/qprefer-d2 \
  --video /path/to/frozen_test.mp4 \
  --prompt "A red fox runs through a snowy forest."
```

A media-free synthetic forward test is also available:

```bash
python scripts/smoke_test.py --adapter ./artifacts/qprefer-d2 --device cuda
```

### Validation against the original D2 pipeline

Before creating this release, we ran one real T2V pair and one real I2V pair
through both preprocessing implementations. Every generated tensor was
element-wise identical, including video pixels, reference-image pixels, token
ids, attention masks, and visual grids; all floating-point preprocessing
differences were exactly zero.

We then ran fresh CUDA inference on an RTX 6000 Ada with the validated runtime
(PyTorch 2.8.0+cu128, Transformers 5.3.0, and PEFT 0.18.1). The first frozen T2V
and I2V pair checks matched the legacy D2 GPU margins exactly.

The final release validation re-ran every locally available example rather than
recomputing metrics from cached predictions: 275 T2V pairs, 274 I2V pairs, 294
near pairs, the 210-pair A/B/Same human subset, the legacy 300-pair forced-choice
human subset, 4,691 unique VideoGen-RewardBench videos, and all 500
VideoScore-Bench-v2 videos. All runs completed without inference errors. Every
pair margin and every OOD raw VQ/TA score compared with the legacy D2 outputs was
bit-exact (maximum absolute difference `0.0`).

Fresh inference reproduces the paper's main `0.540/0.731` macro, near-pair
`0.659/0.640` macro, human A/B/Same `0.583/0.633` macro, VideoGen-RewardBench
VQ `47.5/63.7`, TA `43.1/58.3`, Overall `51.5/61.9`, and
VideoScore-Bench-v2 PLCC `49.1/24.1/36.6` (VQ/TA/average). Values separated by
`/` are with ties / without ties where applicable.

`validation/d2_parity_summary.json` records the initial tensor and frozen-pair
check. `validation/release_validation_summary.json` is the compact, path-free
full-run record. With access to the local benchmark media, regenerate the latter
after running the two validation runners:

```bash
python validation/check_paper_numbers.py
```

Raw media-derived predictions are intentionally ignored by Git. Re-run full
validation whenever CUDA, PyTorch, Transformers, PEFT, qwen-vl-utils, decord, or
the attention implementation changes.

## 12. Training from zero

The reward-model training release is now self-contained except for the dataset,
base model, and CUDA environment. It includes manifest validation, the exact D2
pair processor, Rao--Kupper loss, language-only LoRA configuration, DeepSpeed
ZeRO-2 config, provenance contract, and formal/smoke launchers.

```bash
pip install -e ".[training]"

python scripts/validate_training_manifest.py /data/D2.json \
  --media-root /data/qprefer-media --check-media

python scripts/train_d2.py \
  --manifest /data/D2.json \
  --media-root /data/qprefer-media \
  --output outputs/D2_reproduction \
  --gpus 0,1,2,3
```

See [`training/README.md`](training/README.md) for the two-step GPU smoke test,
manifest schema, packaging command, exact hyperparameters, and the one historical
bitwise-reproducibility boundary. The immutable audited recipe is in
[`training/provenance/d2_training_contract.json`](training/provenance/d2_training_contract.json).

## 13. Limitations

- D2 was trained from Gemini-generated preference labels.
- Existing human and OOD evaluations show limited transfer; downstream human validation is essential.
- The reported tie-aware benchmark metric maximizes a margin threshold on each evaluated label set; it is an in-split comparison metric, not a deployable calibrated threshold.
- MQ is not a supported reward.
- The default D2 I2V prompt receives the reference image but does not explicitly spell out reference-preservation criteria.
- Eight-frame scoring may miss short or localized temporal failures.
- Any online RL run can exploit reward-model blind spots; independent evaluation and qualitative inspection are mandatory.

## 14. License and citation

Add the project's approved license before making the repository public. The base model and dependencies retain their respective licenses. Add the final paper citation after the title and author list are frozen.

---

# 中文说明

本仓库提供 **Q-Prefer D2** 视频奖励模型的清理版、可移植推理代码，面向需要将冻结 reward model 接入 T2V/I2V 生成、候选重排或 Flow-GRPO/Pref-GRPO 后训练的合作者。

> **研究状态说明：** 本代码复现已经验证过的 D2 checkpoint 实现。该模型使用 Gemini preference labels 训练，因此 Q-Prefer reward 上升只能说明优化器成功提高了这个代理目标，不能独立证明人类更喜欢生成结果。论文中的下游结论必须由全新盲测人评和独立指标支撑。

## 1. 发布内容

本包包含：

- D2 实际使用的 Qwen3-VL reward architecture；
- 固定 Qwen base revision 的可移植 PEFT adapter 加载；
- 支持视频路径和内存 rollout tensor 的批量打分；
- T2V 与 I2V 输入；
- 分开的 Visual Quality（VQ）与 Text Alignment（TA）输出；
- 面向 GRPO group 的标量 reward 合成；
- 可选的 tie-aware tournament reward；
- 用于隔离训练环境的单 GPU HTTP 服务；
- checkpoint 整理和 checksum 验证脚本。

模型权重不会放进普通 Git。验证过的 adapter 约 252 MiB，超过 GitHub 普通文件 100 MiB 限制，应存放在 Hugging Face Hub、Git LFS 或项目对象存储中。

## 2. 模型接口约定

D2 模型配置如下：

- Base：`Qwen/Qwen3-VL-4B-Instruct`；
- Base revision：`ebb281ec70b05090aa6165b016eac8ec08e71b17`；
- LoRA：rank 64、alpha 128、dropout 0.05；
- 每个视频均匀采样 8 帧；
- I2V 可以在视频前额外输入 reference image；
- 三个 token：`<|VQ_reward|>`、`<|MQ_reward|>`、`<|TA_reward|>`；
- Reward head：`Linear(2560, 3, bias=False)`；
- 训练目标：`k=5` 的 Rao--Kupper Bradley--Terry-with-ties；
- 对外支持 VQ 和 TA。

checkpoint 内部保留 MQ 输出是为了兼容原始权重，但 D2 训练没有监督 MQ，因此本包不会把 MQ 暴露为可用 reward。

验证过的 adapter SHA256：

```text
3f3788b942cb8e259582ee6cb2ac7c7bb2592a63e7183fe89c8434aef86476cf
```

## 3. 安装

推荐 Python 3.10。先安装与集群 CUDA 匹配的 PyTorch，再安装本包：

```bash
cd Q-Prefer

python -m venv .venv
source .venv/bin/activate

pip install torch==2.8.0
pip install -e .
```

如果需要 HTTP server：

```bash
pip install -e ".[server]"
```

已经验证的版本为 Python 3.10.20、PyTorch 2.8.0+cu128、Transformers 5.3.0、PEFT 0.18.1、qwen-vl-utils 0.0.14、decord 0.6.0 和 safetensors 0.7.0。

## 4. 整理模型文件

从原始 D2 目录生成可交付 artifact：

```bash
python scripts/prepare_adapter.py \
  --source /path/to/D2_broadnear \
  --output ./artifacts/qprefer-d2 \
  --special-embeddings-source /path/to/published/special_token_embeddings.safetensors
```

离线机器可以额外传入本地 snapshot；发布 metadata 仍会保留公开 Qwen model id：

```bash
python scripts/prepare_adapter.py \
  --source /path/to/D2_broadnear \
  --output ./artifacts/qprefer-d2 \
  --base-model-source /path/to/local/Qwen3-VL-4B-Instruct-snapshot \
  --special-embeddings-source /path/to/published/special_token_embeddings.safetensors
```

脚本会：

1. 验证 D2 adapter checksum；
2. 只复制推理所需文件，不复制 optimizer 和 benchmark 结果；
3. 把本机 base 路径替换成公开 Qwen model id；
4. 复制源训练目录保存的三个精确 reward-token embedding；
5. 生成带逐文件 checksum 的 `artifact_manifest.json`。

历史 D2 训练没有优化或保存这三个输入 embedding。发布 artifact 固定了经过完整
推理验证的 compatibility rows，但仅靠旧 adapter 无法反推出它们。整理历史目录时
必须用 `--special-embeddings-source` 指向已发布 artifact 中的精确文件。重新 resize
base model 不是精确替代，脚本会直接拒绝；本仓库启动的新训练会自动保存精确行。

推理时强制要求存在 `special_token_embeddings.safetensors`。如果 Hugging Face
上传不完整，loader 会在启动阶段直接报错，避免长时间生成模型训练开始后才发现问题。

不加载 4B base model 即可检查 artifact：

```bash
python scripts/verify_artifact.py ./artifacts/qprefer-d2
```

该命令会检查固定的模型 metadata、manifest 中每个文件的大小与 SHA256、PEFT 配置、reward head shape，以及特殊 token embedding shape。

随后把整个 `artifacts/qprefer-d2/` 上传到模型仓库或对象存储。不要把 `.safetensors` 直接提交到普通 Git。使用远程模型时，正式复现实验应把 `adapter_revision` 固定到不可变的 commit hash。

### 上传到 Hugging Face Hub

需要上传的根目录是整理后的 `artifacts/qprefer-d2/`，不要上传它的父目录
`artifacts/`，也不能只上传 `adapter_model.safetensors`：

```bash
pip install --upgrade huggingface_hub
hf auth login

hf repos create qgfvadfuvads/Q-Prefer-D2 --private

hf upload qgfvadfuvads/Q-Prefer-D2 \
  ./artifacts/qprefer-d2 \
  . \
  --repo-type model \
  --commit-message "Upload validated Q-Prefer D2 artifact"
```

建议先创建 private repository，完成验证并批准项目 license/model card 后再公开。
上传中断时重新执行相同命令即可复用已上传的内容。

记录合作者实验应固定的不可变 Hub commit：

```bash
python -c "from huggingface_hub import model_info; print(model_info('qgfvadfuvads/Q-Prefer-D2').sha)"
```

然后通过正式调用接口验证远程模型：

```bash
python scripts/smoke_test.py \
  --adapter qgfvadfuvads/Q-Prefer-D2 \
  --adapter-revision 不可变的HF_COMMIT \
  --device cuda
```

Hugging Face 保存 adapter artifact；自定义模型架构保存在 GitHub 代码中并由
`QPreferScorer` 加载。artifact 内复制的双语 model card 会明确说明这一依赖。

## 5. Python 打分

T2V：

```python
import torch
from qprefer_reward import QPreferConfig, QPreferScorer

scorer = QPreferScorer(
    QPreferConfig(
        adapter="qgfvadfuvads/Q-Prefer-D2",
        adapter_revision="不可变的模型commit哈希",
        device="cuda",
        dtype=torch.bfloat16,
    )
)

scores = scorer.score_batch(
    videos=["video_a.mp4", "video_b.mp4"],
    prompts=[prompt_a, prompt_b],
    batch_size=2,
)

print(scores.visual_quality)
print(scores.text_alignment)
```

I2V 只需额外传入与视频一一对应的 reference images：

```python
scores = scorer.score_batch(
    videos=["generated_a.mp4", "generated_b.mp4"],
    prompts=[prompt, prompt],
    reference_images=["condition.png", "condition.png"],
)
```

在线 GRPO 不要把每个 rollout 编码成 MP4。本包直接接受 TCHW/THWC tensor，并在内存中均匀采 8 帧：

```python
videos = [rollouts[i] for i in range(rollouts.shape[0])]
scores = scorer.score_batch(
    videos,
    group_prompts,
    batch_size=2,
    tensor_value_range="minus_one_one",
)
```

支持 `minus_one_one`、`zero_one`、`zero_255` 和 `auto`。正式训练应明确写出 VAE/生成器的 tensor range，不建议依赖自动判断。

## 6. GRPO 标量 reward

VQ 和 TA 的原始尺度不同，不能简单平均。默认实现会在同一 prompt 的候选组内分别标准化两个维度，再按权重合并：

```python
from qprefer_reward import RewardComposer

composer = RewardComposer(
    visual_weight=0.5,
    alignment_weight=0.5,
    normalization="per_prompt",
    std_floor=0.1,
    clip=5.0,
)
rewards = composer(scores, group_prompts)
```

每个 prompt 至少需要两个候选。`std_floor` 用来防止 near-identical 视频之间的微小噪声被放大成很大的 advantage；它应只在开发集上确定，并在最终评测前冻结。

如果希望显式使用 tie-aware group tournament，可以调用 `pairwise_tournament_rewards()`。VQ/TA 的 tie threshold 必须在独立开发集上确定，不能在最终人评测试集上调参。

`examples/grpo_reward_adapter.py` 给出了与具体训练框架无关的 callable。每个 reward worker 只初始化一次，不能在每个训练 step 里重新加载 4B 模型。

## 7. 接入 Flow-GRPO / Pref-GRPO

建议数据流：

```text
Wan rollout tensor
    → 内存采样8帧
    → 冻结的Q-Prefer worker批量打分
    → 每个候选的VQ与TA
    → 同prompt组内合成标量reward
    → GRPO advantage
```

接入 Pref-GRPO/FastVideo 时：

1. Reward worker 启动时创建一个 `QPreferGRPOReward`；
2. 在框架的 reward dispatcher 中注册为 `qprefer`；
3. 把 rollout tensor 列表和按 group 重复的 prompts 传入；
4. 返回每个 rollout 对应的 CPU float reward；
5. 如果 Wan 与 Qwen 的依赖冲突，给 Q-Prefer 单独的进程或 GPU。

建议先用 Wan2.1-T2V-1.3B LoRA 完成端到端 smoke test，稳定以后再把相同接口迁到 Wan2.2。所有下游实验中 reward model 都必须冻结。

## 8. HTTP 服务

当 Wan trainer 与 Q-Prefer 必须使用不同环境时，可以启动基于共享文件系统路径的 HTTP server：

```bash
export QPREFER_API_KEY="替换成随机密钥"

qprefer-serve \
  --adapter qgfvadfuvads/Q-Prefer-D2 \
  --adapter-revision 不可变的模型commit哈希 \
  --device cuda \
  --host 127.0.0.1 \
  --port 18080
```

客户端见 `examples/http_client.py`。

该 server 会读取调用方提供的路径，只适合可信集群，禁止直接暴露到公网。每张 GPU 启动一个单 worker 服务，不要让多个 Uvicorn workers 抢同一张 GPU。高吞吐在线训练仍优先使用内存 tensor API 或 Ray/CUDA IPC；HTTP path API 主要用于稳定隔离环境。

## 9. 复现与论文评测要求

正式训练前应：

- 检查 adapter SHA256；
- 用固定测试视频做 smoke test，并保存其 VQ/TA 输出；
- 记录 base revision、adapter revision、代码 commit、prompt、帧数、像素上限和 reward 合成方式；
- 分别监控 VQ 与 TA，不能只看相加后的曲线；
- 监控多样性、时序和物理质量，防止 reward hacking；
- 保留完全相同 prompts、seeds 和采样参数的 base branch。

论文至少比较：

1. 原始 base generator；
2. Q-Prefer 后训练的相同 generator；
3. 最好增加相同预算下使用已有 reward 的后训练对照。

主要证据必须是多标注者的盲测 A/B/Same 人评和独立自动指标。Q-Prefer 自己的 reward curve 只能作为训练诊断。

## 10. 测试

无需 4B 模型的轻量测试：

```bash
PYTHONPATH=src python -m unittest discover -s tests -v
python -m compileall -q src examples scripts tests validation
```

GPU 完整 smoke test：

```bash
python examples/score_videos.py \
  --adapter ./artifacts/qprefer-d2 \
  --video /path/to/frozen_test.mp4 \
  --prompt "A red fox runs through a snowy forest."
```

也可以运行不依赖外部视频文件的合成前向测试：

```bash
python scripts/smoke_test.py --adapter ./artifacts/qprefer-d2 --device cuda
```

### 与原始 D2 流程的验证

创建该发布目录前，我们分别选择了一组真实 T2V pair 和一组真实 I2V pair，
对原始 D2 预处理与发布版预处理做逐 tensor 对拍。视频像素、参考图像像素、
token ids、attention mask 和 visual grid 全部逐元素相等，所有浮点预处理张量的
最大差值均为 0。

随后我们在 RTX 6000 Ada 上使用验证环境（PyTorch 2.8.0+cu128、Transformers
5.3.0、PEFT 0.18.1）重新执行真实 CUDA inference。最初两组固定 T2V/I2V pair
的 fresh VQ/TA margin 与历史 D2 GPU margin 精确相等。

最终发布验证不是读取缓存预测重算指标，而是把本地可用数据全部重新前向：275
条 T2V pair、274 条 I2V pair、294 条 near pair、210 条 A/B/Same 人评、旧版
300 条 forced-choice 人评、VideoGen-RewardBench 的 4,691 个唯一视频，以及
VideoScore-Bench-v2 的 500 个视频。全部 inference 零错误；所有 pair margin
以及 OOD 的逐视频 VQ/TA raw score 与历史 D2 输出逐项精确一致，最大绝对差为
`0.0`。

Fresh inference 复现了论文中的主 benchmark macro `0.540/0.731`、near-pair
macro `0.659/0.640`、A/B/Same 人评 macro `0.583/0.633`、
VideoGen-RewardBench 的 VQ `47.5/63.7`、TA `43.1/58.3`、Overall
`51.5/61.9`，以及 VideoScore-Bench-v2 PLCC `49.1/24.1/36.6`
（VQ/TA/平均）。适用时，斜杠前后依次为含 tie / 不含 tie。

`validation/d2_parity_summary.json` 保存最初的逐 tensor 与固定 pair 检查；
`validation/release_validation_summary.json` 保存去除本机路径后的完整验证摘要。
拥有本地 benchmark media 时，在两个 validation runner 完成后运行：

```bash
python validation/check_paper_numbers.py
```

逐视频预测会被 Git 忽略，不会随开源仓库发布。CUDA、PyTorch、Transformers、
PEFT、qwen-vl-utils、decord 或 attention 实现变化后，应重新运行完整验证。

## 11. 从零训练 reward model

除数据集、base model 和 CUDA 环境外，reward-model 训练链路现已自包含：包括
manifest 校验、D2 精确 pair 预处理、Rao--Kupper loss、语言侧 LoRA、DeepSpeed
ZeRO-2、provenance contract，以及正式/冒烟启动器。

```bash
pip install -e ".[training]"

python scripts/validate_training_manifest.py /data/D2.json \
  --media-root /data/qprefer-media --check-media

python scripts/train_d2.py \
  --manifest /data/D2.json \
  --media-root /data/qprefer-media \
  --output outputs/D2_reproduction \
  --gpus 0,1,2,3
```

两步真实 GPU smoke test、manifest schema、训练后整理命令、全部超参数和历史逐 bit
复现边界见 [`training/README.md`](training/README.md)，审计后的固定配方见
[`training/provenance/d2_training_contract.json`](training/provenance/d2_training_contract.json)。

## 12. 局限性

- D2 使用 Gemini preference labels 训练；
- 现有人评和 OOD 结果显示迁移有限，因此下游独立人评是必需的；
- 论文的 tie-aware benchmark 指标会在每个被评 label set 上搜索最优 margin threshold；它是 split 内比较指标，不能直接当作部署阈值；
- MQ 不是受支持的 reward；
- D2 默认 I2V prompt 会输入 reference image，但没有明确写出 reference preservation 规则；
- 8 帧评分可能漏掉短暂或局部时序错误；
- 在线 RL 可能利用 reward blind spots，必须进行独立评测与定性检查。

## 13. License 与引用

公开仓库前请加入项目正式批准的 license。Base model 与依赖分别遵循其自身 license。论文标题和作者列表冻结后，再补充最终 BibTeX。