scripts/compare/README.md

# Poetry Backend Compare

This directory prepares a protected local comparison workflow for ancient-poetry teaching videos.

## Standard Input

- `VGen/data/experiments/chusai_standard.json`
  - Fixed `出塞` experiment block.
  - Includes runtime defaults for a ~2 minute teaching video.
  - Includes direct compare prompts for the same poem.

## Safety

- `VGen/scripts/utils/gpu_guard.sh`
  - Reads live GPU memory and utilization from `nvidia-smi`.
  - Only treats a GPU as idle when both conditions pass:
    - `memory.used <= GPU_MEM_USED_LIMIT_MB`
    - `utilization.gpu <= GPU_UTIL_LIMIT`
- Default launch mode is `DRY_RUN=1`, so the compare launcher only prepares normalized run directories and `launch.sh` files.
- Each generated `launch.sh` performs the idle-GPU check again at runtime before it starts any model job.
- Each generated `launch.sh` now checks the local Qwen3.5 service before it spends GPU time on planning or generation.
- If `QWEN35_AUTO_START=1` stays enabled, each generated `launch.sh` will auto-start the local Qwen3.5 service when `127.0.0.1:8002` is still down.

## Qwen Service

- `VGen/scripts/deploy/check_qwen35_service.sh`
  - Health-checks the local OpenAI-compatible Qwen3.5 endpoint.
  - Verifies that the expected served model id is visible from `/models`.
- `VGen/scripts/deploy/start_qwen35_27b_guarded.sh`
  - Picks idle GPUs with the same guard rules before it starts the local vLLM service.
  - Supports `FORCE_GPUS=6,7` when you want to pin the planner service manually.
  - Supports `DETACH=1` for background startup.
  - Supports `WAIT_FOR_READY=1` to block until `/models` responds successfully.

## Multi-backend Launcher

- `VGen/scripts/compare/run_poetry_backend_compare.sh`
  - Prepares compare slots for:
    - `wan_vace`
    - `wan_i2v`
    - `bernini_r`
    - `hunyuanvideo`
  - Planner variants:
    - `agentloop`
    - `baseline`
  - Output layout:
    - `VGen/outputs/compare_poetry/<compare_name>/manifest.json`
    - `VGen/outputs/compare_poetry/<compare_name>/<planner>__<backend>/spec.json`
    - `VGen/outputs/compare_poetry/<compare_name>/<planner>__<backend>/launch.sh`

## Compare Strategy

- Use the same poem `出塞` across all backends.
- `agentloop` and `baseline` keep the planning logic fixed within each compare branch.
- `wan_vace` and `wan_i2v` use the existing DiffSynth-based pipeline.
- `bernini_r` reuses the generated segment reference image(s) and calls local `Bernini-R-Diffusers` in single-GPU `r2v` mode.
- `hunyuanvideo` reuses the first generated segment reference image and calls the local official `HunyuanVideo-1.5/generate.py` wrapper in `i2v` mode.

## Typical Use

```bash
bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/scripts/compare/run_poetry_backend_compare.sh
```

To actually launch all prepared backends after confirming idle GPUs:

```bash
DRY_RUN=0 bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/scripts/compare/run_poetry_backend_compare.sh
```

To start the local Qwen3.5 planner service and pin it to chosen GPUs first:

```bash
DETACH=1 WAIT_FOR_READY=1 FORCE_GPUS=6,7 bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/scripts/deploy/start_qwen35_27b_guarded.sh
```

To verify that the planner service is healthy before launching compare jobs:

```bash
bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/scripts/deploy/check_qwen35_service.sh
```

To let the compare launcher auto-start the planner service on dedicated cards first:

```bash
QWEN35_FORCE_GPUS=0,1 DRY_RUN=0 bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/scripts/compare/run_poetry_backend_compare.sh
```

If you already know which free GPUs are safe and want to pin them manually at launch time:

```bash
FORCE_GPUS=6 bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/outputs/compare_poetry/<compare_name>/agentloop__bernini_r/launch.sh
```

For a 2-GPU Wan VACE run:

```bash
FORCE_GPUS=6,7 bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/outputs/compare_poetry/<compare_name>/agentloop__wan_vace/launch.sh
```

To summarize a prepared or completed compare run:

```bash
python3 /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/scripts/compare/summarize_compare.py
```

## Notes

- The current shell here may not expose `nvidia-smi`, so preparation can succeed while actual launch still needs a GPU-visible shell.
- `Bernini-R` uses the local `venv/bernini` Python by default.
- `HunyuanVideo` uses the shared official repo under `/apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/HunyuanVideo-1.5` but is launched only from wrappers inside `ozhang`.
- Shared `/Models` is read-only, so new model downloads must go under `ozhang/models` if needed later.