File size: 4,822 Bytes
4357d9b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 | # Poetry Backend Compare
This directory prepares a protected local comparison workflow for ancient-poetry teaching videos.
## Standard Input
- `VGen/data/experiments/chusai_standard.json`
- Fixed `出塞` experiment block.
- Includes runtime defaults for a ~2 minute teaching video.
- Includes direct compare prompts for the same poem.
## Safety
- `VGen/scripts/utils/gpu_guard.sh`
- Reads live GPU memory and utilization from `nvidia-smi`.
- Only treats a GPU as idle when both conditions pass:
- `memory.used <= GPU_MEM_USED_LIMIT_MB`
- `utilization.gpu <= GPU_UTIL_LIMIT`
- Default launch mode is `DRY_RUN=1`, so the compare launcher only prepares normalized run directories and `launch.sh` files.
- Each generated `launch.sh` performs the idle-GPU check again at runtime before it starts any model job.
- Each generated `launch.sh` now checks the local Qwen3.5 service before it spends GPU time on planning or generation.
- If `QWEN35_AUTO_START=1` stays enabled, each generated `launch.sh` will auto-start the local Qwen3.5 service when `127.0.0.1:8002` is still down.
## Qwen Service
- `VGen/scripts/deploy/check_qwen35_service.sh`
- Health-checks the local OpenAI-compatible Qwen3.5 endpoint.
- Verifies that the expected served model id is visible from `/models`.
- `VGen/scripts/deploy/start_qwen35_27b_guarded.sh`
- Picks idle GPUs with the same guard rules before it starts the local vLLM service.
- Supports `FORCE_GPUS=6,7` when you want to pin the planner service manually.
- Supports `DETACH=1` for background startup.
- Supports `WAIT_FOR_READY=1` to block until `/models` responds successfully.
## Multi-backend Launcher
- `VGen/scripts/compare/run_poetry_backend_compare.sh`
- Prepares compare slots for:
- `wan_vace`
- `wan_i2v`
- `bernini_r`
- `hunyuanvideo`
- Planner variants:
- `agentloop`
- `baseline`
- Output layout:
- `VGen/outputs/compare_poetry/<compare_name>/manifest.json`
- `VGen/outputs/compare_poetry/<compare_name>/<planner>__<backend>/spec.json`
- `VGen/outputs/compare_poetry/<compare_name>/<planner>__<backend>/launch.sh`
## Compare Strategy
- Use the same poem `出塞` across all backends.
- `agentloop` and `baseline` keep the planning logic fixed within each compare branch.
- `wan_vace` and `wan_i2v` use the existing DiffSynth-based pipeline.
- `bernini_r` reuses the generated segment reference image(s) and calls local `Bernini-R-Diffusers` in single-GPU `r2v` mode.
- `hunyuanvideo` reuses the first generated segment reference image and calls the local official `HunyuanVideo-1.5/generate.py` wrapper in `i2v` mode.
## Typical Use
```bash
bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/scripts/compare/run_poetry_backend_compare.sh
```
To actually launch all prepared backends after confirming idle GPUs:
```bash
DRY_RUN=0 bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/scripts/compare/run_poetry_backend_compare.sh
```
To start the local Qwen3.5 planner service and pin it to chosen GPUs first:
```bash
DETACH=1 WAIT_FOR_READY=1 FORCE_GPUS=6,7 bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/scripts/deploy/start_qwen35_27b_guarded.sh
```
To verify that the planner service is healthy before launching compare jobs:
```bash
bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/scripts/deploy/check_qwen35_service.sh
```
To let the compare launcher auto-start the planner service on dedicated cards first:
```bash
QWEN35_FORCE_GPUS=0,1 DRY_RUN=0 bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/scripts/compare/run_poetry_backend_compare.sh
```
If you already know which free GPUs are safe and want to pin them manually at launch time:
```bash
FORCE_GPUS=6 bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/outputs/compare_poetry/<compare_name>/agentloop__bernini_r/launch.sh
```
For a 2-GPU Wan VACE run:
```bash
FORCE_GPUS=6,7 bash /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/outputs/compare_poetry/<compare_name>/agentloop__wan_vace/launch.sh
```
To summarize a prepared or completed compare run:
```bash
python3 /apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/ozhang/VGen/scripts/compare/summarize_compare.py
```
## Notes
- The current shell here may not expose `nvidia-smi`, so preparation can succeed while actual launch still needs a GPU-visible shell.
- `Bernini-R` uses the local `venv/bernini` Python by default.
- `HunyuanVideo` uses the shared official repo under `/apdcephfs_sgxy/share_304169095/hunyuan/juliayxzhu/dev/HunyuanVideo-1.5` but is launched only from wrappers inside `ozhang`.
- Shared `/Models` is read-only, so new model downloads must go under `ozhang/models` if needed later.
|