diff --git a/code/orarl.egg-info/PKG-INFO b/code/orarl.egg-info/PKG-INFO deleted file mode 100644 index 46b3ccb3ed12bfa6b4f1556877e844f0254836b1..0000000000000000000000000000000000000000 --- a/code/orarl.egg-info/PKG-INFO +++ /dev/null @@ -1,230 +0,0 @@ -Metadata-Version: 2.4 -Name: orarl -Version: 0.1.0 -Summary: Annotations as rollouts for unified video MLLM reinforcement learning -Author: Yunheng Li, Guohong Mu, Hao Li, Shengsheng Qian, Dingwen Zhang, Qibin Hou, Ming-Ming Cheng -License-Expression: Apache-2.0 -Project-URL: Homepage, https://orarl.github.io/ -Project-URL: Documentation, https://orarl.github.io/ -Keywords: reinforcement-learning,video-understanding,multimodal,grpo -Classifier: Development Status :: 3 - Alpha -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3.10 -Classifier: Programming Language :: Python :: 3.11 -Classifier: Programming Language :: Python :: 3.12 -Requires-Python: >=3.10 -Description-Content-Type: text/markdown -License-File: LICENSE -License-File: NOTICE -Requires-Dist: numpy -Requires-Dist: PyYAML -Requires-Dist: torch -Provides-Extra: test -Requires-Dist: pytest; extra == "test" -Provides-Extra: lint -Requires-Dist: ruff; extra == "lint" -Provides-Extra: hf -Requires-Dist: huggingface_hub; extra == "hf" -Dynamic: license-file - -

简体中文

- -
- -# OraRL - -### Annotations as Rollouts - -**Efficient and scalable reinforcement learning for unified video MLLMs** - -Yunheng Li · Guohong Mu · Hao Li · Shengsheng Qian · Dingwen Zhang · -Qibin Hou · Ming-Ming Cheng - -

- 📄 Paper -   ·   - 🌐 Project Page -   ·   - 🤗 Models (4B / 9B) -

-

- ⚙️ Environment -   ·   - 🚀 Training -   ·   - 📊 Evaluation -   ·   - ⚖️ License -

- - - Animated OraRL method preview - - -**▶ Click the image to watch the 1:38 project overview.** - -
- -## Why OraRL - -- **Annotation-as-rollout:** annotations become reliable positive rollouts while - policy samples retain an on-policy baseline. -- **Seven task families:** one update rule covers temporal and spatial grounding, - segmentation, tracking, spatial-temporal grounding, video QA, and spatial - intelligence. -- **Efficient training (4B):** sign-balanced pruning delivers **1.48× faster - updates** (**92.5 → 62.4 s/step**) while reducing peak per-GPU memory from - **62.4 to 50.9 GB**. -- **Efficient inference:** on one H20 with vLLM in BF16, weight loading occupies - **8.6 GiB (4B)** and **17.6 GiB (9B)**. On ten-minute, 2-fps videos, - answer-only decoding cuts median post-TTFT latency from **4.78 s to 130 ms** - and total latency from **29.03 to 24.30 s**. -- **Multimodal veRL infrastructure:** a unified video contract carries cached - artifacts, raw paths, or inline frame tensors through vLLM rollouts and FSDP - updates, with decode-once frame reuse, temporal metadata, task-grouped - batching, asynchronous Ray rewards, and safe hybrid-engine cache handling. - -## OraRL in One Update - -

- Animated OraRL framework -

- -An OraRL update separates reliable annotation guidance from on-policy -normalization: - -1. **Build the group:** append one serialized annotation rollout to the policy - samples generated for the same prompt. -2. **Keep the baseline on-policy:** estimate the group baseline from policy - rewards only. -3. **Guide and select:** convert the annotation-policy reward gap into a - correction, then retain a sign-balanced subset for the update. - -This design uses task-native annotations directly and requires no -chain-of-thought supervision or decoding. - -## Video-ORA Results - -

- Video-ORA-9B results across seven task families -

- -### Dataset-Level Results - - - - - Dataset-level benchmark matrix comparing Video-ORA with multimodal baselines - - -Video-ORA-9B leads the matched seven-family comparison without CoT decoding. -Best and second-best values are highlighted per row; `†` denotes an -original-report value whose frame, prompt, split, or decoding settings may -differ. Averages require complete family coverage. - - - -### Model Scaling - -

- Animated Video-ORA model scaling from 0.8B to 9B -

- -### Data Scaling - -

- Animated OraRL data scaling and reward dynamics -

- -## Models - -| Model | Backbone | Released recipe | Weights | -| --- | --- | --- | --- | -| **Video-ORA-9B** | Qwen3.5-9B | `orarl_9b.yaml` | [Hugging Face](https://huggingface.co/OraRL/Video-ORA-9B) | -| **Video-ORA-4B** | Qwen3.5-4B | `orarl_4b.yaml` | Hugging Face (coming soon) | - -### vLLM Serving - -Both Video-ORA checkpoints load directly with **vLLM 0.19.1** for -OpenAI-compatible serving: - -```bash -MODEL=OraRL/Video-ORA-9B - -vllm serve "$MODEL" \ - --served-model-name Video-ORA-9B \ - --trust-remote-code \ - --dtype bfloat16 \ - --tensor-parallel-size 1 \ - --max-model-len 131072 \ - --limit-mm-per-prompt '{"image": 1, "video": 1}' -``` - -Set `--tensor-parallel-size` to the GPU count for multi-GPU deployment and -lower `--max-model-len` on smaller-memory devices. Use -`enable_thinking=false` in the chat template for answer-only inference. - -## Use OraRL - -The release is organized around three user-facing workflows: - -1. **[Environment](docs/environment.md):** install the pinned CUDA stack that - covers both the bundled trainer and the evaluators. -2. **[Training](docs/training.md):** prepare licensed local training data and - launch GRPO or OraRL on one or multiple nodes. -3. **[Evaluation](docs/evaluation.md):** download Video-ORA and OraRL-Data, - then run a smoke test or the complete paper suite. - -Training and evaluation are dry runs by default; inspect the resolved command -before adding `--run`. Checkpoints and evaluation media are hosted under the -[OraRL Hugging Face organization](https://huggingface.co/OraRL). - -## Acknowledgements - -OraRL is built on [veRL](https://github.com/volcengine/verl) — a -high-performance RL framework with HybridEngine. We thank its authors and -contributors for open-sourcing the training infrastructure. - -## License - -OraRL source is released under [Apache-2.0](LICENSE). Datasets, models, -benchmarks, and optional dependencies retain their original licenses; see -[NOTICE](NOTICE). - -## Citation - -If you find OraRL useful, please consider giving this repository a ⭐ and -citing our [paper](https://arxiv.org/abs/2608.20492). - -```bibtex -@article{li2026orarl, - title = {Annotations as Rollouts: Efficient and Scalable - Reinforcement Learning for Video MLLMs}, - author = {Li, Yunheng and Mu, Guohong and Li, Hao and - Qian, Shengsheng and Zhang, Dingwen and Hou, Qibin - and Cheng, Ming-Ming}, - journal = {arXiv preprint arXiv:2608.20492}, - year = {2026}, - url = {https://arxiv.org/abs/2608.20492} -} -``` diff --git a/code/orarl.egg-info/SOURCES.txt b/code/orarl.egg-info/SOURCES.txt deleted file mode 100644 index 109aa7dc0e190f0ad7152e1ed4dc70e643a673d0..0000000000000000000000000000000000000000 --- a/code/orarl.egg-info/SOURCES.txt +++ /dev/null @@ -1,196 +0,0 @@ -LICENSE -MANIFEST.in -NOTICE -README.md -environment.yml -pyproject.toml -requirements-cu129.txt -configs/data_sources.example.yaml -configs/grpo_4b.yaml -configs/grpo_9b.yaml -configs/orarl_4b.yaml -configs/orarl_9b.yaml -data/eval/README.md -data/eval/datasets.jsonl -docs/environment.md -docs/environment_zh.md -docs/evaluation.md -docs/evaluation_zh.md -docs/training.md -docs/training_zh.md -eval/README.md -eval/task/canonical_data.py -eval/task/eval.sh -eval/task/eval_image_mc_vllm.py -eval/task/eval_prompt.py -eval/task/eval_revsi_multinode.sh -eval/task/eval_vllm.py -eval/task/qwenvl_decord_patch.py -eval/task/mindcube/data_utils.py -eval/task/mmsi/eval_mmsi_transformers.py -eval/task/mmsi/run_eval_transformers.sh -eval/task/revsi/eval_revsi_vllm.py -eval/task/revsi/merge_multinode_shards.py -eval/task/revsi/run_eval_vllm.sh -eval/task/segmentation/eval_seg_vllm.py -eval/task/segmentation/post_sam2.py -eval/task/segmentation/run_eval_vllm.sh -eval/task/spatial_grounding/_grounding_utils.py -eval/task/spatial_grounding/eval_refcoco_vllm.py -eval/task/spatial_temporal_grounding/eval_stvg_vllm.py -eval/task/temporal_grounding/eval_timelens_hf.py -eval/task/temporal_grounding/run_eval.sh -eval/task/tracking/eval_tracking_vllm.py -orarl/__init__.py -orarl/resources.py -orarl.egg-info/PKG-INFO -orarl.egg-info/SOURCES.txt -orarl.egg-info/dependency_links.txt -orarl.egg-info/entry_points.txt -orarl.egg-info/requires.txt -orarl.egg-info/top_level.txt -orarl/algorithm/__init__.py -orarl/algorithm/_utils.py -orarl/algorithm/advantages.py -orarl/algorithm/config.py -orarl/algorithm/correction.py -orarl/algorithm/selection.py -orarl/cli/__init__.py -orarl/cli/eval_data.py -orarl/cli/evaluate.py -orarl/cli/prepare.py -orarl/cli/train.py -orarl/data/__init__.py -orarl/data/build.py -orarl/data/identity.py -orarl/data/schema.py -orarl/evaluation/__init__.py -orarl/evaluation/card.py -orarl/evaluation/converters.py -orarl/evaluation/hub.py -orarl/evaluation/layout.py -orarl/evaluation/manifest.py -orarl/evaluation/schema.py -orarl/evaluation/sources.py -orarl/evaluation/staging.py -orarl/rewards/__init__.py -orarl/rewards/router.py -orarl/rewards/types.py -orarl/rewards/adapters/__init__.py -orarl/rewards/adapters/_common.py -orarl/rewards/adapters/segmentation.py -orarl/rewards/adapters/spatial_grounding.py -orarl/rewards/adapters/spatial_intelligence.py -orarl/rewards/adapters/spatial_temporal_grounding.py -orarl/rewards/adapters/temporal_grounding.py -orarl/rewards/adapters/tracking.py -orarl/rewards/adapters/video_qa.py -scripts/check_environment.py -scripts/check_release.py -scripts/create_conda_env.sh -scripts/create_eval_source_manifest.py -scripts/evaluate.sh -scripts/finalize_eval_index.py -scripts/install_conda_runtime_hook.sh -scripts/launch_multinode.sh -scripts/model_merger.py -scripts/organize_evaluation_outputs.py -scripts/prepare_data.sh -scripts/smoke_evaluation.sh -scripts/smoke_training.sh -scripts/train_grpo.sh -scripts/train_orarl.sh -tests/test_algorithm_advantages.py -tests/test_algorithm_correction.py -tests/test_algorithm_selection.py -tests/test_builtin_reward_adapters.py -tests/test_cli_evaluate.py -tests/test_cli_metadata.py -tests/test_cli_train.py -tests/test_data_build.py -tests/test_data_schema.py -tests/test_environment_spec.py -tests/test_eval_canonical_data.py -tests/test_eval_source_discovery.py -tests/test_evaluation_manifest.py -tests/test_evaluation_release.py -tests/test_evaluation_runtime.py -tests/test_evaluation_schema.py -tests/test_evaluation_staging.py -tests/test_organize_evaluation_outputs.py -tests/test_release_hygiene.py -tests/test_resources.py -tests/test_rewards.py -tests/test_runtime_reward_entrypoint.py -tests/test_training_jsonl_dataset.py -tests/test_training_runtime.py -verl/__init__.py -verl/protocol.py -verl/models/__init__.py -verl/models/monkey_patch.py -verl/models/transformers/__init__.py -verl/models/transformers/flash_attention_utils.py -verl/models/transformers/qwen2_vl.py -verl/models/transformers/qwen3_5.py -verl/models/transformers/qwen3_vl.py -verl/single_controller/__init__.py -verl/single_controller/base/__init__.py -verl/single_controller/base/decorator.py -verl/single_controller/base/worker.py -verl/single_controller/base/worker_group.py -verl/single_controller/base/register_center/__init__.py -verl/single_controller/base/register_center/ray.py -verl/single_controller/ray/__init__.py -verl/single_controller/ray/base.py -verl/trainer/__init__.py -verl/trainer/config.py -verl/trainer/core_algos.py -verl/trainer/data_loader.py -verl/trainer/main.py -verl/trainer/metrics.py -verl/trainer/orarl_config.py -verl/trainer/orarl_post_selection.py -verl/trainer/orarl_selection.py -verl/trainer/ray_trainer.py -verl/utils/__init__.py -verl/utils/dataset.py -verl/utils/flops_counter.py -verl/utils/fsdp_utils.py -verl/utils/model_utils.py -verl/utils/multimodal_contract.py -verl/utils/prompt_template.py -verl/utils/py_functional.py -verl/utils/seqlen_balancing.py -verl/utils/tokenizer.py -verl/utils/torch_dtypes.py -verl/utils/torch_functional.py -verl/utils/ulysses.py -verl/utils/checkpoint/__init__.py -verl/utils/checkpoint/checkpoint_manager.py -verl/utils/checkpoint/fsdp_checkpoint_manager.py -verl/utils/logger/__init__.py -verl/utils/logger/gen_logger.py -verl/utils/logger/logger.py -verl/workers/__init__.py -verl/workers/config.py -verl/workers/fsdp_workers.py -verl/workers/actor/__init__.py -verl/workers/actor/base.py -verl/workers/actor/config.py -verl/workers/actor/dp_actor.py -verl/workers/critic/__init__.py -verl/workers/critic/base.py -verl/workers/critic/config.py -verl/workers/critic/dp_critic.py -verl/workers/reward/__init__.py -verl/workers/reward/config.py -verl/workers/reward/function.py -verl/workers/rollout/__init__.py -verl/workers/rollout/base.py -verl/workers/rollout/config.py -verl/workers/rollout/hf_rollout.py -verl/workers/rollout/vllm_rollout_spmd.py -verl/workers/sharding_manager/__init__.py -verl/workers/sharding_manager/base.py -verl/workers/sharding_manager/fsdp_ulysses.py -verl/workers/sharding_manager/fsdp_vllm.py \ No newline at end of file diff --git a/code/orarl.egg-info/dependency_links.txt b/code/orarl.egg-info/dependency_links.txt deleted file mode 100644 index 8b137891791fe96927ad78e64b0aad7bded08bdc..0000000000000000000000000000000000000000 --- a/code/orarl.egg-info/dependency_links.txt +++ /dev/null @@ -1 +0,0 @@ - diff --git a/code/orarl.egg-info/entry_points.txt b/code/orarl.egg-info/entry_points.txt deleted file mode 100644 index 709973346314612295b0da8478f38837c2896cfc..0000000000000000000000000000000000000000 --- a/code/orarl.egg-info/entry_points.txt +++ /dev/null @@ -1,5 +0,0 @@ -[console_scripts] -orarl-eval = orarl.cli.evaluate:main -orarl-eval-data = orarl.cli.eval_data:main -orarl-prepare = orarl.cli.prepare:main -orarl-train = orarl.cli.train:main diff --git a/code/orarl.egg-info/requires.txt b/code/orarl.egg-info/requires.txt deleted file mode 100644 index 3a476f18c1915282ed840b17c37848c6a66ca26d..0000000000000000000000000000000000000000 --- a/code/orarl.egg-info/requires.txt +++ /dev/null @@ -1,12 +0,0 @@ -numpy -PyYAML -torch - -[hf] -huggingface_hub - -[lint] -ruff - -[test] -pytest diff --git a/code/orarl.egg-info/top_level.txt b/code/orarl.egg-info/top_level.txt deleted file mode 100644 index 515595152e429980809f3de1df69091ae10204fe..0000000000000000000000000000000000000000 --- a/code/orarl.egg-info/top_level.txt +++ /dev/null @@ -1,2 +0,0 @@ -orarl -verl diff --git a/code/runs/smoke-training-9b-20260825_174304/smoke.log b/code/runs/smoke-training-9b-20260825_174304/smoke.log deleted file mode 100644 index 572849e23e215d0ddbbea096ad47301531b07ba3..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_174304/smoke.log +++ /dev/null @@ -1,467 +0,0 @@ -OraRL root: /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official -Python: /data/miniconda3/envs/orarl/bin/python -Model: /apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B -Train data: /mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_joint_100k_maskaware.jsonl -Val data: /mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_phase3_49152.canary.jsonl -Recipe scale: 9b -World size: 1 x 8 -Steps per run: 1 -Results: /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_174304 -Execute: 1 -0, NVIDIA H20, 0 MiB, 97871 MiB -1, NVIDIA H20, 0 MiB, 97871 MiB -2, NVIDIA H20, 0 MiB, 97871 MiB -3, NVIDIA H20, 0 MiB, 97871 MiB -4, NVIDIA H20, 0 MiB, 97871 MiB -5, NVIDIA H20, 0 MiB, 97871 MiB -6, NVIDIA H20, 0 MiB, 97871 MiB -7, NVIDIA H20, 0 MiB, 97871 MiB -/data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. - warnings.warn( -dependency OK: numpy 2.2.6 -dependency OK: torch 2.10.0+cu129 -dependency OK: transformers 5.5.4 -dependency OK: vllm 0.19.1 -dependency OK: ray 2.54.0 -dependency OK: tensordict 0.12.2 -dependency OK: codetiming 1.4.0 -dependency OK: omegaconf 2.3.0 -dependency OK: verl.trainer.main /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/trainer/main.py -dependency OK: orarl.rewards /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/orarl/rewards/__init__.py - -=== grpo smoke (1 update(s)) === -/data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. - warnings.warn( -2026-08-25 17:43:40,058 INFO worker.py:2004 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265  -/data/miniconda3/envs/orarl/lib/python3.11/site-packages/ray/_private/worker.py:2052: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 - warnings.warn( -(pid=1944439) W0825 17:43:56.478000 1944439 site-packages/torch/utils/cpp_extension.py:118] No CUDA runtime is found, using CUDA_HOME='/data/miniconda3/envs/orarl' -(Runner pid=1944439) { -(Runner pid=1944439) "data": { -(Runner pid=1944439) "train_files": "/mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_joint_100k_maskaware.jsonl", -(Runner pid=1944439) "val_files": "/mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_phase3_49152.canary.jsonl", -(Runner pid=1944439) "prompt_key": "problem", -(Runner pid=1944439) "answer_key": "answer", -(Runner pid=1944439) "image_key": "images", -(Runner pid=1944439) "video_key": "videos", -(Runner pid=1944439) "image_dir": "/", -(Runner pid=1944439) "video_fps": 2.0, -(Runner pid=1944439) "video_max_frames": 128, -(Runner pid=1944439) "max_prompt_length": 24576, -(Runner pid=1944439) "max_response_length": 4096, -(Runner pid=1944439) "rollout_batch_size": 8, -(Runner pid=1944439) "mini_rollout_batch_size": null, -(Runner pid=1944439) "val_batch_size": 16, -(Runner pid=1944439) "format_prompt": null, -(Runner pid=1944439) "override_chat_template": null, -(Runner pid=1944439) "enable_thinking": false, -(Runner pid=1944439) "response_prefix": "", -(Runner pid=1944439) "shuffle": true, -(Runner pid=1944439) "seed": 42, -(Runner pid=1944439) "min_pixels": 4096, -(Runner pid=1944439) "max_pixels": 4194304, -(Runner pid=1944439) "image_min_pixels": 4096, -(Runner pid=1944439) "image_max_pixels": 1048576, -(Runner pid=1944439) "video_min_pixels": 4096, -(Runner pid=1944439) "video_max_pixels": 786432, -(Runner pid=1944439) "video_total_pixels": 8388608, -(Runner pid=1944439) "val_video_fps": 2.0, -(Runner pid=1944439) "val_video_max_frames": 128, -(Runner pid=1944439) "val_video_min_pixels": 4096, -(Runner pid=1944439) "val_video_max_pixels": 786432, -(Runner pid=1944439) "val_video_total_pixels": 8388608, -(Runner pid=1944439) "filter_overlong_prompts": false, -(Runner pid=1944439) "filter_overlong_prompts_workers": 16, -(Runner pid=1944439) "use_preprocessed_videos": false, -(Runner pid=1944439) "video_source_mode": "realtime_only", -(Runner pid=1944439) "preprocessed_video_dir": null, -(Runner pid=1944439) "val_preprocessed_video_dir": null, -(Runner pid=1944439) "val_video_source_mode": "realtime_only", -(Runner pid=1944439) "inline_video_tensors": true, -(Runner pid=1944439) "group_by_task": true, -(Runner pid=1944439) "group_by_task_key": "problem_type", -(Runner pid=1944439) "dataloader_num_workers": 1 -(Runner pid=1944439) }, -(Runner pid=1944439) "worker": { -(Runner pid=1944439) "hybrid_engine": true, -(Runner pid=1944439) "actor": { -(Runner pid=1944439) "strategy": "fsdp", -(Runner pid=1944439) "global_batch_size": 8, -(Runner pid=1944439) "micro_batch_size_per_device_for_update": 1, -(Runner pid=1944439) "micro_batch_size_per_device_for_experience": 1, -(Runner pid=1944439) "max_grad_norm": 1.0, -(Runner pid=1944439) "clip_ratio_low": 0.2, -(Runner pid=1944439) "clip_ratio_high": 0.2, -(Runner pid=1944439) "clip_ratio_dual": 3.0, -(Runner pid=1944439) "loss_avg_mode": "token", -(Runner pid=1944439) "loss_type": "default", -(Runner pid=1944439) "ppo_epochs": 1, -(Runner pid=1944439) "padding_free": true, -(Runner pid=1944439) "dynamic_batching": true, -(Runner pid=1944439) "max_token_len_per_gpu": 32768, -(Runner pid=1944439) "ulysses_size": 1, -(Runner pid=1944439) "use_torch_compile": false, -(Runner pid=1944439) "model": { -(Runner pid=1944439) "model_path": "/apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B", -(Runner pid=1944439) "tokenizer_path": "/apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B", -(Runner pid=1944439) "override_config": {}, -(Runner pid=1944439) "enable_gradient_checkpointing": true, -(Runner pid=1944439) "trust_remote_code": false, -(Runner pid=1944439) "freeze_vision_tower": true, -(Runner pid=1944439) "train_vision_merger": false -(Runner pid=1944439) }, -(Runner pid=1944439) "optim": { -(Runner pid=1944439) "lr": 1e-06, -(Runner pid=1944439) "betas": [ -(Runner pid=1944439) 0.9, -(Runner pid=1944439) 0.999 -(Runner pid=1944439) ], -(Runner pid=1944439) "weight_decay": 0.0, -(Runner pid=1944439) "strategy": "adamw", -(Runner pid=1944439) "lr_warmup_ratio": 0.0, -(Runner pid=1944439) "lr_warmup_steps": null, -(Runner pid=1944439) "min_lr_ratio": null, -(Runner pid=1944439) "lr_scheduler_type": "constant", -(Runner pid=1944439) "training_steps": -1 -(Runner pid=1944439) }, -(Runner pid=1944439) "fsdp": { -(Runner pid=1944439) "enable_full_shard": true, -(Runner pid=1944439) "enable_cpu_offload": false, -(Runner pid=1944439) "enable_rank0_init": true, -(Runner pid=1944439) "use_orig_params": false, -(Runner pid=1944439) "torch_dtype": "bf16", -(Runner pid=1944439) "fsdp_size": -1, -(Runner pid=1944439) "mp_param_dtype": "bf16", -(Runner pid=1944439) "mp_reduce_dtype": "fp32", -(Runner pid=1944439) "mp_buffer_dtype": "fp32" -(Runner pid=1944439) }, -(Runner pid=1944439) "offload": { -(Runner pid=1944439) "offload_params": false, -(Runner pid=1944439) "offload_optimizer": false -(Runner pid=1944439) }, -(Runner pid=1944439) "global_batch_size_per_device": -1, -(Runner pid=1944439) "disable_kl": true, -(Runner pid=1944439) "use_kl_loss": false, -(Runner pid=1944439) "kl_penalty": "low_var_kl", -(Runner pid=1944439) "kl_coef": 0.0 -(Runner pid=1944439) }, -(Runner pid=1944439) "critic": { -(Runner pid=1944439) "strategy": "fsdp", -(Runner pid=1944439) "global_batch_size": 256, -(Runner pid=1944439) "micro_batch_size_per_device_for_update": 4, -(Runner pid=1944439) "micro_batch_size_per_device_for_experience": 16, -(Runner pid=1944439) "max_grad_norm": 1.0, -(Runner pid=1944439) "cliprange_value": 0.5, -(Runner pid=1944439) "loss_avg_mode": "token", -(Runner pid=1944439) "ppo_epochs": 1, -(Runner pid=1944439) "padding_free": false, -(Runner pid=1944439) "dynamic_batching": true, -(Runner pid=1944439) "max_token_len_per_gpu": null, -(Runner pid=1944439) "ulysses_size": 1, -(Runner pid=1944439) "model": { -(Runner pid=1944439) "model_path": null, -(Runner pid=1944439) "tokenizer_path": null, -(Runner pid=1944439) "override_config": {}, -(Runner pid=1944439) "enable_gradient_checkpointing": true, -(Runner pid=1944439) "trust_remote_code": true, -(Runner pid=1944439) "freeze_vision_tower": false, -(Runner pid=1944439) "train_vision_merger": false -(Runner pid=1944439) }, -(Runner pid=1944439) "optim": { -(Runner pid=1944439) "lr": 1e-06, -(Runner pid=1944439) "betas": [ -(Runner pid=1944439) 0.9,(Runner pid=1944439) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. -(Runner pid=1944439) Generating train split: 0 examples [00:00, ? examples/s] -(Runner pid=1944439) Generating train split: 2684 examples [00:00, 5741.64 examples/s] -(Runner pid=1944439) Generating train split: 3894 examples [00:00, 5566.39 examples/s] -(Runner pid=1944439) Generating train split: 6050 examples [00:01, 5916.34 examples/s] -(Runner pid=1944439) Generating train split: 7310 examples [00:01, 5917.78 examples/s] -(Runner pid=1944439) Generating train split: 8652 examples [00:01, 7062.55 examples/s] -(Runner pid=1944439) Generating train split: 10017 examples [00:01, 5373.29 examples/s] -(Runner pid=1944439) Generating train split: 11061 examples [00:01, 6122.95 examples/s] -(Runner pid=1944439) Generating train split: 12345 examples [00:01, 6764.91 examples/s] -(Runner pid=1944439) Generating train split: 13564 examples [00:02, 4332.70 examples/s] -(Runner pid=1944439) Generating train split: 14792 examples [00:02, 5251.29 examples/s] -(Runner pid=1944439) Generating train split: 15958 examples [00:02, 5219.09 examples/s] -(Runner pid=1944439) Generating train split: 17208 examples [00:03, 5223.77 examples/s] -(Runner pid=1944439) Generating train split: 18484 examples [00:03, 6319.60 examples/s] -(Runner pid=1944439) Generating train split: 19610 examples [00:03, 5646.33 examples/s] -(Runner pid=1944439) Generating train split: 20953 examples [00:03, 6782.24 examples/s] -(Runner pid=1944439) Generating train split: 22170 examples [00:03, 7115.81 examples/s] -(Runner pid=1944439) Generating train split: 23412 examples [00:03, 6220.19 examples/s] -(Runner pid=1944439) Generating train split: 24542 examples [00:04, 7068.41 examples/s] -(Runner pid=1944439) Generating train split: 25620 examples [00:04, 5887.92 examples/s] -(Runner pid=1944439) Generating train split: 27027 examples [00:05, 2384.25 examples/s] -(Runner pid=1944439) Generating train split: 29107 examples [00:06, 2709.31 examples/s] -(Runner pid=1944439) Generating train split: 30179 examples [00:06, 2852.90 examples/s] -(Runner pid=1944439) Generating train split: 32832 examples [00:06, 4368.47 examples/s] -(Runner pid=1944439) Generating train split: 35008 examples [00:06, 5689.84 examples/s] -(Runner pid=1944439) Generating train split: 38278 examples [00:07, 7779.78 examples/s] -(Runner pid=1944439) Generating train split: 41012 examples [00:07, 9843.69 examples/s] -(Runner pid=1944439) Generating train split: 43744 examples [00:07, 10491.50 examples/s] -(Runner pid=1944439) Generating train split: 46312 examples [00:07, 12026.91 examples/s] -(Runner pid=1944439) Generating train split: 49391 examples [00:08, 8630.42 examples/s] -(Runner pid=1944439) Generating train split: 52225 examples [00:08, 10551.99 examples/s] -(Runner pid=1944439) Generating train split: 52225 examples [00:08, 6076.51 examples/s] -Traceback (most recent call last): - File "", line 198, in _run_module_as_main - File "", line 88, in _run_code - File "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/trainer/main.py", line 137, in - main() - File "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/trainer/main.py", line 129, in main - ray.get(runner.run.remote(ppo_config)) - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper - return fn(*args, **kwargs) - ^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper - return func(*args, **kwargs) - ^^^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/ray/_private/worker.py", line 2981, in get - values, debugger_breakpoint = worker.get_objects( - ^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/ray/_private/worker.py", line 1012, in get_objects - raise value.as_instanceof_cause() -ray.exceptions.RayTaskError(DatasetGenerationError): ray::Runner.run() (pid=1944439, ip=29.127.69.4, actor_id=e981ae3ef4ecbc78705cfeea01000000, repr=) - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/datasets/packaged_modules/json/json.py", line 265, in _generate_tables - self._cast_table(pa_table, json_field_paths=json_field_paths), - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/datasets/packaged_modules/json/json.py", line 120, in _cast_table - pa_table = table_cast(pa_table, self.info.features.arrow_schema) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/datasets/table.py", line 2272, in table_cast - return cast_table_to_schema(table, schema) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/datasets/table.py", line 2218, in cast_table_to_schema - raise CastError( -datasets.table.CastError: Couldn't cast -problem: string -answer: string -ground_truth: string -problem_type: string -videos: list - child 0, item: string -data_type: string -task_source: string -segmentation_output: string -meta: string -images: list - child 0, item: string -fps: int64 -video_second: double -resolution: struct - child 0, width: int64 - child 1, height: int64 -path: string -data_source: string -messages: list> - child 0, item: struct - child 0, role: string - child 1, content: string -solution: string -difficulty: double -pred_iou: double -pred_span: list - child 0, item: double -pred_text: string -checkpoint_score: double -hardness_band: string -selection_role: string -selection_checkpoint: string -identity_key: string -media_anchors: list - child 0, item: string -selection_temporal_bucket: string -question_type: string -source_file: string -source_id: string -selection_weak_buckets: list - child 0, item: string -selection_bucket: string -source_block: string -selection_scores: struct) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/trainer/main.py", line 74, in run - train_dataloader, val_dataloader = create_dataloader( - ^^^^^^^^^^^^^^^^^^ - File "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/trainer/data_loader.py", line 41, in create_dataloader - train_dataset = RLHFDataset( - ^^^^^^^^^^^^ - File "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/utils/dataset.py", line 444, in __init__ - self.dataset = load_dataset(file_type, data_files=data_path, split=data_split) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/datasets/load.py", line 1708, in load_dataset - builder_instance.download_and_prepare( - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/datasets/builder.py", line 884, in download_and_prepare - self._download_and_prepare( - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/datasets/builder.py", line 947, in _download_and_prepare - self._prepare_split(split_generator, **prepare_split_kwargs) - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/datasets/builder.py", line 1739, in _prepare_split - for job_id, done, content in self._prepare_split_single( - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/datasets/builder.py", line 1922, in _prepare_split_single - raise DatasetGenerationError("An error occurred while generating the dataset") from e -datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset - -(Runner pid=1944439) 0.999 -(Runner pid=1944439) ], -(Runner pid=1944439) "weight_decay": 0.01, -(Runner pid=1944439) "strategy": "adamw", -(Runner pid=1944439) "lr_warmup_ratio": 0.0, -(Runner pid=1944439) "lr_warmup_steps": null, -(Runner pid=1944439) "min_lr_ratio": null, -(Runner pid=1944439) "lr_scheduler_type": "constant", -(Runner pid=1944439) "training_steps": -1 -(Runner pid=1944439) }, -(Runner pid=1944439) "fsdp": { -(Runner pid=1944439) "enable_full_shard": true, -(Runner pid=1944439) "enable_cpu_offload": false, -(Runner pid=1944439) "enable_rank0_init": true, -(Runner pid=1944439) "use_orig_params": false, -(Runner pid=1944439) "torch_dtype": null, -(Runner pid=1944439) "fsdp_size": -1, -(Runner pid=1944439) "mp_param_dtype": "bf16", -(Runner pid=1944439) "mp_reduce_dtype": "fp32", -(Runner pid=1944439) "mp_buffer_dtype": "fp32" -(Runner pid=1944439) }, -(Runner pid=1944439) "offload": { -(Runner pid=1944439) "offload_params": false, -(Runner pid=1944439) "offload_optimizer": false -(Runner pid=1944439) }, -(Runner pid=1944439) "global_batch_size_per_device": -1 -(Runner pid=1944439) }, -(Runner pid=1944439) "ref": { -(Runner pid=1944439) "strategy": "fsdp", -(Runner pid=1944439) "fsdp": { -(Runner pid=1944439) "enable_full_shard": true, -(Runner pid=1944439) "enable_cpu_offload": false, -(Runner pid=1944439) "enable_rank0_init": true, -(Runner pid=1944439) "use_orig_params": false, -(Runner pid=1944439) "torch_dtype": "bf16", -(Runner pid=1944439) "fsdp_size": -1, -(Runner pid=1944439) "mp_param_dtype": "bf16", -(Runner pid=1944439) "mp_reduce_dtype": "fp32", -(Runner pid=1944439) "mp_buffer_dtype": "fp32" -(Runner pid=1944439) }, -(Runner pid=1944439) "offload": { -(Runner pid=1944439) "offload_params": false, -(Runner pid=1944439) "offload_optimizer": false -(Runner pid=1944439) }, -(Runner pid=1944439) "micro_batch_size_per_device_for_experience": 1, -(Runner pid=1944439) "padding_free": true, -(Runner pid=1944439) "dynamic_batching": true, -(Runner pid=1944439) "max_token_len_per_gpu": 32768, -(Runner pid=1944439) "ulysses_size": 1, -(Runner pid=1944439) "use_torch_compile": false -(Runner pid=1944439) }, -(Runner pid=1944439) "reward": { -(Runner pid=1944439) "reward_function": "orarl.rewards", -(Runner pid=1944439) "reward_function_kwargs": {}, -(Runner pid=1944439) "skip_special_tokens": true, -(Runner pid=1944439) "num_cpus": 1, -(Runner pid=1944439) "reward_function_name": "compute_score", -(Runner pid=1944439) "reward_function_is_module": true -(Runner pid=1944439) }, -(Runner pid=1944439) "rollout": { -(Runner pid=1944439) "name": "vllm", -(Runner pid=1944439) "n": 8, -(Runner pid=1944439) "temperature": 1.0, -(Runner pid=1944439) "top_p": 0.85, -(Runner pid=1944439) "top_k": -1, -(Runner pid=1944439) "seed": 42, -(Runner pid=1944439) "limit_images": 10, -(Runner pid=1944439) "dtype": "bf16", -(Runner pid=1944439) "gpu_memory_utilization": 0.45, -(Runner pid=1944439) "ignore_eos": false, -(Runner pid=1944439) "enforce_eager": false, -(Runner pid=1944439) "enable_chunked_prefill": true, -(Runner pid=1944439) "tensor_parallel_size": 1, -(Runner pid=1944439) "max_model_len": null, -(Runner pid=1944439) "max_num_batched_tokens": 32768, -(Runner pid=1944439) "disable_log_stats": true, -(Runner pid=1944439) "disable_tqdm": true, -(Runner pid=1944439) "val_override_config": { -(Runner pid=1944439) "temperature": 0.7, -(Runner pid=1944439) "top_p": 0.95, -(Runner pid=1944439) "n": 1 -(Runner pid=1944439) }, -(Runner pid=1944439) "kv_cache_dtype": "auto", -(Runner pid=1944439) "calculate_log_probs": false, -(Runner pid=1944439) "collect_seq_logprob_for_filter": false, -(Runner pid=1944439) "prompt_length": 24576, -(Runner pid=1944439) "response_length": 4096, -(Runner pid=1944439) "trust_remote_code": false -(Runner pid=1944439) } -(Runner pid=1944439) }, -(Runner pid=1944439) "algorithm": { -(Runner pid=1944439) "name": "grpo", -(Runner pid=1944439) "gamma": 1.0, -(Runner pid=1944439) "lam": 1.0, -(Runner pid=1944439) "adv_estimator": "grpo", -(Runner pid=1944439) "scale_rewards": true, -(Runner pid=1944439) "disable_kl": true, -(Runner pid=1944439) "use_kl_loss": false, -(Runner pid=1944439) "kl_penalty": "low_var_kl", -(Runner pid=1944439) "kl_coef": 0.0, -(Runner pid=1944439) "kl_type": "fixed", -(Runner pid=1944439) "kl_horizon": 10000.0, -(Runner pid=1944439) "kl_target": 0.1 -(Runner pid=1944439) }, -(Runner pid=1944439) "trainer": { -(Runner pid=1944439) "total_epochs": 1, -(Runner pid=1944439) "max_steps": 1, -(Runner pid=1944439) "project_name": "OraRL", -(Runner pid=1944439) "experiment_name": "smoke-grpo-9b", -(Runner pid=1944439) "logger": [ -(Runner pid=1944439) "console" -(Runner pid=1944439) ], -(Runner pid=1944439) "nnodes": 1, -(Runner pid=1944439) "n_gpus_per_node": 8, -(Runner pid=1944439) "critic_warmup": 0, -(Runner pid=1944439) "val_freq": -1, -(Runner pid=1944439) "val_before_train": false, -(Runner pid=1944439) "val_only": false, -(Runner pid=1944439) "val_generations_to_log": 0, -(Runner pid=1944439) "save_freq": 1, -(Runner pid=1944439) "save_limit": 3, -(Runner pid=1944439) "save_model_only": false, -(Runner pid=1944439) "keep_optim_only_latest": true, -(Runner pid=1944439) "save_checkpoint_path": "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_174304/grpo", -(Runner pid=1944439) "load_checkpoint_path": null, -(Runner pid=1944439) "ray_timeline": null, -(Runner pid=1944439) "find_last_checkpoint": false, -(Runner pid=1944439) "keep_best_train_ckpt": false, -(Runner pid=1944439) "best_train_metric_key": "reward/overall", -(Runner pid=1944439) "best_train_smooth_window": 5, -(Runner pid=1944439) "best_train_min_step": 10 -(Runner pid=1944439) } -(Runner pid=1944439) } -(AutoRewardManager pid=1946036) Using reward function `compute_score` from `orarl.rewards`. -(AutoRewardManager pid=1946036) Reward name: orarl, reward type: batch. -(AutoRewardManager pid=1946038) Using reward function `compute_score` from `orarl.rewards`. -(AutoRewardManager pid=1946038) Reward name: orarl, reward type: batch. -method=grpo mode=run -/data/miniconda3/envs/orarl/bin/python -m verl.trainer.main config=/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/configs/grpo_9b.yaml trainer.max_steps=1 trainer.save_freq=1 trainer.val_before_train=false trainer.experiment_name=smoke-grpo-9b data.rollout_batch_size=8 worker.actor.global_batch_size=8 worker.actor.model.model_path=/apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B worker.actor.model.tokenizer_path=/apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B data.train_files=/mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_joint_100k_maskaware.jsonl data.val_files=/mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_phase3_49152.canary.jsonl trainer.save_checkpoint_path=/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_174304/grpo trainer.nnodes=1 trainer.n_gpus_per_node=8 diff --git a/code/runs/smoke-training-9b-20260825_175213/smoke.log b/code/runs/smoke-training-9b-20260825_175213/smoke.log deleted file mode 100644 index ec43ca43443fafe8272b4cd9c86b41fc634ccb26..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175213/smoke.log +++ /dev/null @@ -1,827 +0,0 @@ -OraRL root: /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official -Python: /data/miniconda3/envs/orarl/bin/python -Model: /apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B -Train data: /mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_joint_100k_maskaware.jsonl -Val data: /mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_phase3_49152.canary.jsonl -Recipe scale: 9b -World size: 1 x 8 -Steps per run: 1 -Results: /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175213 -Execute: 1 -0, NVIDIA H20, 0 MiB, 97871 MiB -1, NVIDIA H20, 0 MiB, 97871 MiB -2, NVIDIA H20, 0 MiB, 97871 MiB -3, NVIDIA H20, 0 MiB, 97871 MiB -4, NVIDIA H20, 0 MiB, 97871 MiB -5, NVIDIA H20, 0 MiB, 97871 MiB -6, NVIDIA H20, 0 MiB, 97871 MiB -7, NVIDIA H20, 0 MiB, 97871 MiB -/data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. - warnings.warn( -dependency OK: numpy 2.2.6 -dependency OK: torch 2.10.0+cu129 -dependency OK: transformers 5.5.4 -dependency OK: vllm 0.19.1 -dependency OK: ray 2.54.0 -dependency OK: tensordict 0.12.2 -dependency OK: codetiming 1.4.0 -dependency OK: omegaconf 2.3.0 -dependency OK: verl.trainer.main /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/trainer/main.py -dependency OK: orarl.rewards /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/orarl/rewards/__init__.py - -=== grpo smoke (1 update(s)) === -/data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. - warnings.warn( -2026-08-25 17:52:41,284 INFO worker.py:2004 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265  -/data/miniconda3/envs/orarl/lib/python3.11/site-packages/ray/_private/worker.py:2052: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 - warnings.warn( -(pid=1984436) W0825 17:52:57.110000 1984436 site-packages/torch/utils/cpp_extension.py:118] No CUDA runtime is found, using CUDA_HOME='/data/miniconda3/envs/orarl' -(Runner pid=1984436) { -(Runner pid=1984436) "data": { -(Runner pid=1984436) "train_files": "/mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_joint_100k_maskaware.jsonl", -(Runner pid=1984436) "val_files": "/mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_phase3_49152.canary.jsonl", -(Runner pid=1984436) "prompt_key": "problem", -(Runner pid=1984436) "answer_key": "answer", -(Runner pid=1984436) "image_key": "images", -(Runner pid=1984436) "video_key": "videos", -(Runner pid=1984436) "image_dir": "/", -(Runner pid=1984436) "video_fps": 2.0, -(Runner pid=1984436) "video_max_frames": 128, -(Runner pid=1984436) "max_prompt_length": 24576, -(Runner pid=1984436) "max_response_length": 4096, -(Runner pid=1984436) "rollout_batch_size": 8, -(Runner pid=1984436) "mini_rollout_batch_size": null, -(Runner pid=1984436) "val_batch_size": 16, -(Runner pid=1984436) "format_prompt": null, -(Runner pid=1984436) "override_chat_template": null, -(Runner pid=1984436) "enable_thinking": false, -(Runner pid=1984436) "response_prefix": "", -(Runner pid=1984436) "shuffle": true, -(Runner pid=1984436) "seed": 42, -(Runner pid=1984436) "min_pixels": 4096, -(Runner pid=1984436) "max_pixels": 4194304, -(Runner pid=1984436) "image_min_pixels": 4096, -(Runner pid=1984436) "image_max_pixels": 1048576, -(Runner pid=1984436) "video_min_pixels": 4096, -(Runner pid=1984436) "video_max_pixels": 786432, -(Runner pid=1984436) "video_total_pixels": 8388608, -(Runner pid=1984436) "val_video_fps": 2.0, -(Runner pid=1984436) "val_video_max_frames": 128, -(Runner pid=1984436) "val_video_min_pixels": 4096, -(Runner pid=1984436) "val_video_max_pixels": 786432, -(Runner pid=1984436) "val_video_total_pixels": 8388608, -(Runner pid=1984436) "filter_overlong_prompts": false, -(Runner pid=1984436) "filter_overlong_prompts_workers": 16, -(Runner pid=1984436) "use_preprocessed_videos": false, -(Runner pid=1984436) "video_source_mode": "realtime_only", -(Runner pid=1984436) "preprocessed_video_dir": null, -(Runner pid=1984436) "val_preprocessed_video_dir": null, -(Runner pid=1984436) "val_video_source_mode": "realtime_only", -(Runner pid=1984436) "inline_video_tensors": true, -(Runner pid=1984436) "group_by_task": true, -(Runner pid=1984436) "group_by_task_key": "problem_type", -(Runner pid=1984436) "dataloader_num_workers": 1 -(Runner pid=1984436) }, -(Runner pid=1984436) "worker": { -(Runner pid=1984436) "hybrid_engine": true, -(Runner pid=1984436) "actor": { -(Runner pid=1984436) "strategy": "fsdp", -(Runner pid=1984436) "global_batch_size": 8, -(Runner pid=1984436) "micro_batch_size_per_device_for_update": 1, -(Runner pid=1984436) "micro_batch_size_per_device_for_experience": 1, -(Runner pid=1984436) "max_grad_norm": 1.0, -(Runner pid=1984436) "clip_ratio_low": 0.2, -(Runner pid=1984436) "clip_ratio_high": 0.2, -(Runner pid=1984436) "clip_ratio_dual": 3.0, -(Runner pid=1984436) "loss_avg_mode": "token", -(Runner pid=1984436) "loss_type": "default", -(Runner pid=1984436) "ppo_epochs": 1, -(Runner pid=1984436) "padding_free": true, -(Runner pid=1984436) "dynamic_batching": true, -(Runner pid=1984436) "max_token_len_per_gpu": 32768, -(Runner pid=1984436) "ulysses_size": 1, -(Runner pid=1984436) "use_torch_compile": false, -(Runner pid=1984436) "model": { -(Runner pid=1984436) "model_path": "/apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B", -(Runner pid=1984436) "tokenizer_path": "/apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B", -(Runner pid=1984436) "override_config": {}, -(Runner pid=1984436) "enable_gradient_checkpointing": true, -(Runner pid=1984436) "trust_remote_code": false, -(Runner pid=1984436) "freeze_vision_tower": true, -(Runner pid=1984436) "train_vision_merger": false -(Runner pid=1984436) }, -(Runner pid=1984436) "optim": { -(Runner pid=1984436) "lr": 1e-06, -(Runner pid=1984436) "betas": [ -(Runner pid=1984436) 0.9, -(Runner pid=1984436) 0.999 -(Runner pid=1984436) ], -(Runner pid=1984436) "weight_decay": 0.0, -(Runner pid=1984436) "strategy": "adamw", -(Runner pid=1984436) "lr_warmup_ratio": 0.0, -(Runner pid=1984436) "lr_warmup_steps": null, -(Runner pid=1984436) "min_lr_ratio": null, -(Runner pid=1984436) "lr_scheduler_type": "constant", -(Runner pid=1984436) "training_steps": -1 -(Runner pid=1984436) }, -(Runner pid=1984436) "fsdp": { -(Runner pid=1984436) "enable_full_shard": true, -(Runner pid=1984436) "enable_cpu_offload": false, -(Runner pid=1984436) "enable_rank0_init": true, -(Runner pid=1984436) "use_orig_params": false, -(Runner pid=1984436) "torch_dtype": "bf16", -(Runner pid=1984436) "fsdp_size": -1, -(Runner pid=1984436) "mp_param_dtype": "bf16", -(Runner pid=1984436) "mp_reduce_dtype": "fp32", -(Runner pid=1984436) "mp_buffer_dtype": "fp32" -(Runner pid=1984436) }, -(Runner pid=1984436) "offload": { -(Runner pid=1984436) "offload_params": false, -(Runner pid=1984436) "offload_optimizer": false -(Runner pid=1984436) }, -(Runner pid=1984436) "global_batch_size_per_device": -1, -(Runner pid=1984436) "disable_kl": true, -(Runner pid=1984436) "use_kl_loss": false, -(Runner pid=1984436) "kl_penalty": "low_var_kl", -(Runner pid=1984436) "kl_coef": 0.0 -(Runner pid=1984436) }, -(Runner pid=1984436) "critic": { -(Runner pid=1984436) "strategy": "fsdp", -(Runner pid=1984436) "global_batch_size": 256, -(Runner pid=1984436) "micro_batch_size_per_device_for_update": 4, -(Runner pid=1984436) "micro_batch_size_per_device_for_experience": 16, -(Runner pid=1984436) "max_grad_norm": 1.0, -(Runner pid=1984436) "cliprange_value": 0.5, -(Runner pid=1984436) "loss_avg_mode": "token", -(Runner pid=1984436) "ppo_epochs": 1, -(Runner pid=1984436) "padding_free": false, -(Runner pid=1984436) "dynamic_batching": true, -(Runner pid=1984436) "max_token_len_per_gpu": null, -(Runner pid=1984436) "ulysses_size": 1, -(Runner pid=1984436) "model": { -(Runner pid=1984436) "model_path": null, -(Runner pid=1984436) "tokenizer_path": null, -(Runner pid=1984436) "override_config": {}, -(Runner pid=1984436) "enable_gradient_checkpointing": true, -(Runner pid=1984436) "trust_remote_code": true, -(Runner pid=1984436) "freeze_vision_tower": false, -(Runner pid=1984436) "train_vision_merger": false -(Runner pid=1984436) }, -(Runner pid=1984436) "optim": { -(Runner pid=1984436) "lr": 1e-06, -(Runner pid=1984436) "betas": [ -(Runner pid=1984436) 0.9,(Runner pid=1984436) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. - -(Runner pid=1984436) 0.999 -(Runner pid=1984436) ], -(Runner pid=1984436) "weight_decay": 0.01, -(Runner pid=1984436) "strategy": "adamw", -(Runner pid=1984436) "lr_warmup_ratio": 0.0, -(Runner pid=1984436) "lr_warmup_steps": null, -(Runner pid=1984436) "min_lr_ratio": null, -(Runner pid=1984436) "lr_scheduler_type": "constant", -(Runner pid=1984436) "training_steps": -1 -(Runner pid=1984436) }, -(Runner pid=1984436) "fsdp": { -(Runner pid=1984436) "enable_full_shard": true, -(Runner pid=1984436) "enable_cpu_offload": false, -(Runner pid=1984436) "enable_rank0_init": true, -(Runner pid=1984436) "use_orig_params": false, -(Runner pid=1984436) "torch_dtype": null, -(Runner pid=1984436) "fsdp_size": -1, -(Runner pid=1984436) "mp_param_dtype": "bf16", -(Runner pid=1984436) "mp_reduce_dtype": "fp32", -(Runner pid=1984436) "mp_buffer_dtype": "fp32" -(Runner pid=1984436) }, -(Runner pid=1984436) "offload": { -(Runner pid=1984436) "offload_params": false, -(Runner pid=1984436) "offload_optimizer": false -(Runner pid=1984436) }, -(Runner pid=1984436) "global_batch_size_per_device": -1 -(Runner pid=1984436) }, -(Runner pid=1984436) "ref": { -(Runner pid=1984436) "strategy": "fsdp", -(Runner pid=1984436) "fsdp": { -(Runner pid=1984436) "enable_full_shard": true, -(Runner pid=1984436) "enable_cpu_offload": false, -(Runner pid=1984436) "enable_rank0_init": true, -(Runner pid=1984436) "use_orig_params": false, -(Runner pid=1984436) "torch_dtype": "bf16", -(Runner pid=1984436) "fsdp_size": -1, -(Runner pid=1984436) "mp_param_dtype": "bf16", -(Runner pid=1984436) "mp_reduce_dtype": "fp32", -(Runner pid=1984436) "mp_buffer_dtype": "fp32" -(Runner pid=1984436) }, -(Runner pid=1984436) "offload": { -(Runner pid=1984436) "offload_params": false, -(Runner pid=1984436) "offload_optimizer": false -(Runner pid=1984436) }, -(Runner pid=1984436) "micro_batch_size_per_device_for_experience": 1, -(Runner pid=1984436) "padding_free": true, -(Runner pid=1984436) "dynamic_batching": true, -(Runner pid=1984436) "max_token_len_per_gpu": 32768, -(Runner pid=1984436) "ulysses_size": 1, -(Runner pid=1984436) "use_torch_compile": false -(Runner pid=1984436) }, -(Runner pid=1984436) "reward": { -(Runner pid=1984436) "reward_function": "orarl.rewards", -(Runner pid=1984436) "reward_function_kwargs": {}, -(Runner pid=1984436) "skip_special_tokens": true, -(Runner pid=1984436) "num_cpus": 1, -(Runner pid=1984436) "reward_function_name": "compute_score", -(Runner pid=1984436) "reward_function_is_module": true -(Runner pid=1984436) }, -(Runner pid=1984436) "rollout": { -(Runner pid=1984436) "name": "vllm", -(Runner pid=1984436) "n": 8, -(Runner pid=1984436) "temperature": 1.0, -(Runner pid=1984436) "top_p": 0.85, -(Runner pid=1984436) "top_k": -1, -(Runner pid=1984436) "seed": 42, -(Runner pid=1984436) "limit_images": 10, -(Runner pid=1984436) "dtype": "bf16", -(Runner pid=1984436) "gpu_memory_utilization": 0.45, -(Runner pid=1984436) "ignore_eos": false, -(Runner pid=1984436) "enforce_eager": false, -(Runner pid=1984436) "enable_chunked_prefill": true, -(Runner pid=1984436) "tensor_parallel_size": 1, -(Runner pid=1984436) "max_model_len": null, -(Runner pid=1984436) "max_num_batched_tokens": 32768, -(Runner pid=1984436) "disable_log_stats": true, -(Runner pid=1984436) "disable_tqdm": true, -(Runner pid=1984436) "val_override_config": { -(Runner pid=1984436) "temperature": 0.7, -(Runner pid=1984436) "top_p": 0.95, -(Runner pid=1984436) "n": 1 -(Runner pid=1984436) }, -(Runner pid=1984436) "kv_cache_dtype": "auto", -(Runner pid=1984436) "calculate_log_probs": false, -(Runner pid=1984436) "collect_seq_logprob_for_filter": false, -(Runner pid=1984436) "prompt_length": 24576, -(Runner pid=1984436) "response_length": 4096, -(Runner pid=1984436) "trust_remote_code": false -(Runner pid=1984436) } -(Runner pid=1984436) }, -(Runner pid=1984436) "algorithm": { -(Runner pid=1984436) "name": "grpo", -(Runner pid=1984436) "gamma": 1.0, -(Runner pid=1984436) "lam": 1.0, -(Runner pid=1984436) "adv_estimator": "grpo", -(Runner pid=1984436) "scale_rewards": true, -(Runner pid=1984436) "disable_kl": true, -(Runner pid=1984436) "use_kl_loss": false, -(Runner pid=1984436) "kl_penalty": "low_var_kl", -(Runner pid=1984436) "kl_coef": 0.0, -(Runner pid=1984436) "kl_type": "fixed", -(Runner pid=1984436) "kl_horizon": 10000.0, -(Runner pid=1984436) "kl_target": 0.1 -(Runner pid=1984436) }, -(Runner pid=1984436) "trainer": { -(Runner pid=1984436) "total_epochs": 1, -(Runner pid=1984436) "max_steps": 1, -(Runner pid=1984436) "project_name": "OraRL", -(Runner pid=1984436) "experiment_name": "smoke-grpo-9b", -(Runner pid=1984436) "logger": [ -(Runner pid=1984436) "console" -(Runner pid=1984436) ], -(Runner pid=1984436) "nnodes": 1, -(Runner pid=1984436) "n_gpus_per_node": 8, -(Runner pid=1984436) "critic_warmup": 0, -(Runner pid=1984436) "val_freq": -1, -(Runner pid=1984436) "val_before_train": false, -(Runner pid=1984436) "val_only": false, -(Runner pid=1984436) "val_generations_to_log": 0, -(Runner pid=1984436) "save_freq": 1, -(Runner pid=1984436) "save_limit": 3, -(Runner pid=1984436) "save_model_only": false, -(Runner pid=1984436) "keep_optim_only_latest": true, -(Runner pid=1984436) "save_checkpoint_path": "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175213/grpo", -(Runner pid=1984436) "load_checkpoint_path": null, -(Runner pid=1984436) "ray_timeline": null, -(Runner pid=1984436) "find_last_checkpoint": false, -(Runner pid=1984436) "keep_best_train_ckpt": false, -(Runner pid=1984436) "best_train_metric_key": "reward/overall", -(Runner pid=1984436) "best_train_smooth_window": 5, -(Runner pid=1984436) "best_train_min_step": 10 -(Runner pid=1984436) } -(Runner pid=1984436) } -(Runner pid=1984436) [TaskGroupedBatchSampler] 15 groups: image_sequence_mc_answer_only(10304), obj_appearance_order(64), object_abs_distance(1344), object_counting(960), object_rel_direction(448), object_rel_distance(640), object_size_estimation(1408), room_size_estimation(960), route_planning(960), segmentation(12032), spatial grounding(7040), spatial-temporal grounding(9536), temporal grounding(20096), tracking(13952), video_qa_mc(20288) | batch_size=8 | total_batches=12504 -(Runner pid=1984436) Size of train dataloader: 12504 -(Runner pid=1984436) Size of val dataloader: 32 -(Runner pid=1984436) KL is disabled, no KL metrics will be logged. Please set `kl_coef=0` to log KL metrics.(pid=1986239) /data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. -(pid=1986239) warnings.warn( -(pid=1986747) /data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. -(pid=1986747) warnings.warn( -(pid=1986750) /data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. -(pid=1986750) warnings.warn( -(WorkerDict pid=1986750) [W825 17:53:20.598420573 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator()) -(pid=1986745) /data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. [repeated 5x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) -(pid=1986745) warnings.warn( [repeated 5x across cluster] -(WorkerDict pid=1986750) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. -(WorkerDict pid=1986750) `torch_dtype` is deprecated! Use `dtype` instead! -(WorkerDict pid=1986745) [W825 17:53:22.611308342 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator()) [repeated 7x across cluster] -(WorkerDict pid=1986750) The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d -(WorkerDict pid=1986750) [rank7]:[W825 17:53:25.252368567 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group() -(WorkerDict pid=1986239) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. [repeated 7x across cluster] -(WorkerDict pid=1986239) Loading weights: 0%| | 0/760 [00:00:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. -(WorkerDict pid=1986747) :1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. -(WorkerDict pid=1986749) `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead. [repeated 7x across cluster] -(WorkerDict pid=1986239) 2026-08-25 17:54:33,940 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ... -(WorkerDict pid=1986239) 2026-08-25 17:54:33,958 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends -(WorkerDict pid=1986748) :1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. [repeated 7x across cluster] -(WorkerDict pid=1986748) :1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. [repeated 7x across cluster] -(WorkerDict pid=1986239) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/51 [00:00, transformer_layer_cls={, }). -(WorkerDict pid=1986239) After FSDP module init: 15.42 GB / 95.00 GB. -(WorkerDict pid=1986239) After optimizer init: 15.42 GB / 95.00 GB. -(WorkerDict pid=1986744) Sampling params: {'max_tokens': 4096, 'detokenize': False, 'logit_bias': {248056: -100, 248057: -100}, 'n': 8, 'temperature': 1.0, 'top_p': 0.85, 'top_k': -1, 'ignore_eos': False}. -(WorkerDict pid=1986239) After vllm init: 7.76 GB / 95.00 GB. -(Runner pid=1984436) Config -(Runner pid=1984436) algorithm: -(Runner pid=1984436) adv_estimator: grpo -(Runner pid=1984436) disable_kl: true -(Runner pid=1984436) gamma: 1.0 -(Runner pid=1984436) kl_coef: 0.0 -(Runner pid=1984436) kl_horizon: 10000.0 -(Runner pid=1984436) kl_penalty: low_var_kl -(Runner pid=1984436) kl_target: 0.1 -(Runner pid=1984436) kl_type: fixed -(Runner pid=1984436) lam: 1.0 -(Runner pid=1984436) name: grpo -(Runner pid=1984436) scale_rewards: true -(Runner pid=1984436) use_kl_loss: false -(Runner pid=1984436) data: -(Runner pid=1984436) answer_key: answer -(Runner pid=1984436) dataloader_num_workers: 1 -(Runner pid=1984436) enable_thinking: false -(Runner pid=1984436) filter_overlong_prompts: false -(Runner pid=1984436) filter_overlong_prompts_workers: 16 -(Runner pid=1984436) format_prompt: null -(Runner pid=1984436) group_by_task: true -(Runner pid=1984436) group_by_task_key: problem_type -(Runner pid=1984436) image_dir: / -(Runner pid=1984436) image_key: images -(Runner pid=1984436) image_max_pixels: 1048576 -(Runner pid=1984436) image_min_pixels: 4096 -(Runner pid=1984436) inline_video_tensors: true -(Runner pid=1984436) max_pixels: 4194304 -(Runner pid=1984436) max_prompt_length: 24576 -(Runner pid=1984436) max_response_length: 4096 -(Runner pid=1984436) min_pixels: 4096 -(Runner pid=1984436) mini_rollout_batch_size: null -(Runner pid=1984436) override_chat_template: null -(Runner pid=1984436) preprocessed_video_dir: null -(Runner pid=1984436) prompt_key: problem -(Runner pid=1984436) response_prefix: '' -(Runner pid=1984436) rollout_batch_size: 8 -(Runner pid=1984436) seed: 42 -(Runner pid=1984436) shuffle: true -(Runner pid=1984436) train_files: /mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_joint_100k_maskaware.jsonl -(Runner pid=1984436) use_preprocessed_videos: false -(Runner pid=1984436) val_batch_size: 16 -(Runner pid=1984436) val_files: /mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_phase3_49152.canary.jsonl -(Runner pid=1984436) val_preprocessed_video_dir: null -(Runner pid=1984436) val_video_fps: 2.0 -(Runner pid=1984436) val_video_max_frames: 128 -(Runner pid=1984436) val_video_max_pixels: 786432 -(Runner pid=1984436) val_video_min_pixels: 4096 -(Runner pid=1984436) val_video_source_mode: realtime_only -(Runner pid=1984436) val_video_total_pixels: 8388608 -(Runner pid=1984436) video_fps: 2.0 -(Runner pid=1984436) video_key: videos -(Runner pid=1984436) video_max_frames: 128 -(Runner pid=1984436) video_max_pixels: 786432 -(Runner pid=1984436) video_min_pixels: 4096 -(Runner pid=1984436) video_source_mode: realtime_only -(Runner pid=1984436) video_total_pixels: 8388608 -(Runner pid=1984436) trainer: -(Runner pid=1984436) best_train_metric_key: reward/overall -(Runner pid=1984436) best_train_min_step: 10 -(Runner pid=1984436) best_train_smooth_window: 5 -(Runner pid=1984436) critic_warmup: 0 -(Runner pid=1984436) experiment_name: smoke-grpo-9b -(Runner pid=1984436) find_last_checkpoint: false -(Runner pid=1984436) keep_best_train_ckpt: false -(Runner pid=1984436) keep_optim_only_latest: true -(Runner pid=1984436) load_checkpoint_path: null -(Runner pid=1984436) logger: -(Runner pid=1984436) - console -(Runner pid=1984436) max_steps: 1 -(Runner pid=1984436) n_gpus_per_node: 8 -(Runner pid=1984436) nnodes: 1 -(Runner pid=1984436) project_name: OraRL -(Runner pid=1984436) ray_timeline: null -(Runner pid=1984436) save_checkpoint_path: /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175213/grpo -(Runner pid=1984436) save_freq: 1 -(Runner pid=1984436) save_limit: 3 -(Runner pid=1984436) save_model_only: false -(Runner pid=1984436) total_epochs: 1 -(Runner pid=1984436) val_before_train: false -(Runner pid=1984436) val_freq: -1 -(Runner pid=1984436) val_generations_to_log: 0 -(Runner pid=1984436) val_only: false -(Runner pid=1984436) worker: -(Runner pid=1984436) actor: -(Runner pid=1984436) clip_ratio_dual: 3.0 -(Runner pid=1984436) clip_ratio_high: 0.2 -(Runner pid=1984436) clip_ratio_low: 0.2 -(Runner pid=1984436) disable_kl: true -(Runner pid=1984436) dynamic_batching: true -(Runner pid=1984436) fsdp: -(Runner pid=1984436) enable_cpu_offload: false -(Runner pid=1984436) enable_full_shard: true -(Runner pid=1984436) enable_rank0_init: true -(Runner pid=1984436) fsdp_size: -1 -(Runner pid=1984436) mp_buffer_dtype: fp32 -(Runner pid=1984436) mp_param_dtype: bf16 -(Runner pid=1984436) mp_reduce_dtype: fp32 -(Runner pid=1984436) torch_dtype: bf16 -(Runner pid=1984436) use_orig_params: false -(Runner pid=1984436) global_batch_size: 8 -(Runner pid=1984436) global_batch_size_per_device: -1 -(Runner pid=1984436) kl_coef: 0.0 -(Runner pid=1984436) kl_penalty: low_var_kl -(Runner pid=1984436) loss_avg_mode: token -(Runner pid=1984436) loss_type: default -(Runner pid=1984436) max_grad_norm: 1.0 -(Runner pid=1984436) max_token_len_per_gpu: 32768 -(Runner pid=1984436) micro_batch_size_per_device_for_experience: 1 -(Runner pid=1984436) micro_batch_size_per_device_for_update: 1 -(Runner pid=1984436) model: -(Runner pid=1984436) enable_gradient_checkpointing: true -(Runner pid=1984436) freeze_vision_tower: true -(Runner pid=1984436) model_path: /apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B -(Runner pid=1984436) override_config: {} -(Runner pid=1984436) tokenizer_path: /apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B -(Runner pid=1984436) train_vision_merger: false -(Runner pid=1984436) trust_remote_code: false -(Runner pid=1984436) offload: -(Runner pid=1984436) offload_optimizer: false -(Runner pid=1984436) offload_params: false -(Runner pid=1984436) optim: -(Runner pid=1984436) betas: -(Runner pid=1984436) - 0.9 -(Runner pid=1984436) - 0.999 -(Runner pid=1984436) lr: 1.0e-06 -(Runner pid=1984436) lr_scheduler_type: constant -(Runner pid=1984436) lr_warmup_ratio: 0.0 -(Runner pid=1984436) lr_warmup_steps: null -(Runner pid=1984436) min_lr_ratio: null -(Runner pid=1984436) strategy: adamw -(Runner pid=1984436) training_steps: 1 -(Runner pid=1984436) weight_decay: 0.0 -(Runner pid=1984436) padding_free: true -(Runner pid=1984436) ppo_epochs: 1 -(Runner pid=1984436) strategy: fsdp -(Runner pid=1984436) ulysses_size: 1 -(Runner pid=1984436) use_kl_loss: false -(Runner pid=1984436) use_torch_compile: false -(Runner pid=1984436) critic: -(Runner pid=1984436) cliprange_value: 0.5 -(Runner pid=1984436) dynamic_batching: true -(Runner pid=1984436) fsdp: -(Runner pid=1984436) enable_cpu_offload: false -(Runner pid=1984436) enable_full_shard: true -(Runner pid=1984436) enable_rank0_init: true -(Runner pid=1984436) fsdp_size: -1 -(Runner pid=1984436) mp_buffer_dtype: fp32 -(Runner pid=1984436) mp_param_dtype: bf16 -(Runner pid=1984436) mp_reduce_dtype: fp32 -(Runner pid=1984436) torch_dtype: null -(Runner pid=1984436) use_orig_params: false -(Runner pid=1984436) global_batch_size: 256 -(Runner pid=1984436) global_batch_size_per_device: -1 -(Runner pid=1984436) loss_avg_mode: token -(Runner pid=1984436) max_grad_norm: 1.0 -(Runner pid=1984436) max_token_len_per_gpu: null -(Runner pid=1984436) micro_batch_size_per_device_for_experience: 16 -(Runner pid=1984436) micro_batch_size_per_device_for_update: 4 -(Runner pid=1984436) model: -(Runner pid=1984436) enable_gradient_checkpointing: true -(Runner pid=1984436) freeze_vision_tower: false -(Runner pid=1984436) model_path: null -(Runner pid=1984436) override_config: {} -(Runner pid=1984436) tokenizer_path: null -(Runner pid=1984436) train_vision_merger: false -(Runner pid=1984436) trust_remote_code: true -(Runner pid=1984436) offload: -(Runner pid=1984436) offload_optimizer: false -(Runner pid=1984436) offload_params: false -(Runner pid=1984436) optim: -(Runner pid=1984436) betas: -(Runner pid=1984436) - 0.9 -(Runner pid=1984436) - 0.999 -(Runner pid=1984436) lr: 1.0e-06 -(Runner pid=1984436) lr_scheduler_type: constant -(Runner pid=1984436) lr_warmup_ratio: 0.0 -(Runner pid=1984436) lr_warmup_steps: null -(Runner pid=1984436) min_lr_ratio: null -(Runner pid=1984436) strategy: adamw -(Runner pid=1984436) training_steps: 1 -(Runner pid=1984436) weight_decay: 0.01 -(Runner pid=1984436) padding_free: false -(Runner pid=1984436) ppo_epochs: 1 -(Runner pid=1984436) strategy: fsdp -(Runner pid=1984436) ulysses_size: 1 -(Runner pid=1984436) hybrid_engine: true -(Runner pid=1984436) ref: -(Runner pid=1984436) dynamic_batching: true -(Runner pid=1984436) fsdp: -(Runner pid=1984436) enable_cpu_offload: false -(Runner pid=1984436) enable_full_shard: true -(Runner pid=1984436) enable_rank0_init: true -(Runner pid=1984436) fsdp_size: -1 -(Runner pid=1984436) mp_buffer_dtype: fp32 -(Runner pid=1984436) mp_param_dtype: bf16 -(Runner pid=1984436) mp_reduce_dtype: fp32 -(Runner pid=1984436) torch_dtype: bf16 -(Runner pid=1984436) use_orig_params: false -(Runner pid=1984436) max_token_len_per_gpu: 32768 -(Runner pid=1984436) micro_batch_size_per_device_for_experience: 1 -(Runner pid=1984436) offload: -(Runner pid=1984436) offload_optimizer: false -(Runner pid=1984436) offload_params: false -(Runner pid=1984436) padding_free: true -(Runner pid=1984436) strategy: fsdp -(Runner pid=1984436) ulysses_size: 1 -(Runner pid=1984436) use_torch_compile: false -(Runner pid=1984436) reward: -(Runner pid=1984436) num_cpus: 1 -(Runner pid=1984436) reward_function: orarl.rewards -(Runner pid=1984436) reward_function_is_module: true -(Runner pid=1984436) reward_function_kwargs: {} -(Runner pid=1984436) reward_function_name: compute_score -(Runner pid=1984436) skip_special_tokens: true -(Runner pid=1984436) rollout: -(Runner pid=1984436) calculate_log_probs: false -(Runner pid=1984436) collect_seq_logprob_for_filter: false -(Runner pid=1984436) disable_log_stats: true -(Runner pid=1984436) disable_tqdm: true -(Runner pid=1984436) dtype: bf16 -(Runner pid=1984436) enable_chunked_prefill: true -(Runner pid=1984436) enforce_eager: false -(Runner pid=1984436) gpu_memory_utilization: 0.45 -(Runner pid=1984436) ignore_eos: false -(Runner pid=1984436) kv_cache_dtype: auto -(Runner pid=1984436) limit_images: 10 -(Runner pid=1984436) max_model_len: null -(Runner pid=1984436) max_num_batched_tokens: 32768 -(Runner pid=1984436) n: 8 -(Runner pid=1984436) name: vllm -(Runner pid=1984436) prompt_length: 24576 -(Runner pid=1984436) response_length: 4096 -(Runner pid=1984436) seed: 42 -(Runner pid=1984436) temperature: 1.0 -(Runner pid=1984436) tensor_parallel_size: 1 -(Runner pid=1984436) top_k: -1 -(Runner pid=1984436) top_p: 0.85 -(Runner pid=1984436) trust_remote_code: false -(Runner pid=1984436) val_override_config: -(Runner pid=1984436) n: 1 -(Runner pid=1984436) temperature: 0.7 -(Runner pid=1984436) top_p: 0.95 -(Runner pid=1984436) - (pid=1984436) Running step: 0%| | 0.00/1.00 [00:00", line 198, in _run_module_as_main - File "", line 88, in _run_code - File "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/trainer/main.py", line 137, in - main() - File "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/trainer/main.py", line 129, in main - ray.get(runner.run.remote(ppo_config)) - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper - return fn(*args, **kwargs) - ^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper - return func(*args, **kwargs) - ^^^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/ray/_private/worker.py", line 2981, in get - values, debugger_breakpoint = worker.get_objects( - ^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/ray/_private/worker.py", line 1012, in get_objects - raise value.as_instanceof_cause() -ray.exceptions.RayTaskError(IndexError): ray::Runner.run() (pid=1984436, ip=29.127.69.4, actor_id=2700fe7dc5d33e51d56c429c01000000, repr=) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/trainer/main.py", line 92, in run - trainer.fit() - File "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/trainer/ray_trainer.py", line 1870, in fit - batch = self._make_batch_data(metrics=metrics) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/trainer/ray_trainer.py", line 1662, in _make_batch_data - batch_dict = next(self.data_iterator) - ^^^^^^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/torchdata/stateful_dataloader/stateful_dataloader.py", line 450, in __next__ - return super().__next__() - ^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__ - data = self._next_data() - ^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/torchdata/stateful_dataloader/stateful_dataloader.py", line 1456, in _next_data - return self._process_data(data, worker_id, state_dict) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/torchdata/stateful_dataloader/stateful_dataloader.py", line 1543, in _process_data - data.reraise() - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/torch/_utils.py", line 775, in reraise - raise exception -IndexError: Caught IndexError in DataLoader worker process 0. -Original Traceback (most recent call last): - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/torchdata/stateful_dataloader/worker.py", line 242, in _worker_loop - data = fetcher.fetch(index) # type: ignore[union-attr] - ^^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch - data = [self.dataset[idx] for idx in possibly_batched_index] - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 54, in - data = [self.dataset[idx] for idx in possibly_batched_index] - ~~~~~~~~~~~~^^^^^ - File "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/utils/dataset.py", line 826, in __getitem__ - model_inputs = self.processor(processed_images, [prompt], add_special_tokens=False, return_tensors="pt") - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/models/qwen3_vl/processing_qwen3_vl.py", line 132, in __call__ - num_image_tokens = image_grid_thw[index].prod() // merge_length - ~~~~~~~~~~~~~~^^^^^^^ -IndexError: index 8 is out of bounds for dimension 0 with size 8 - (Runner pid=1984436) qwen-vl-utils using decord to read video. - (pid=1984436) Running step: 0%| | 0.00/1.00 [00:02<|image_pad|><|vision_end|>' }} - {%- elif 'video' in item or item.type == 'video' %} - {%- if is_system_content %} - {{- raise_exception('System message cannot contain videos.') }} - {%- endif %} - {%- if do_vision_count %} - {%- set video_count.value = video_count.value + 1 %} - {%- endif %} - {%- if add_vision_id %} - {{- 'Video ' ~ video_count.value ~ ': ' }} - {%- endif %} - {{- '<|vision_start|><|video_pad|><|vision_end|>' }} - {%- elif 'text' in item %} - {{- item.text }} - {%- else %} - {{- raise_exception('Unexpected item type in content.') }} - {%- endif %} - {%- endfor %} - {%- elif content is none or content is undefined %} - {{- '' }} - {%- else %} - {{- raise_exception('Unexpected content type.') }} - {%- endif %} -{%- endmacro %} -{%- if not messages %} - {{- raise_exception('No messages provided.') }} -{%- endif %} -{%- if tools and tools is iterable and tools is not mapping %} - {{- '<|im_start|>system\n' }} - {{- "# Tools\n\nYou have access to the following functions:\n\n" }} - {%- for tool in tools %} - {{- "\n" }} - {{- tool | tojson }} - {%- endfor %} - {{- "\n" }} - {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n\n\n\nvalue_1\n\n\nThis is the value for the second parameter\nthat can span\nmultiple lines\n\n\n\n\n\nReminder:\n- Function calls MUST follow the specified format: an inner block must be nested within XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n' }} - {%- if messages[0].role == 'system' %} - {%- set content = render_content(messages[0].content, false, true)|trim %} - {%- if content %} - {{- '\n\n' + content }} - {%- endif %} - {%- endif %} - {{- '<|im_end|>\n' }} -{%- else %} - {%- if messages[0].role == 'system' %} - {%- set content = render_content(messages[0].content, false, true)|trim %} - {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }} - {%- endif %} -{%- endif %} -{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} -{%- for message in messages[::-1] %} - {%- set index = (messages|length - 1) - loop.index0 %} - {%- if ns.multi_step_tool and message.role == "user" %} - {%- set content = render_content(message.content, false)|trim %} - {%- if not(content.startswith('') and content.endswith('')) %} - {%- set ns.multi_step_tool = false %} - {%- set ns.last_query_index = index %} - {%- endif %} - {%- endif %} -{%- endfor %} -{%- if ns.multi_step_tool %} - {{- raise_exception('No user query found in messages.') }} -{%- endif %} -{%- for message in messages %} - {%- set content = render_content(message.content, true)|trim %} - {%- if message.role == "system" %} - {%- if not loop.first %} - {{- raise_exception('System message must be at the beginning.') }} - {%- endif %} - {%- elif message.role == "user" %} - {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} - {%- elif message.role == "assistant" %} - {%- set reasoning_content = '' %} - {%- if message.reasoning_content is string %} - {%- set reasoning_content = message.reasoning_content %} - {%- else %} - {%- if '' in content %} - {%- set reasoning_content = content.split('')[0].rstrip('\n').split('')[-1].lstrip('\n') %} - {%- set content = content.split('')[-1].lstrip('\n') %} - {%- endif %} - {%- endif %} - {%- set reasoning_content = reasoning_content|trim %} - {%- if loop.index0 > ns.last_query_index %} - {{- '<|im_start|>' + message.role + '\n\n' + reasoning_content + '\n\n\n' + content }} - {%- else %} - {{- '<|im_start|>' + message.role + '\n' + content }} - {%- endif %} - {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %} - {%- for tool_call in message.tool_calls %} - {%- if tool_call.function is defined %} - {%- set tool_call = tool_call.function %} - {%- endif %} - {%- if loop.first %} - {%- if content|trim %} - {{- '\n\n\n\n' }} - {%- else %} - {{- '\n\n' }} - {%- endif %} - {%- else %} - {{- '\n\n\n' }} - {%- endif %} - {%- if tool_call.arguments is defined %} - {%- for args_name, args_value in tool_call.arguments|items %} - {{- '\n' }} - {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %} - {{- args_value }} - {{- '\n\n' }} - {%- endfor %} - {%- endif %} - {{- '\n' }} - {%- endfor %} - {%- endif %} - {{- '<|im_end|>\n' }} - {%- elif message.role == "tool" %} - {%- if loop.previtem and loop.previtem.role != "tool" %} - {{- '<|im_start|>user' }} - {%- endif %} - {{- '\n\n' }} - {{- content }} - {{- '\n' }} - {%- if not loop.last and loop.nextitem.role != "tool" %} - {{- '<|im_end|>\n' }} - {%- elif loop.last %} - {{- '<|im_end|>\n' }} - {%- endif %} - {%- else %} - {{- raise_exception('Unexpected message role.') }} - {%- endif %} -{%- endfor %} -{%- if add_generation_prompt %} - {{- '<|im_start|>assistant\n' }} - {%- if enable_thinking is defined and enable_thinking is false %} - {{- '\n\n\n\n' }} - {%- else %} - {{- '\n' }} - {%- endif %} -{%- endif %} \ No newline at end of file diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/config.json b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/config.json deleted file mode 100644 index d13e73613c775be9cd78fc28ff9c7aa65090debc..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/config.json +++ /dev/null @@ -1,109 +0,0 @@ -{ - "architectures": [ - "Qwen3_5ForConditionalGeneration" - ], - "dtype": "bfloat16", - "image_token_id": 248056, - "model_type": "qwen3_5", - "text_config": { - "attention_bias": false, - "attention_dropout": 0.0, - "attn_output_gate": true, - "bos_token_id": null, - "dtype": "bfloat16", - "eos_token_id": 248044, - "full_attention_interval": 4, - "head_dim": 256, - "hidden_act": "silu", - "hidden_size": 4096, - "initializer_range": 0.02, - "intermediate_size": 12288, - "layer_types": [ - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention" - ], - "linear_conv_kernel_dim": 4, - "linear_key_head_dim": 128, - "linear_num_key_heads": 16, - "linear_num_value_heads": 32, - "linear_value_head_dim": 128, - "mamba_ssm_dtype": "float32", - "max_position_embeddings": 262144, - "mlp_only_layers": [], - "model_type": "qwen3_5_text", - "mtp_num_hidden_layers": 0, - "mtp_use_dedicated_embeddings": false, - "num_attention_heads": 16, - "num_hidden_layers": 32, - "num_key_value_heads": 4, - "pad_token_id": null, - "partial_rotary_factor": 0.25, - "rms_norm_eps": 1e-06, - "rope_parameters": { - "mrope_interleaved": true, - "mrope_section": [ - 11, - 11, - 10 - ], - "partial_rotary_factor": 0.25, - "rope_theta": 10000000, - "rope_type": "default" - }, - "tie_word_embeddings": false, - "use_cache": true, - "vocab_size": 248320 - }, - "tie_word_embeddings": false, - "transformers_version": "5.5.4", - "video_token_id": 248057, - "vision_config": { - "deepstack_visual_indexes": [], - "depth": 27, - "dtype": "bfloat16", - "hidden_act": "gelu_pytorch_tanh", - "hidden_size": 1152, - "in_channels": 3, - "initializer_range": 0.02, - "intermediate_size": 4304, - "model_type": "qwen3_5", - "num_heads": 16, - "num_position_embeddings": 2304, - "out_hidden_size": 4096, - "patch_size": 16, - "spatial_merge_size": 2, - "temporal_patch_size": 2 - }, - "vision_end_token_id": 248054, - "vision_start_token_id": 248053 -} diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/generation_config.json b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/generation_config.json deleted file mode 100644 index 23a0a961f467960631ac5faedc9547a7523a99c3..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/generation_config.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "bos_token_id": 248044, - "do_sample": true, - "eos_token_id": [ - 248046, - 248044 - ], - "pad_token_id": 248044, - "temperature": 1.0, - "top_k": 20, - "top_p": 0.95, - "transformers_version": "5.5.4" -} diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/processor_config.json b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/processor_config.json deleted file mode 100644 index 33818c7f9e991ad735fd240209f4fa73e6c28c50..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/processor_config.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "image_processor": { - "do_convert_rgb": true, - "do_normalize": true, - "do_rescale": true, - "do_resize": true, - "image_mean": [ - 0.5, - 0.5, - 0.5 - ], - "image_processor_type": "Qwen2VLImageProcessor", - "image_std": [ - 0.5, - 0.5, - 0.5 - ], - "merge_size": 2, - "patch_size": 16, - "resample": 3, - "rescale_factor": 0.00392156862745098, - "size": { - "longest_edge": 16777216, - "shortest_edge": 65536 - }, - "temporal_patch_size": 2 - }, - "processor_class": "Qwen3VLProcessor", - "video_processor": { - "do_convert_rgb": true, - "do_normalize": true, - "do_rescale": true, - "do_resize": true, - "do_sample_frames": true, - "fps": 2, - "image_mean": [ - 0.5, - 0.5, - 0.5 - ], - "image_std": [ - 0.5, - 0.5, - 0.5 - ], - "max_frames": 768, - "merge_size": 2, - "min_frames": 4, - "patch_size": 16, - "resample": 3, - "rescale_factor": 0.00392156862745098, - "return_metadata": false, - "size": { - "longest_edge": 25165824, - "shortest_edge": 4096 - }, - "temporal_patch_size": 2, - "video_processor_type": "Qwen3VLVideoProcessor" - } -} diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/tokenizer.json b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/tokenizer.json deleted file mode 100644 index 67741b04f23bfdb46501f748ce27865ec82eccfb..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/tokenizer.json +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:87a7830d63fcf43bf241c3c5242e96e62dd3fdc29224ca26fed8ea333db72de4 -size 19989343 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/tokenizer_config.json b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/tokenizer_config.json deleted file mode 100644 index a068e2468cff426a9b105006e74e044030a6faf4..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/huggingface/tokenizer_config.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "add_prefix_space": false, - "audio_bos_token": "<|audio_start|>", - "audio_eos_token": "<|audio_end|>", - "audio_token": "<|audio_pad|>", - "backend": "tokenizers", - "bos_token": null, - "clean_up_tokenization_spaces": false, - "eos_token": "<|im_end|>", - "errors": "replace", - "image_token": "<|image_pad|>", - "is_local": true, - "model_max_length": 262144, - "model_specific_special_tokens": { - "audio_bos_token": "<|audio_start|>", - "audio_eos_token": "<|audio_end|>", - "audio_token": "<|audio_pad|>", - "image_token": "<|image_pad|>", - "video_token": "<|video_pad|>", - "vision_bos_token": "<|vision_start|>", - "vision_eos_token": "<|vision_end|>" - }, - "pad_token": "<|endoftext|>", - "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+", - "processor_class": "Qwen3VLProcessor", - "split_special_tokens": false, - "tokenizer_class": "TokenizersBackend", - "unk_token": null, - "video_token": "<|video_pad|>", - "vision_bos_token": "<|vision_start|>", - "vision_eos_token": "<|vision_end|>" -} diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_0.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_0.pt deleted file mode 100644 index 880f25f27588d18338bad74731696b06ad1d5069..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_0.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e7a112d3ef8e9efd0c7404343835e8d1cacb68f1bf47ec9e70a6035d628bfeae -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_1.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_1.pt deleted file mode 100644 index 8ef782c0875bd58d1b344b7b9210181f0a73c911..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_1.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:929bde074b4648a5e40ea9d59f1b37f587f605044b3013c2d9663496ff270997 -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_2.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_2.pt deleted file mode 100644 index f058aa02326eebbfdd6ada84a9fca495c4e8d8c2..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_2.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:5f191a9b777c259be21b12b8ed02585b790915f74c547f0ac2600a45ce3a598a -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_3.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_3.pt deleted file mode 100644 index 4eaa7a0a3388cef91ac20367d03a2844da7264ef..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_3.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ff8e73e975bbe1ea882acbebe7533b050a9c127beb1c39b8a5d3cfd72ffbc17c -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_4.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_4.pt deleted file mode 100644 index 43b28746068bd1bc4f689d867d183ee0a6b2d845..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_4.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:49a0e15c74448d5debb04be1f773463910f08df82e32e33fc164e1e674a1d3e2 -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_5.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_5.pt deleted file mode 100644 index 99a1dcf49b6a2a434da998b7c75ed95ca8cf2f84..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_5.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:84a14b9ff89d4c3769d06de5e98af7f71b0b2c7d82226c3e9761d37af480a7db -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_6.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_6.pt deleted file mode 100644 index 93876eceb78280fcfdeebc8a43cb000e143864b7..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_6.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:53206b46e825c183bee092c93d115c7863e368745f0b4a9340f477845fff6521 -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_7.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_7.pt deleted file mode 100644 index 6dad326e8199db4e1d35257f90ea26820f3d8378..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/model_world_size_8_rank_7.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:5f69907e184a8e128ebd0087e9e47ebd62f2cfa47dd93b356b2b03df20317cda -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_0.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_0.pt deleted file mode 100644 index 96582c2dad336962a44f62e4cac0203859483a87..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_0.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:dc1018349737fd19af9fe2ae532ee8aed53a1cd2a6289bd51efc0e893092b5e7 -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_1.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_1.pt deleted file mode 100644 index f62cef8911fdf35efc7061d2221bc4606731850c..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_1.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:72b7df9ff763ae286ab239e33b2d6711bff9f19a16f9043418b8c68accc08cb2 -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_2.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_2.pt deleted file mode 100644 index 09b9bd2a1e73a35547da89bba6ad3648bb0920ad..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_2.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:fad4ce51bcfbc75a0bd746f41af31c8c44cd57974acc1a36adcc4153bf303a6b -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_3.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_3.pt deleted file mode 100644 index aa6b5487fb06482430ec76e98a0654e0605b4cab..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_3.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:11ed1bcf8aa7199792a3f094d8844b6840cdd53efb3fcdfd7f1ea6931b07ea75 -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_4.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_4.pt deleted file mode 100644 index 2b5480769f41a56f2df48815aec56d3e40ce5384..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_4.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ccb3231b0406316170091c991dd49652d43e89bcc3f8544cbe436742baf2e960 -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_5.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_5.pt deleted file mode 100644 index 2b57e8782e887d1442ec5b5c931eb9ad2249a8e6..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_5.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:8b1e07b9349724c6f93fd95d24146c217483921a201c48db3cd1742e22527b70 -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_6.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_6.pt deleted file mode 100644 index 5e407504f5f07f27deaa2e30082b11fd8111c5f1..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_6.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:92c3605f87040e3ee654db866052d94620101854bc2ee697c10a0536bc5b3dbb -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_7.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_7.pt deleted file mode 100644 index 4829ff68eb7dff55f11034b8df1e65bbfc6769c0..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/actor/optim_world_size_8_rank_7.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:c2f8b77f8faef58fce850c407f800d2cd0e7c9e4388e7c7a08f6465357fb9891 -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/dataloader.pt b/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/dataloader.pt deleted file mode 100644 index 237b2a50543b7937e88f70166e880f4e7a02342f..0000000000000000000000000000000000000000 Binary files a/code/runs/smoke-training-9b-20260825_175949/grpo/global_step_1/dataloader.pt and /dev/null differ diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/checkpoint_tracker.json b/code/runs/smoke-training-9b-20260825_175949/orarl/checkpoint_tracker.json deleted file mode 100644 index 89e57145ec755ed0b11d88f01c9cda722eb0237d..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/checkpoint_tracker.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "best_global_step": null, - "best_val_reward_score": -1.0, - "last_global_step": 1, - "last_actor_path": "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor" -} \ No newline at end of file diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_0.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_0.pt deleted file mode 100644 index 9ddf9fa959f9f7e04210a74d1c92d33701d6b33e..0000000000000000000000000000000000000000 Binary files a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_0.pt and /dev/null differ diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_1.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_1.pt deleted file mode 100644 index ff1c6d6f862719840b74d28237a295340dabc090..0000000000000000000000000000000000000000 Binary files a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_1.pt and /dev/null differ diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_2.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_2.pt deleted file mode 100644 index 36e5d1767618d404ae9ef7a17bb7ab62bce1e0ac..0000000000000000000000000000000000000000 Binary files a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_2.pt and /dev/null differ diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_3.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_3.pt deleted file mode 100644 index e8e74bd7a72db76984c7c1d48e12ef2c44d43042..0000000000000000000000000000000000000000 Binary files a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_3.pt and /dev/null differ diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_4.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_4.pt deleted file mode 100644 index f02841fbe96b46e5590e8a124c6636fcc2848b80..0000000000000000000000000000000000000000 Binary files a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_4.pt and /dev/null differ diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_5.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_5.pt deleted file mode 100644 index 09e2a3f4856e2e4e56c7538dfc8eb632e89f0987..0000000000000000000000000000000000000000 Binary files a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_5.pt and /dev/null differ diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_6.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_6.pt deleted file mode 100644 index 63eb7ddfc6d3921af129980eaa14baed168b2dc4..0000000000000000000000000000000000000000 Binary files a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_6.pt and /dev/null differ diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_7.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_7.pt deleted file mode 100644 index 7f00686a15d8d21702a4c9063f36fcc958eb18ba..0000000000000000000000000000000000000000 Binary files a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/extra_state_world_size_8_rank_7.pt and /dev/null differ diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/chat_template.jinja b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/chat_template.jinja deleted file mode 100644 index a585dec894e63da457d9440ec6aa7caa16d20860..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/chat_template.jinja +++ /dev/null @@ -1,154 +0,0 @@ -{%- set image_count = namespace(value=0) %} -{%- set video_count = namespace(value=0) %} -{%- macro render_content(content, do_vision_count, is_system_content=false) %} - {%- if content is string %} - {{- content }} - {%- elif content is iterable and content is not mapping %} - {%- for item in content %} - {%- if 'image' in item or 'image_url' in item or item.type == 'image' %} - {%- if is_system_content %} - {{- raise_exception('System message cannot contain images.') }} - {%- endif %} - {%- if do_vision_count %} - {%- set image_count.value = image_count.value + 1 %} - {%- endif %} - {%- if add_vision_id %} - {{- 'Picture ' ~ image_count.value ~ ': ' }} - {%- endif %} - {{- '<|vision_start|><|image_pad|><|vision_end|>' }} - {%- elif 'video' in item or item.type == 'video' %} - {%- if is_system_content %} - {{- raise_exception('System message cannot contain videos.') }} - {%- endif %} - {%- if do_vision_count %} - {%- set video_count.value = video_count.value + 1 %} - {%- endif %} - {%- if add_vision_id %} - {{- 'Video ' ~ video_count.value ~ ': ' }} - {%- endif %} - {{- '<|vision_start|><|video_pad|><|vision_end|>' }} - {%- elif 'text' in item %} - {{- item.text }} - {%- else %} - {{- raise_exception('Unexpected item type in content.') }} - {%- endif %} - {%- endfor %} - {%- elif content is none or content is undefined %} - {{- '' }} - {%- else %} - {{- raise_exception('Unexpected content type.') }} - {%- endif %} -{%- endmacro %} -{%- if not messages %} - {{- raise_exception('No messages provided.') }} -{%- endif %} -{%- if tools and tools is iterable and tools is not mapping %} - {{- '<|im_start|>system\n' }} - {{- "# Tools\n\nYou have access to the following functions:\n\n" }} - {%- for tool in tools %} - {{- "\n" }} - {{- tool | tojson }} - {%- endfor %} - {{- "\n" }} - {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n\n\n\nvalue_1\n\n\nThis is the value for the second parameter\nthat can span\nmultiple lines\n\n\n\n\n\nReminder:\n- Function calls MUST follow the specified format: an inner block must be nested within XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n' }} - {%- if messages[0].role == 'system' %} - {%- set content = render_content(messages[0].content, false, true)|trim %} - {%- if content %} - {{- '\n\n' + content }} - {%- endif %} - {%- endif %} - {{- '<|im_end|>\n' }} -{%- else %} - {%- if messages[0].role == 'system' %} - {%- set content = render_content(messages[0].content, false, true)|trim %} - {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }} - {%- endif %} -{%- endif %} -{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} -{%- for message in messages[::-1] %} - {%- set index = (messages|length - 1) - loop.index0 %} - {%- if ns.multi_step_tool and message.role == "user" %} - {%- set content = render_content(message.content, false)|trim %} - {%- if not(content.startswith('') and content.endswith('')) %} - {%- set ns.multi_step_tool = false %} - {%- set ns.last_query_index = index %} - {%- endif %} - {%- endif %} -{%- endfor %} -{%- if ns.multi_step_tool %} - {{- raise_exception('No user query found in messages.') }} -{%- endif %} -{%- for message in messages %} - {%- set content = render_content(message.content, true)|trim %} - {%- if message.role == "system" %} - {%- if not loop.first %} - {{- raise_exception('System message must be at the beginning.') }} - {%- endif %} - {%- elif message.role == "user" %} - {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} - {%- elif message.role == "assistant" %} - {%- set reasoning_content = '' %} - {%- if message.reasoning_content is string %} - {%- set reasoning_content = message.reasoning_content %} - {%- else %} - {%- if '' in content %} - {%- set reasoning_content = content.split('')[0].rstrip('\n').split('')[-1].lstrip('\n') %} - {%- set content = content.split('')[-1].lstrip('\n') %} - {%- endif %} - {%- endif %} - {%- set reasoning_content = reasoning_content|trim %} - {%- if loop.index0 > ns.last_query_index %} - {{- '<|im_start|>' + message.role + '\n\n' + reasoning_content + '\n\n\n' + content }} - {%- else %} - {{- '<|im_start|>' + message.role + '\n' + content }} - {%- endif %} - {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %} - {%- for tool_call in message.tool_calls %} - {%- if tool_call.function is defined %} - {%- set tool_call = tool_call.function %} - {%- endif %} - {%- if loop.first %} - {%- if content|trim %} - {{- '\n\n\n\n' }} - {%- else %} - {{- '\n\n' }} - {%- endif %} - {%- else %} - {{- '\n\n\n' }} - {%- endif %} - {%- if tool_call.arguments is defined %} - {%- for args_name, args_value in tool_call.arguments|items %} - {{- '\n' }} - {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %} - {{- args_value }} - {{- '\n\n' }} - {%- endfor %} - {%- endif %} - {{- '\n' }} - {%- endfor %} - {%- endif %} - {{- '<|im_end|>\n' }} - {%- elif message.role == "tool" %} - {%- if loop.previtem and loop.previtem.role != "tool" %} - {{- '<|im_start|>user' }} - {%- endif %} - {{- '\n\n' }} - {{- content }} - {{- '\n' }} - {%- if not loop.last and loop.nextitem.role != "tool" %} - {{- '<|im_end|>\n' }} - {%- elif loop.last %} - {{- '<|im_end|>\n' }} - {%- endif %} - {%- else %} - {{- raise_exception('Unexpected message role.') }} - {%- endif %} -{%- endfor %} -{%- if add_generation_prompt %} - {{- '<|im_start|>assistant\n' }} - {%- if enable_thinking is defined and enable_thinking is false %} - {{- '\n\n\n\n' }} - {%- else %} - {{- '\n' }} - {%- endif %} -{%- endif %} \ No newline at end of file diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/config.json b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/config.json deleted file mode 100644 index d13e73613c775be9cd78fc28ff9c7aa65090debc..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/config.json +++ /dev/null @@ -1,109 +0,0 @@ -{ - "architectures": [ - "Qwen3_5ForConditionalGeneration" - ], - "dtype": "bfloat16", - "image_token_id": 248056, - "model_type": "qwen3_5", - "text_config": { - "attention_bias": false, - "attention_dropout": 0.0, - "attn_output_gate": true, - "bos_token_id": null, - "dtype": "bfloat16", - "eos_token_id": 248044, - "full_attention_interval": 4, - "head_dim": 256, - "hidden_act": "silu", - "hidden_size": 4096, - "initializer_range": 0.02, - "intermediate_size": 12288, - "layer_types": [ - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention", - "linear_attention", - "linear_attention", - "linear_attention", - "full_attention" - ], - "linear_conv_kernel_dim": 4, - "linear_key_head_dim": 128, - "linear_num_key_heads": 16, - "linear_num_value_heads": 32, - "linear_value_head_dim": 128, - "mamba_ssm_dtype": "float32", - "max_position_embeddings": 262144, - "mlp_only_layers": [], - "model_type": "qwen3_5_text", - "mtp_num_hidden_layers": 0, - "mtp_use_dedicated_embeddings": false, - "num_attention_heads": 16, - "num_hidden_layers": 32, - "num_key_value_heads": 4, - "pad_token_id": null, - "partial_rotary_factor": 0.25, - "rms_norm_eps": 1e-06, - "rope_parameters": { - "mrope_interleaved": true, - "mrope_section": [ - 11, - 11, - 10 - ], - "partial_rotary_factor": 0.25, - "rope_theta": 10000000, - "rope_type": "default" - }, - "tie_word_embeddings": false, - "use_cache": true, - "vocab_size": 248320 - }, - "tie_word_embeddings": false, - "transformers_version": "5.5.4", - "video_token_id": 248057, - "vision_config": { - "deepstack_visual_indexes": [], - "depth": 27, - "dtype": "bfloat16", - "hidden_act": "gelu_pytorch_tanh", - "hidden_size": 1152, - "in_channels": 3, - "initializer_range": 0.02, - "intermediate_size": 4304, - "model_type": "qwen3_5", - "num_heads": 16, - "num_position_embeddings": 2304, - "out_hidden_size": 4096, - "patch_size": 16, - "spatial_merge_size": 2, - "temporal_patch_size": 2 - }, - "vision_end_token_id": 248054, - "vision_start_token_id": 248053 -} diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/generation_config.json b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/generation_config.json deleted file mode 100644 index 23a0a961f467960631ac5faedc9547a7523a99c3..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/generation_config.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "bos_token_id": 248044, - "do_sample": true, - "eos_token_id": [ - 248046, - 248044 - ], - "pad_token_id": 248044, - "temperature": 1.0, - "top_k": 20, - "top_p": 0.95, - "transformers_version": "5.5.4" -} diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/processor_config.json b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/processor_config.json deleted file mode 100644 index 33818c7f9e991ad735fd240209f4fa73e6c28c50..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/processor_config.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "image_processor": { - "do_convert_rgb": true, - "do_normalize": true, - "do_rescale": true, - "do_resize": true, - "image_mean": [ - 0.5, - 0.5, - 0.5 - ], - "image_processor_type": "Qwen2VLImageProcessor", - "image_std": [ - 0.5, - 0.5, - 0.5 - ], - "merge_size": 2, - "patch_size": 16, - "resample": 3, - "rescale_factor": 0.00392156862745098, - "size": { - "longest_edge": 16777216, - "shortest_edge": 65536 - }, - "temporal_patch_size": 2 - }, - "processor_class": "Qwen3VLProcessor", - "video_processor": { - "do_convert_rgb": true, - "do_normalize": true, - "do_rescale": true, - "do_resize": true, - "do_sample_frames": true, - "fps": 2, - "image_mean": [ - 0.5, - 0.5, - 0.5 - ], - "image_std": [ - 0.5, - 0.5, - 0.5 - ], - "max_frames": 768, - "merge_size": 2, - "min_frames": 4, - "patch_size": 16, - "resample": 3, - "rescale_factor": 0.00392156862745098, - "return_metadata": false, - "size": { - "longest_edge": 25165824, - "shortest_edge": 4096 - }, - "temporal_patch_size": 2, - "video_processor_type": "Qwen3VLVideoProcessor" - } -} diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/tokenizer.json b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/tokenizer.json deleted file mode 100644 index 67741b04f23bfdb46501f748ce27865ec82eccfb..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/tokenizer.json +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:87a7830d63fcf43bf241c3c5242e96e62dd3fdc29224ca26fed8ea333db72de4 -size 19989343 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/tokenizer_config.json b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/tokenizer_config.json deleted file mode 100644 index a068e2468cff426a9b105006e74e044030a6faf4..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/huggingface/tokenizer_config.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "add_prefix_space": false, - "audio_bos_token": "<|audio_start|>", - "audio_eos_token": "<|audio_end|>", - "audio_token": "<|audio_pad|>", - "backend": "tokenizers", - "bos_token": null, - "clean_up_tokenization_spaces": false, - "eos_token": "<|im_end|>", - "errors": "replace", - "image_token": "<|image_pad|>", - "is_local": true, - "model_max_length": 262144, - "model_specific_special_tokens": { - "audio_bos_token": "<|audio_start|>", - "audio_eos_token": "<|audio_end|>", - "audio_token": "<|audio_pad|>", - "image_token": "<|image_pad|>", - "video_token": "<|video_pad|>", - "vision_bos_token": "<|vision_start|>", - "vision_eos_token": "<|vision_end|>" - }, - "pad_token": "<|endoftext|>", - "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+", - "processor_class": "Qwen3VLProcessor", - "split_special_tokens": false, - "tokenizer_class": "TokenizersBackend", - "unk_token": null, - "video_token": "<|video_pad|>", - "vision_bos_token": "<|vision_start|>", - "vision_eos_token": "<|vision_end|>" -} diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_0.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_0.pt deleted file mode 100644 index 965bcc65404bdb192443f340dcb3d2ae70da0d82..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_0.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:68d916aca907a48f093fcb05501c28b5e0a46a127dae4e7077e1bf7da273a578 -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_1.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_1.pt deleted file mode 100644 index 59aac22dc11af461393131f923317783bbf79a9a..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_1.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:59a86054eef4b1dc5ad060c150e8435918f4221890f037bef4800944e065ac9c -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_2.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_2.pt deleted file mode 100644 index d61124c62cf6d85809c98cde069ab62cc93e5fbc..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_2.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:dbdea55e015c41e8b330efd62a209990d6396cc7e9bbefb3db3a01615e130094 -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_3.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_3.pt deleted file mode 100644 index b943ad5dee63423f2944d482cafc8fb16d7b28af..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_3.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:2a4b11162a92caf7e46a3df91b572fc32bd6e899d751e80b5c29e24f44bb3d3d -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_4.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_4.pt deleted file mode 100644 index 533a04be9c2909452ad93c4908a226ac8f2b0f8c..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_4.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:b967583161f9f4c2e32ad4e4e1e6143aef915a0eb86d8cc648bbba99b1110fd4 -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_5.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_5.pt deleted file mode 100644 index 94deff6a447bd4c4a12ef7e467bd95b190ebd5ed..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_5.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:681ba029790747845f2e435e9515175d2f62475f8c5df86d8cd73641c8805790 -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_6.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_6.pt deleted file mode 100644 index 8896d9b1ec7c92dd781bb0f3c16a2305db3b4313..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_6.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:3a8d4c40826eb2dd2f293b9d24132b7099c718f232badccc7595930d9567b8e8 -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_7.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_7.pt deleted file mode 100644 index 00bacbdd1dbdeaae2ff6794802f06bb5d2dd5f14..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/model_world_size_8_rank_7.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:629e7633a242b848283f4f61d2a8e63a7b3e1bc7ed1fa96ee8820c7145700c4d -size 2352797236 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_0.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_0.pt deleted file mode 100644 index f14b6d6ad7ea58e61effea441cb67d17545f3561..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_0.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9f55ce17accdc1232fa2501877d6ed35f409bd8d6dc4cc9374f193bc5623ed20 -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_1.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_1.pt deleted file mode 100644 index 4908c9a622c8f0bbe76f4102754bd7f2e9b5c71c..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_1.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4c8607d4247f9852139f11f1ac2fea7630177c0bccdb25383a75b549ebb3fe0c -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_2.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_2.pt deleted file mode 100644 index 3082d4018cd73ad883996034c5bde012cb49576f..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_2.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:783a7bdb5c7289b31bb284c1b5f4e1fbcdc14b0386e61c12d33a29a624f619e5 -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_3.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_3.pt deleted file mode 100644 index 6bf10bb33bf914ba7a4692396058035a1d445e5d..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_3.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:1e47f4381f34e93df73d5005da02800a003e906bab5ec093ec433a4b9fc0234f -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_4.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_4.pt deleted file mode 100644 index cbe647e71dd1f6074b22248d1a279e2c387fb130..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_4.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:a275a020afff2f926632a3c18dfc66d881e44a8a39922107128102d7afdfa076 -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_5.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_5.pt deleted file mode 100644 index 1e46e5c6f599fd7888466d152a8665715293600c..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_5.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:f1fb064bf27a05dd600fcac0bf25685dae85254178e71fc05e7413e547a415cb -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_6.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_6.pt deleted file mode 100644 index c8e0a2d15efe3f25b6cbc6ec894443c9489707c7..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_6.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:719541230bc7a694e4f98a0162f19b5bf70adabd627e43df1a7cb3832661f45d -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_7.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_7.pt deleted file mode 100644 index 16bc4663efd6064401fa86e7ae3bb19bc6488665..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/actor/optim_world_size_8_rank_7.pt +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:53aabad2855f21fb69be25b681885b9c45d2cbbe9ce0ab8bd077f9f81be6b6b5 -size 4477429805 diff --git a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/dataloader.pt b/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/dataloader.pt deleted file mode 100644 index 237b2a50543b7937e88f70166e880f4e7a02342f..0000000000000000000000000000000000000000 Binary files a/code/runs/smoke-training-9b-20260825_175949/orarl/global_step_1/dataloader.pt and /dev/null differ diff --git a/code/runs/smoke-training-9b-20260825_175949/smoke.log b/code/runs/smoke-training-9b-20260825_175949/smoke.log deleted file mode 100644 index 6c47f32837c70e758a87251ad7564cfeba5494e0..0000000000000000000000000000000000000000 --- a/code/runs/smoke-training-9b-20260825_175949/smoke.log +++ /dev/null @@ -1,5748 +0,0 @@ -OraRL root: /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official -Python: /data/miniconda3/envs/orarl/bin/python -Model: /apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B -Train data: /mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_joint_100k_maskaware.jsonl -Val data: /mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_phase3_49152.canary.jsonl -Recipe scale: 9b -World size: 1 x 8 -Steps per run: 1 -Results: /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175949 -Execute: 1 -0, NVIDIA H20, 0 MiB, 97871 MiB -1, NVIDIA H20, 0 MiB, 97871 MiB -2, NVIDIA H20, 0 MiB, 97871 MiB -3, NVIDIA H20, 0 MiB, 97871 MiB -4, NVIDIA H20, 0 MiB, 97871 MiB -5, NVIDIA H20, 0 MiB, 97871 MiB -6, NVIDIA H20, 0 MiB, 97871 MiB -7, NVIDIA H20, 0 MiB, 97871 MiB -/data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. - warnings.warn( -dependency OK: numpy 2.2.6 -dependency OK: torch 2.10.0+cu129 -dependency OK: transformers 5.5.4 -dependency OK: vllm 0.19.1 -dependency OK: ray 2.54.0 -dependency OK: tensordict 0.12.2 -dependency OK: codetiming 1.4.0 -dependency OK: omegaconf 2.3.0 -dependency OK: verl.trainer.main /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/verl/trainer/main.py -dependency OK: orarl.rewards /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/orarl/rewards/__init__.py - -=== grpo smoke (1 update(s)) === -/data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. - warnings.warn( -2026-08-25 18:00:17,309 INFO worker.py:2004 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265  -/data/miniconda3/envs/orarl/lib/python3.11/site-packages/ray/_private/worker.py:2052: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 - warnings.warn( -(pid=2031595) W0825 18:00:33.227000 2031595 site-packages/torch/utils/cpp_extension.py:118] No CUDA runtime is found, using CUDA_HOME='/data/miniconda3/envs/orarl' -(Runner pid=2031595) { -(Runner pid=2031595) "data": { -(Runner pid=2031595) "train_files": "/mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_joint_100k_maskaware.jsonl", -(Runner pid=2031595) "val_files": "/mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_phase3_49152.canary.jsonl", -(Runner pid=2031595) "prompt_key": "problem", -(Runner pid=2031595) "answer_key": "answer", -(Runner pid=2031595) "image_key": "images", -(Runner pid=2031595) "video_key": "videos", -(Runner pid=2031595) "image_dir": "/", -(Runner pid=2031595) "video_fps": 2.0, -(Runner pid=2031595) "video_max_frames": 128, -(Runner pid=2031595) "max_prompt_length": 24576, -(Runner pid=2031595) "max_response_length": 4096, -(Runner pid=2031595) "rollout_batch_size": 8, -(Runner pid=2031595) "mini_rollout_batch_size": null, -(Runner pid=2031595) "val_batch_size": 16, -(Runner pid=2031595) "format_prompt": null, -(Runner pid=2031595) "override_chat_template": null, -(Runner pid=2031595) "enable_thinking": false, -(Runner pid=2031595) "response_prefix": "", -(Runner pid=2031595) "shuffle": true, -(Runner pid=2031595) "seed": 42, -(Runner pid=2031595) "min_pixels": 4096, -(Runner pid=2031595) "max_pixels": 4194304, -(Runner pid=2031595) "image_min_pixels": 4096, -(Runner pid=2031595) "image_max_pixels": 1048576, -(Runner pid=2031595) "video_min_pixels": 4096, -(Runner pid=2031595) "video_max_pixels": 786432, -(Runner pid=2031595) "video_total_pixels": 8388608, -(Runner pid=2031595) "val_video_fps": 2.0, -(Runner pid=2031595) "val_video_max_frames": 128, -(Runner pid=2031595) "val_video_min_pixels": 4096, -(Runner pid=2031595) "val_video_max_pixels": 786432, -(Runner pid=2031595) "val_video_total_pixels": 8388608, -(Runner pid=2031595) "filter_overlong_prompts": false, -(Runner pid=2031595) "filter_overlong_prompts_workers": 16, -(Runner pid=2031595) "use_preprocessed_videos": false, -(Runner pid=2031595) "video_source_mode": "realtime_only", -(Runner pid=2031595) "preprocessed_video_dir": null, -(Runner pid=2031595) "val_preprocessed_video_dir": null, -(Runner pid=2031595) "val_video_source_mode": "realtime_only", -(Runner pid=2031595) "inline_video_tensors": true, -(Runner pid=2031595) "group_by_task": true, -(Runner pid=2031595) "group_by_task_key": "problem_type", -(Runner pid=2031595) "dataloader_num_workers": 0 -(Runner pid=2031595) }, -(Runner pid=2031595) "worker": { -(Runner pid=2031595) "hybrid_engine": true, -(Runner pid=2031595) "actor": { -(Runner pid=2031595) "strategy": "fsdp", -(Runner pid=2031595) "global_batch_size": 8, -(Runner pid=2031595) "micro_batch_size_per_device_for_update": 1, -(Runner pid=2031595) "micro_batch_size_per_device_for_experience": 1, -(Runner pid=2031595) "max_grad_norm": 1.0, -(Runner pid=2031595) "clip_ratio_low": 0.2, -(Runner pid=2031595) "clip_ratio_high": 0.2, -(Runner pid=2031595) "clip_ratio_dual": 3.0, -(Runner pid=2031595) "loss_avg_mode": "token", -(Runner pid=2031595) "loss_type": "default", -(Runner pid=2031595) "ppo_epochs": 1, -(Runner pid=2031595) "padding_free": true, -(Runner pid=2031595) "dynamic_batching": true, -(Runner pid=2031595) "max_token_len_per_gpu": 32768, -(Runner pid=2031595) "ulysses_size": 1, -(Runner pid=2031595) "use_torch_compile": false, -(Runner pid=2031595) "model": { -(Runner pid=2031595) "model_path": "/apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B", -(Runner pid=2031595) "tokenizer_path": "/apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B", -(Runner pid=2031595) "override_config": {}, -(Runner pid=2031595) "enable_gradient_checkpointing": true, -(Runner pid=2031595) "trust_remote_code": false, -(Runner pid=2031595) "freeze_vision_tower": true, -(Runner pid=2031595) "train_vision_merger": false -(Runner pid=2031595) }, -(Runner pid=2031595) "optim": { -(Runner pid=2031595) "lr": 1e-06, -(Runner pid=2031595) "betas": [ -(Runner pid=2031595) 0.9, -(Runner pid=2031595) 0.999 -(Runner pid=2031595) ], -(Runner pid=2031595) "weight_decay": 0.0, -(Runner pid=2031595) "strategy": "adamw", -(Runner pid=2031595) "lr_warmup_ratio": 0.0, -(Runner pid=2031595) "lr_warmup_steps": null, -(Runner pid=2031595) "min_lr_ratio": null, -(Runner pid=2031595) "lr_scheduler_type": "constant", -(Runner pid=2031595) "training_steps": -1 -(Runner pid=2031595) }, -(Runner pid=2031595) "fsdp": { -(Runner pid=2031595) "enable_full_shard": true, -(Runner pid=2031595) "enable_cpu_offload": false, -(Runner pid=2031595) "enable_rank0_init": true, -(Runner pid=2031595) "use_orig_params": false, -(Runner pid=2031595) "torch_dtype": "bf16", -(Runner pid=2031595) "fsdp_size": -1, -(Runner pid=2031595) "mp_param_dtype": "bf16", -(Runner pid=2031595) "mp_reduce_dtype": "fp32", -(Runner pid=2031595) "mp_buffer_dtype": "fp32" -(Runner pid=2031595) }, -(Runner pid=2031595) "offload": { -(Runner pid=2031595) "offload_params": false, -(Runner pid=2031595) "offload_optimizer": false -(Runner pid=2031595) }, -(Runner pid=2031595) "global_batch_size_per_device": -1, -(Runner pid=2031595) "disable_kl": true, -(Runner pid=2031595) "use_kl_loss": false, -(Runner pid=2031595) "kl_penalty": "low_var_kl", -(Runner pid=2031595) "kl_coef": 0.0 -(Runner pid=2031595) }, -(Runner pid=2031595) "critic": { -(Runner pid=2031595) "strategy": "fsdp", -(Runner pid=2031595) "global_batch_size": 256, -(Runner pid=2031595) "micro_batch_size_per_device_for_update": 4, -(Runner pid=2031595) "micro_batch_size_per_device_for_experience": 16, -(Runner pid=2031595) "max_grad_norm": 1.0, -(Runner pid=2031595) "cliprange_value": 0.5, -(Runner pid=2031595) "loss_avg_mode": "token", -(Runner pid=2031595) "ppo_epochs": 1, -(Runner pid=2031595) "padding_free": false, -(Runner pid=2031595) "dynamic_batching": true, -(Runner pid=2031595) "max_token_len_per_gpu": null, -(Runner pid=2031595) "ulysses_size": 1, -(Runner pid=2031595) "model": { -(Runner pid=2031595) "model_path": null, -(Runner pid=2031595) "tokenizer_path": null, -(Runner pid=2031595) "override_config": {}, -(Runner pid=2031595) "enable_gradient_checkpointing": true, -(Runner pid=2031595) "trust_remote_code": true, -(Runner pid=2031595) "freeze_vision_tower": false, -(Runner pid=2031595) "train_vision_merger": false -(Runner pid=2031595) }, -(Runner pid=2031595) "optim": { -(Runner pid=2031595) "lr": 1e-06, -(Runner pid=2031595) "betas": [ -(Runner pid=2031595) 0.9,(Runner pid=2031595) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. - -(Runner pid=2031595) 0.999 -(Runner pid=2031595) ], -(Runner pid=2031595) "weight_decay": 0.01, -(Runner pid=2031595) "strategy": "adamw", -(Runner pid=2031595) "lr_warmup_ratio": 0.0, -(Runner pid=2031595) "lr_warmup_steps": null, -(Runner pid=2031595) "min_lr_ratio": null, -(Runner pid=2031595) "lr_scheduler_type": "constant", -(Runner pid=2031595) "training_steps": -1 -(Runner pid=2031595) }, -(Runner pid=2031595) "fsdp": { -(Runner pid=2031595) "enable_full_shard": true, -(Runner pid=2031595) "enable_cpu_offload": false, -(Runner pid=2031595) "enable_rank0_init": true, -(Runner pid=2031595) "use_orig_params": false, -(Runner pid=2031595) "torch_dtype": null, -(Runner pid=2031595) "fsdp_size": -1, -(Runner pid=2031595) "mp_param_dtype": "bf16", -(Runner pid=2031595) "mp_reduce_dtype": "fp32", -(Runner pid=2031595) "mp_buffer_dtype": "fp32" -(Runner pid=2031595) }, -(Runner pid=2031595) "offload": { -(Runner pid=2031595) "offload_params": false, -(Runner pid=2031595) "offload_optimizer": false -(Runner pid=2031595) }, -(Runner pid=2031595) "global_batch_size_per_device": -1 -(Runner pid=2031595) }, -(Runner pid=2031595) "ref": { -(Runner pid=2031595) "strategy": "fsdp", -(Runner pid=2031595) "fsdp": { -(Runner pid=2031595) "enable_full_shard": true, -(Runner pid=2031595) "enable_cpu_offload": false, -(Runner pid=2031595) "enable_rank0_init": true, -(Runner pid=2031595) "use_orig_params": false, -(Runner pid=2031595) "torch_dtype": "bf16", -(Runner pid=2031595) "fsdp_size": -1, -(Runner pid=2031595) "mp_param_dtype": "bf16", -(Runner pid=2031595) "mp_reduce_dtype": "fp32", -(Runner pid=2031595) "mp_buffer_dtype": "fp32" -(Runner pid=2031595) }, -(Runner pid=2031595) "offload": { -(Runner pid=2031595) "offload_params": false, -(Runner pid=2031595) "offload_optimizer": false -(Runner pid=2031595) }, -(Runner pid=2031595) "micro_batch_size_per_device_for_experience": 1, -(Runner pid=2031595) "padding_free": true, -(Runner pid=2031595) "dynamic_batching": true, -(Runner pid=2031595) "max_token_len_per_gpu": 32768, -(Runner pid=2031595) "ulysses_size": 1, -(Runner pid=2031595) "use_torch_compile": false -(Runner pid=2031595) }, -(Runner pid=2031595) "reward": { -(Runner pid=2031595) "reward_function": "orarl.rewards", -(Runner pid=2031595) "reward_function_kwargs": {}, -(Runner pid=2031595) "skip_special_tokens": true, -(Runner pid=2031595) "num_cpus": 1, -(Runner pid=2031595) "reward_function_name": "compute_score", -(Runner pid=2031595) "reward_function_is_module": true -(Runner pid=2031595) }, -(Runner pid=2031595) "rollout": { -(Runner pid=2031595) "name": "vllm", -(Runner pid=2031595) "n": 8, -(Runner pid=2031595) "temperature": 1.0, -(Runner pid=2031595) "top_p": 0.85, -(Runner pid=2031595) "top_k": -1, -(Runner pid=2031595) "seed": 42, -(Runner pid=2031595) "limit_images": 10, -(Runner pid=2031595) "dtype": "bf16", -(Runner pid=2031595) "gpu_memory_utilization": 0.45, -(Runner pid=2031595) "ignore_eos": false, -(Runner pid=2031595) "enforce_eager": false, -(Runner pid=2031595) "enable_chunked_prefill": true, -(Runner pid=2031595) "tensor_parallel_size": 1, -(Runner pid=2031595) "max_model_len": null, -(Runner pid=2031595) "max_num_batched_tokens": 32768, -(Runner pid=2031595) "disable_log_stats": true, -(Runner pid=2031595) "disable_tqdm": true, -(Runner pid=2031595) "val_override_config": { -(Runner pid=2031595) "temperature": 0.7, -(Runner pid=2031595) "top_p": 0.95, -(Runner pid=2031595) "n": 1 -(Runner pid=2031595) }, -(Runner pid=2031595) "kv_cache_dtype": "auto", -(Runner pid=2031595) "calculate_log_probs": false, -(Runner pid=2031595) "collect_seq_logprob_for_filter": false, -(Runner pid=2031595) "prompt_length": 24576, -(Runner pid=2031595) "response_length": 4096, -(Runner pid=2031595) "trust_remote_code": false -(Runner pid=2031595) } -(Runner pid=2031595) }, -(Runner pid=2031595) "algorithm": { -(Runner pid=2031595) "name": "grpo", -(Runner pid=2031595) "gamma": 1.0, -(Runner pid=2031595) "lam": 1.0, -(Runner pid=2031595) "adv_estimator": "grpo", -(Runner pid=2031595) "scale_rewards": true, -(Runner pid=2031595) "disable_kl": true, -(Runner pid=2031595) "use_kl_loss": false, -(Runner pid=2031595) "kl_penalty": "low_var_kl", -(Runner pid=2031595) "kl_coef": 0.0, -(Runner pid=2031595) "kl_type": "fixed", -(Runner pid=2031595) "kl_horizon": 10000.0, -(Runner pid=2031595) "kl_target": 0.1 -(Runner pid=2031595) }, -(Runner pid=2031595) "trainer": { -(Runner pid=2031595) "total_epochs": 1, -(Runner pid=2031595) "max_steps": 1, -(Runner pid=2031595) "project_name": "OraRL", -(Runner pid=2031595) "experiment_name": "smoke-grpo-9b", -(Runner pid=2031595) "logger": [ -(Runner pid=2031595) "console" -(Runner pid=2031595) ], -(Runner pid=2031595) "nnodes": 1, -(Runner pid=2031595) "n_gpus_per_node": 8, -(Runner pid=2031595) "critic_warmup": 0, -(Runner pid=2031595) "val_freq": -1, -(Runner pid=2031595) "val_before_train": false, -(Runner pid=2031595) "val_only": false, -(Runner pid=2031595) "val_generations_to_log": 0, -(Runner pid=2031595) "save_freq": 1, -(Runner pid=2031595) "save_limit": 3, -(Runner pid=2031595) "save_model_only": false, -(Runner pid=2031595) "keep_optim_only_latest": true, -(Runner pid=2031595) "save_checkpoint_path": "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175949/grpo", -(Runner pid=2031595) "load_checkpoint_path": null, -(Runner pid=2031595) "ray_timeline": null, -(Runner pid=2031595) "find_last_checkpoint": false, -(Runner pid=2031595) "keep_best_train_ckpt": false, -(Runner pid=2031595) "best_train_metric_key": "reward/overall", -(Runner pid=2031595) "best_train_smooth_window": 5, -(Runner pid=2031595) "best_train_min_step": 10 -(Runner pid=2031595) } -(Runner pid=2031595) } -(Runner pid=2031595) [TaskGroupedBatchSampler] 15 groups: image_sequence_mc_answer_only(10304), obj_appearance_order(64), object_abs_distance(1344), object_counting(960), object_rel_direction(448), object_rel_distance(640), object_size_estimation(1408), room_size_estimation(960), route_planning(960), segmentation(12032), spatial grounding(7040), spatial-temporal grounding(9536), temporal grounding(20096), tracking(13952), video_qa_mc(20288) | batch_size=8 | total_batches=12504 -(Runner pid=2031595) Size of train dataloader: 12504 -(Runner pid=2031595) Size of val dataloader: 32 -(Runner pid=2031595) KL is disabled, no KL metrics will be logged. Please set `kl_coef=0` to log KL metrics.(pid=2033394) /data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. -(pid=2033394) warnings.warn( -(pid=2033896) /data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. -(pid=2033896) warnings.warn( -(pid=2033897) /data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. -(pid=2033897) warnings.warn( -(WorkerDict pid=2033898) [W825 18:00:56.039186514 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator()) -(pid=2033901) /data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. [repeated 5x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) -(pid=2033901) warnings.warn( [repeated 5x across cluster] -(WorkerDict pid=2033898) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. -(WorkerDict pid=2033898) `torch_dtype` is deprecated! Use `dtype` instead! -(WorkerDict pid=2033899) [W825 18:00:58.504154548 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator()) [repeated 7x across cluster] -(WorkerDict pid=2033898) The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d -(WorkerDict pid=2033898) [rank3]:[W825 18:01:02.812976524 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group() -(WorkerDict pid=2033394) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. [repeated 7x across cluster] -(WorkerDict pid=2033394) Loading weights: 0%| | 0/760 [00:00:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. -(WorkerDict pid=2033896) :1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. -(WorkerDict pid=2033394) `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead. [repeated 7x across cluster] -(WorkerDict pid=2033897) 2026-08-25 18:02:00,379 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ... -(WorkerDict pid=2033902) :1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. [repeated 7x across cluster] -(WorkerDict pid=2033902) :1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. [repeated 7x across cluster] -(WorkerDict pid=2033897) 2026-08-25 18:02:00,394 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends -(WorkerDict pid=2033394) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/51 [00:00, transformer_layer_cls={, }). -(WorkerDict pid=2033394) After FSDP module init: 15.42 GB / 95.00 GB. -(WorkerDict pid=2033394) After optimizer init: 15.42 GB / 95.00 GB. -(WorkerDict pid=2033899) Sampling params: {'max_tokens': 4096, 'detokenize': False, 'logit_bias': {248056: -100, 248057: -100}, 'n': 8, 'temperature': 1.0, 'top_p': 0.85, 'top_k': -1, 'ignore_eos': False}. -(Runner pid=2031595) Config -(Runner pid=2031595) algorithm: -(Runner pid=2031595) adv_estimator: grpo -(Runner pid=2031595) disable_kl: true -(Runner pid=2031595) gamma: 1.0 -(Runner pid=2031595) kl_coef: 0.0 -(Runner pid=2031595) kl_horizon: 10000.0 -(Runner pid=2031595) kl_penalty: low_var_kl -(Runner pid=2031595) kl_target: 0.1 -(Runner pid=2031595) kl_type: fixed -(Runner pid=2031595) lam: 1.0 -(Runner pid=2031595) name: grpo -(Runner pid=2031595) scale_rewards: true -(Runner pid=2031595) use_kl_loss: false -(Runner pid=2031595) data: -(Runner pid=2031595) answer_key: answer -(Runner pid=2031595) dataloader_num_workers: 0 -(Runner pid=2031595) enable_thinking: false -(Runner pid=2031595) filter_overlong_prompts: false -(Runner pid=2031595) filter_overlong_prompts_workers: 16 -(Runner pid=2031595) format_prompt: null -(Runner pid=2031595) group_by_task: true -(Runner pid=2031595) group_by_task_key: problem_type -(Runner pid=2031595) image_dir: / -(Runner pid=2031595) image_key: images -(Runner pid=2031595) image_max_pixels: 1048576 -(Runner pid=2031595) image_min_pixels: 4096 -(Runner pid=2031595) inline_video_tensors: true -(Runner pid=2031595) max_pixels: 4194304 -(Runner pid=2031595) max_prompt_length: 24576 -(Runner pid=2031595) max_response_length: 4096 -(Runner pid=2031595) min_pixels: 4096 -(Runner pid=2031595) mini_rollout_batch_size: null -(Runner pid=2031595) override_chat_template: null -(Runner pid=2031595) preprocessed_video_dir: null -(Runner pid=2031595) prompt_key: problem -(Runner pid=2031595) response_prefix: '' -(Runner pid=2031595) rollout_batch_size: 8 -(Runner pid=2031595) seed: 42 -(Runner pid=2031595) shuffle: true -(Runner pid=2031595) train_files: /mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_joint_100k_maskaware.jsonl -(Runner pid=2031595) use_preprocessed_videos: false -(Runner pid=2031595) val_batch_size: 16 -(Runner pid=2031595) val_files: /mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_phase3_49152.canary.jsonl -(Runner pid=2031595) val_preprocessed_video_dir: null -(Runner pid=2031595) val_video_fps: 2.0 -(Runner pid=2031595) val_video_max_frames: 128 -(Runner pid=2031595) val_video_max_pixels: 786432 -(Runner pid=2031595) val_video_min_pixels: 4096 -(Runner pid=2031595) val_video_source_mode: realtime_only -(Runner pid=2031595) val_video_total_pixels: 8388608 -(Runner pid=2031595) video_fps: 2.0 -(Runner pid=2031595) video_key: videos -(Runner pid=2031595) video_max_frames: 128 -(Runner pid=2031595) video_max_pixels: 786432 -(Runner pid=2031595) video_min_pixels: 4096 -(Runner pid=2031595) video_source_mode: realtime_only -(Runner pid=2031595) video_total_pixels: 8388608 -(Runner pid=2031595) trainer: -(Runner pid=2031595) best_train_metric_key: reward/overall -(Runner pid=2031595) best_train_min_step: 10 -(Runner pid=2031595) best_train_smooth_window: 5 -(Runner pid=2031595) critic_warmup: 0 -(Runner pid=2031595) experiment_name: smoke-grpo-9b -(Runner pid=2031595) find_last_checkpoint: false -(Runner pid=2031595) keep_best_train_ckpt: false -(Runner pid=2031595) keep_optim_only_latest: true -(Runner pid=2031595) load_checkpoint_path: null -(Runner pid=2031595) logger: -(Runner pid=2031595) - console -(Runner pid=2031595) max_steps: 1 -(Runner pid=2031595) n_gpus_per_node: 8 -(Runner pid=2031595) nnodes: 1 -(Runner pid=2031595) project_name: OraRL -(Runner pid=2031595) ray_timeline: null -(Runner pid=2031595) save_checkpoint_path: /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175949/grpo -(Runner pid=2031595) save_freq: 1 -(Runner pid=2031595) save_limit: 3 -(Runner pid=2031595) save_model_only: false -(Runner pid=2031595) total_epochs: 1 -(Runner pid=2031595) val_before_train: false -(Runner pid=2031595) val_freq: -1 -(Runner pid=2031595) val_generations_to_log: 0 -(Runner pid=2031595) val_only: false -(Runner pid=2031595) worker: -(Runner pid=2031595) actor: -(Runner pid=2031595) clip_ratio_dual: 3.0 -(Runner pid=2031595) clip_ratio_high: 0.2 -(Runner pid=2031595) clip_ratio_low: 0.2 -(Runner pid=2031595) disable_kl: true -(Runner pid=2031595) dynamic_batching: true -(Runner pid=2031595) fsdp: -(Runner pid=2031595) enable_cpu_offload: false -(Runner pid=2031595) enable_full_shard: true -(Runner pid=2031595) enable_rank0_init: true -(Runner pid=2031595) fsdp_size: -1 -(Runner pid=2031595) mp_buffer_dtype: fp32 -(Runner pid=2031595) mp_param_dtype: bf16 -(Runner pid=2031595) mp_reduce_dtype: fp32 -(Runner pid=2031595) torch_dtype: bf16 -(Runner pid=2031595) use_orig_params: false -(Runner pid=2031595) global_batch_size: 8 -(Runner pid=2031595) global_batch_size_per_device: -1 -(Runner pid=2031595) kl_coef: 0.0 -(Runner pid=2031595) kl_penalty: low_var_kl -(Runner pid=2031595) loss_avg_mode: token -(Runner pid=2031595) loss_type: default -(Runner pid=2031595) max_grad_norm: 1.0 -(Runner pid=2031595) max_token_len_per_gpu: 32768 -(Runner pid=2031595) micro_batch_size_per_device_for_experience: 1 -(Runner pid=2031595) micro_batch_size_per_device_for_update: 1 -(Runner pid=2031595) model: -(Runner pid=2031595) enable_gradient_checkpointing: true -(Runner pid=2031595) freeze_vision_tower: true -(Runner pid=2031595) model_path: /apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B -(Runner pid=2031595) override_config: {} -(Runner pid=2031595) tokenizer_path: /apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B -(Runner pid=2031595) train_vision_merger: false -(Runner pid=2031595) trust_remote_code: false -(Runner pid=2031595) offload: -(Runner pid=2031595) offload_optimizer: false -(Runner pid=2031595) offload_params: false -(Runner pid=2031595) optim: -(Runner pid=2031595) betas: -(Runner pid=2031595) - 0.9 -(Runner pid=2031595) - 0.999 -(Runner pid=2031595) lr: 1.0e-06 -(Runner pid=2031595) lr_scheduler_type: constant -(Runner pid=2031595) lr_warmup_ratio: 0.0 -(Runner pid=2031595) lr_warmup_steps: null -(Runner pid=2031595) min_lr_ratio: null -(Runner pid=2031595) strategy: adamw -(Runner pid=2031595) training_steps: 1 -(Runner pid=2031595) weight_decay: 0.0 -(Runner pid=2031595) padding_free: true -(Runner pid=2031595) ppo_epochs: 1 -(Runner pid=2031595) strategy: fsdp -(Runner pid=2031595) ulysses_size: 1 -(Runner pid=2031595) use_kl_loss: false -(Runner pid=2031595) use_torch_compile: false -(Runner pid=2031595) critic: -(Runner pid=2031595) cliprange_value: 0.5 -(Runner pid=2031595) dynamic_batching: true -(Runner pid=2031595) fsdp: -(Runner pid=2031595) enable_cpu_offload: false -(Runner pid=2031595) enable_full_shard: true -(Runner pid=2031595) enable_rank0_init: true -(Runner pid=2031595) fsdp_size: -1 -(Runner pid=2031595) mp_buffer_dtype: fp32 -(Runner pid=2031595) mp_param_dtype: bf16 -(Runner pid=2031595) mp_reduce_dtype: fp32 -(Runner pid=2031595) torch_dtype: null -(Runner pid=2031595) use_orig_params: false -(Runner pid=2031595) global_batch_size: 256 -(Runner pid=2031595) global_batch_size_per_device: -1 -(Runner pid=2031595) loss_avg_mode: token -(Runner pid=2031595) max_grad_norm: 1.0 -(Runner pid=2031595) max_token_len_per_gpu: null -(Runner pid=2031595) micro_batch_size_per_device_for_experience: 16 -(Runner pid=2031595) micro_batch_size_per_device_for_update: 4 -(Runner pid=2031595) model: -(Runner pid=2031595) enable_gradient_checkpointing: true -(Runner pid=2031595) freeze_vision_tower: false -(Runner pid=2031595) model_path: null -(Runner pid=2031595) override_config: {} -(Runner pid=2031595) tokenizer_path: null -(Runner pid=2031595) train_vision_merger: false -(Runner pid=2031595) trust_remote_code: true -(Runner pid=2031595) offload: -(Runner pid=2031595) offload_optimizer: false -(Runner pid=2031595) offload_params: false -(Runner pid=2031595) optim: -(Runner pid=2031595) betas: -(Runner pid=2031595) - 0.9 -(Runner pid=2031595) - 0.999 -(Runner pid=2031595) lr: 1.0e-06 -(Runner pid=2031595) lr_scheduler_type: constant -(Runner pid=2031595) lr_warmup_ratio: 0.0 -(Runner pid=2031595) lr_warmup_steps: null -(Runner pid=2031595) min_lr_ratio: null -(Runner pid=2031595) strategy: adamw -(Runner pid=2031595) training_steps: 1 -(Runner pid=2031595) weight_decay: 0.01 -(Runner pid=2031595) padding_free: false -(Runner pid=2031595) ppo_epochs: 1 -(Runner pid=2031595) strategy: fsdp -(Runner pid=2031595) ulysses_size: 1 -(Runner pid=2031595) hybrid_engine: true -(Runner pid=2031595) ref: -(Runner pid=2031595) dynamic_batching: true -(Runner pid=2031595) fsdp: -(Runner pid=2031595) enable_cpu_offload: false -(Runner pid=2031595) enable_full_shard: true -(Runner pid=2031595) enable_rank0_init: true -(Runner pid=2031595) fsdp_size: -1 -(Runner pid=2031595) mp_buffer_dtype: fp32 -(Runner pid=2031595) mp_param_dtype: bf16 -(Runner pid=2031595) mp_reduce_dtype: fp32 -(Runner pid=2031595) torch_dtype: bf16 -(Runner pid=2031595) use_orig_params: false -(Runner pid=2031595) max_token_len_per_gpu: 32768 -(Runner pid=2031595) micro_batch_size_per_device_for_experience: 1 -(Runner pid=2031595) offload: -(Runner pid=2031595) offload_optimizer: false -(Runner pid=2031595) offload_params: false -(Runner pid=2031595) padding_free: true -(Runner pid=2031595) strategy: fsdp -(Runner pid=2031595) ulysses_size: 1 -(Runner pid=2031595) use_torch_compile: false -(Runner pid=2031595) reward: -(Runner pid=2031595) num_cpus: 1 -(Runner pid=2031595) reward_function: orarl.rewards -(Runner pid=2031595) reward_function_is_module: true -(Runner pid=2031595) reward_function_kwargs: {} -(Runner pid=2031595) reward_function_name: compute_score -(Runner pid=2031595) skip_special_tokens: true -(Runner pid=2031595) rollout: -(Runner pid=2031595) calculate_log_probs: false -(Runner pid=2031595) collect_seq_logprob_for_filter: false -(Runner pid=2031595) disable_log_stats: true -(Runner pid=2031595) disable_tqdm: true -(Runner pid=2031595) dtype: bf16 -(Runner pid=2031595) enable_chunked_prefill: true -(Runner pid=2031595) enforce_eager: false -(Runner pid=2031595) gpu_memory_utilization: 0.45 -(Runner pid=2031595) ignore_eos: false -(Runner pid=2031595) kv_cache_dtype: auto -(Runner pid=2031595) limit_images: 10 -(Runner pid=2031595) max_model_len: null -(Runner pid=2031595) max_num_batched_tokens: 32768 -(Runner pid=2031595) n: 8 -(Runner pid=2031595) name: vllm -(Runner pid=2031595) prompt_length: 24576 -(Runner pid=2031595) response_length: 4096 -(Runner pid=2031595) seed: 42 -(Runner pid=2031595) temperature: 1.0 -(Runner pid=2031595) tensor_parallel_size: 1 -(Runner pid=2031595) top_k: -1 -(Runner pid=2031595) top_p: 0.85 -(Runner pid=2031595) trust_remote_code: false -(Runner pid=2031595) val_override_config: -(Runner pid=2031595) n: 1 -(Runner pid=2031595) temperature: 0.7 -(Runner pid=2031595) top_p: 0.95 -(Runner pid=2031595) - (pid=2031595) Running step: 0%| | 0.00/1.00 [00:00 /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175949/grpo/global_step_1 - -=== orarl smoke (1 update(s)) === -/data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. - warnings.warn( -2026-08-25 18:07:46,220 INFO worker.py:2004 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265  -/data/miniconda3/envs/orarl/lib/python3.11/site-packages/ray/_private/worker.py:2052: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 - warnings.warn( -(pid=2088024) W0825 18:08:01.584000 2088024 site-packages/torch/utils/cpp_extension.py:118] No CUDA runtime is found, using CUDA_HOME='/data/miniconda3/envs/orarl' -(Runner pid=2088024) { -(Runner pid=2088024) "data": { -(Runner pid=2088024) "train_files": "/mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_joint_100k_maskaware.jsonl", -(Runner pid=2088024) "val_files": "/mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_phase3_49152.canary.jsonl", -(Runner pid=2088024) "prompt_key": "problem", -(Runner pid=2088024) "answer_key": "answer", -(Runner pid=2088024) "image_key": "images", -(Runner pid=2088024) "video_key": "videos", -(Runner pid=2088024) "image_dir": "/", -(Runner pid=2088024) "video_fps": 2.0, -(Runner pid=2088024) "video_max_frames": 128, -(Runner pid=2088024) "max_prompt_length": 24576, -(Runner pid=2088024) "max_response_length": 4096, -(Runner pid=2088024) "rollout_batch_size": 8, -(Runner pid=2088024) "mini_rollout_batch_size": null, -(Runner pid=2088024) "val_batch_size": 16, -(Runner pid=2088024) "format_prompt": null, -(Runner pid=2088024) "override_chat_template": null, -(Runner pid=2088024) "enable_thinking": false, -(Runner pid=2088024) "response_prefix": "", -(Runner pid=2088024) "shuffle": true, -(Runner pid=2088024) "seed": 42, -(Runner pid=2088024) "min_pixels": 4096, -(Runner pid=2088024) "max_pixels": 4194304, -(Runner pid=2088024) "image_min_pixels": 4096, -(Runner pid=2088024) "image_max_pixels": 1048576, -(Runner pid=2088024) "video_min_pixels": 4096, -(Runner pid=2088024) "video_max_pixels": 786432, -(Runner pid=2088024) "video_total_pixels": 8388608, -(Runner pid=2088024) "val_video_fps": 2.0, -(Runner pid=2088024) "val_video_max_frames": 128, -(Runner pid=2088024) "val_video_min_pixels": 4096, -(Runner pid=2088024) "val_video_max_pixels": 786432, -(Runner pid=2088024) "val_video_total_pixels": 8388608, -(Runner pid=2088024) "filter_overlong_prompts": false, -(Runner pid=2088024) "filter_overlong_prompts_workers": 16, -(Runner pid=2088024) "use_preprocessed_videos": false, -(Runner pid=2088024) "video_source_mode": "realtime_only", -(Runner pid=2088024) "preprocessed_video_dir": null, -(Runner pid=2088024) "val_preprocessed_video_dir": null, -(Runner pid=2088024) "val_video_source_mode": "realtime_only", -(Runner pid=2088024) "inline_video_tensors": true, -(Runner pid=2088024) "group_by_task": true, -(Runner pid=2088024) "group_by_task_key": "problem_type", -(Runner pid=2088024) "dataloader_num_workers": 0 -(Runner pid=2088024) }, -(Runner pid=2088024) "worker": { -(Runner pid=2088024) "hybrid_engine": true, -(Runner pid=2088024) "actor": { -(Runner pid=2088024) "strategy": "fsdp", -(Runner pid=2088024) "global_batch_size": 8, -(Runner pid=2088024) "micro_batch_size_per_device_for_update": 1, -(Runner pid=2088024) "micro_batch_size_per_device_for_experience": 1, -(Runner pid=2088024) "max_grad_norm": 1.0, -(Runner pid=2088024) "clip_ratio_low": 0.2, -(Runner pid=2088024) "clip_ratio_high": 0.2, -(Runner pid=2088024) "clip_ratio_dual": 3.0, -(Runner pid=2088024) "loss_avg_mode": "token", -(Runner pid=2088024) "loss_type": "default", -(Runner pid=2088024) "ppo_epochs": 1, -(Runner pid=2088024) "padding_free": true, -(Runner pid=2088024) "dynamic_batching": true, -(Runner pid=2088024) "max_token_len_per_gpu": 32768, -(Runner pid=2088024) "ulysses_size": 1, -(Runner pid=2088024) "use_torch_compile": false, -(Runner pid=2088024) "model": { -(Runner pid=2088024) "model_path": "/apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B", -(Runner pid=2088024) "tokenizer_path": "/apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B", -(Runner pid=2088024) "override_config": {}, -(Runner pid=2088024) "enable_gradient_checkpointing": true, -(Runner pid=2088024) "trust_remote_code": false, -(Runner pid=2088024) "freeze_vision_tower": true, -(Runner pid=2088024) "train_vision_merger": false -(Runner pid=2088024) }, -(Runner pid=2088024) "optim": { -(Runner pid=2088024) "lr": 1e-06, -(Runner pid=2088024) "betas": [ -(Runner pid=2088024) 0.9, -(Runner pid=2088024) 0.999 -(Runner pid=2088024) ], -(Runner pid=2088024) "weight_decay": 0.0, -(Runner pid=2088024) "strategy": "adamw", -(Runner pid=2088024) "lr_warmup_ratio": 0.0, -(Runner pid=2088024) "lr_warmup_steps": null, -(Runner pid=2088024) "min_lr_ratio": null, -(Runner pid=2088024) "lr_scheduler_type": "constant", -(Runner pid=2088024) "training_steps": -1 -(Runner pid=2088024) }, -(Runner pid=2088024) "fsdp": { -(Runner pid=2088024) "enable_full_shard": true, -(Runner pid=2088024) "enable_cpu_offload": false, -(Runner pid=2088024) "enable_rank0_init": true, -(Runner pid=2088024) "use_orig_params": false, -(Runner pid=2088024) "torch_dtype": "bf16", -(Runner pid=2088024) "fsdp_size": -1, -(Runner pid=2088024) "mp_param_dtype": "bf16", -(Runner pid=2088024) "mp_reduce_dtype": "fp32", -(Runner pid=2088024) "mp_buffer_dtype": "fp32" -(Runner pid=2088024) }, -(Runner pid=2088024) "offload": { -(Runner pid=2088024) "offload_params": false, -(Runner pid=2088024) "offload_optimizer": false -(Runner pid=2088024) }, -(Runner pid=2088024) "global_batch_size_per_device": -1, -(Runner pid=2088024) "disable_kl": true, -(Runner pid=2088024) "use_kl_loss": false, -(Runner pid=2088024) "kl_penalty": "low_var_kl", -(Runner pid=2088024) "kl_coef": 0.0, -(Runner pid=2088024) "selection_prune_ratio": 0.5 -(Runner pid=2088024) }, -(Runner pid=2088024) "critic": { -(Runner pid=2088024) "strategy": "fsdp", -(Runner pid=2088024) "global_batch_size": 256, -(Runner pid=2088024) "micro_batch_size_per_device_for_update": 4, -(Runner pid=2088024) "micro_batch_size_per_device_for_experience": 16, -(Runner pid=2088024) "max_grad_norm": 1.0, -(Runner pid=2088024) "cliprange_value": 0.5, -(Runner pid=2088024) "loss_avg_mode": "token", -(Runner pid=2088024) "ppo_epochs": 1, -(Runner pid=2088024) "padding_free": false, -(Runner pid=2088024) "dynamic_batching": true, -(Runner pid=2088024) "max_token_len_per_gpu": null, -(Runner pid=2088024) "ulysses_size": 1, -(Runner pid=2088024) "model": { -(Runner pid=2088024) "model_path": null, -(Runner pid=2088024) "tokenizer_path": null, -(Runner pid=2088024) "override_config": {}, -(Runner pid=2088024) "enable_gradient_checkpointing": true, -(Runner pid=2088024) "trust_remote_code": true, -(Runner pid=2088024) "freeze_vision_tower": false, -(Runner pid=2088024) "train_vision_merger": false -(Runner pid=2088024) }, -(Runner pid=2088024) "optim": { -(Runner pid=2088024) "lr": 1e-06, -(Runner pid=2088024) "betas": [ -(Runner pid=2088024) 0.9, -(Runner pid=2088024) 0.999 -(Runner pid=2088024) ], -(Runner pid=2088024) "weight_decay": 0.01, -(Runner pid=2088024) "strategy": "adamw", -(Runner pid=2088024) "lr_warmup_ratio": 0.0, -(Runner pid=2088024) "lr_warmup_steps": null, -(Runner pid=2088024) "min_lr_ratio": null, -(Runner pid=2088024) "lr_scheduler_type": "constant", -(Runner pid=2088024) "training_steps": -1 -(Runner pid=2088024) }, -(Runner pid=2088024) "fsdp": { -(Runner pid=2088024) "enable_full_shard": true, -(Runner pid=2088024) "enable_cpu_offload": false, -(Runner pid=2088024) "enable_rank0_init": true, -(Runner pid=2088024) "use_orig_params": false, -(Runner pid=2088024) "torch_dtype": null, -(Runner pid=2088024) "fsdp_size": -1, -(Runner pid=2088024) "mp_param_dtype": "bf16", -(Runner pid=2088024) "mp_reduce_dtype": "fp32", -(Runner pid=2088024) "mp_buffer_dtype": "fp32" -(Runner pid=2088024) }, -(Runner pid=2088024) "offload": { -(Runner pid=2088024) "offload_params": false, -(Runner pid=2088024) "offload_optimizer": false -(Runner pid=2088024) }, -(Runner pid=2088024) "global_batch_size_per_device": -1 -(Runner pid=2088024) }, -(Runner pid=2088024) "ref": { -(Runner pid=2088024) "strategy": "fsdp", -(Runner pid=2088024) "fsdp": { -(Runner pid=2088024) "enable_full_shard": true, -(Runner pid=2088024) "enable_cpu_offload": false, -(Runner pid=2088024) "enable_rank0_init": true, -(Runner pid=2088024) "use_orig_params": false, -(Runner pid=2088024) "torch_dtype": "bf16", -(Runner pid=2088024) "fsdp_size": -1, -(Runner pid=2088024) "mp_param_dtype": "bf16", -(Runner pid=2088024) "mp_reduce_dtype": "fp32", -(Runner pid=2088024) "mp_buffer_dtype": "fp32" -(Runner pid=2088024) }, -(Runner pid=2088024) "offload": { -(Runner pid=2088024) "offload_params": false, -(Runner pid=2088024) "offload_optimizer": false -(Runner pid=2088024) }, -(Runner pid=2088024) "micro_batch_size_per_device_for_experience": 1, -(Runner pid=2088024) "padding_free": true, -(Runner pid=2088024) "dynamic_batching": true, -(Runner pid=2088024) "max_token_len_per_gpu": 32768, -(Runner pid=2088024) "ulysses_size": 1, -(Runner pid=2088024) "use_torch_compile": false -(Runner pid=2088024) }, -(Runner pid=2088024) "reward": { -(Runner pid=2088024) "reward_function": "orarl.rewards", -(Runner pid=2088024) "reward_function_kwargs": {}, -(Runner pid=2088024) "skip_special_tokens": true, -(Runner pid=2088024) "num_cpus": 1, -(Runner pid=2088024) "reward_function_name": "compute_score", -(Runner pid=2088024) "reward_function_is_module": true -(Runner pid=2088024) }, -(Runner pid=2088024) "rollout": { -(Runner pid=2088024) "name": "vllm", -(Runner pid=2088024) "n": 8, -(Runner pid=2088024) "temperature": 1.0, -(Runner pid=2088024) "top_p": 0.85, -(Runner pid=2088024) "top_k": -1, -(Runner pid=2088024) "seed": 42, -(Runner pid=2088024) "limit_images": 10, -(Runner pid=2088024) "dtype": "bf16", -(Runner pid=2088024) "gpu_memory_utilization": 0.45, -(Runner pid=2088024) "ignore_eos": false, -(Runner pid=2088024) "enforce_eager": false, -(Runner pid=2088024) "enable_chunked_prefill": true, -(Runner pid=2088024) "tensor_parallel_size": 1, -(Runner pid=2088024) "max_model_len": null, -(Runner pid=2088024) "max_num_batched_tokens": 32768, -(Runner pid=2088024) "disable_log_stats": true, -(Runner pid=2088024) "disable_tqdm": true, -(Runner pid=2088024) "val_override_config": { -(Runner pid=2088024) "temperature": 0.7, -(Runner pid=2088024) "top_p": 0.95, -(Runner pid=2088024) "n": 1 -(Runner pid=2088024) }, -(Runner pid=2088024) "kv_cache_dtype": "auto", -(Runner pid=2088024) "calculate_log_probs": false, -(Runner pid=2088024) "collect_seq_logprob_for_filter": false, -(Runner pid=2088024) "prompt_length": 24576, -(Runner pid=2088024) "response_length": 4096, -(Runner pid=2088024) "trust_remote_code": false -(Runner pid=2088024) } -(Runner pid=2088024) }, -(Runner pid=2088024) "algorithm": { -(Runner pid=2088024) "name": "orarl", -(Runner pid=2088024) "gamma": 1.0, -(Runner pid=2088024) "lam": 1.0, -(Runner pid=2088024) "adv_estimator": "grpo", -(Runner pid=2088024) "scale_rewards": false, -(Runner pid=2088024) "disable_kl": true, -(Runner pid=2088024) "use_kl_loss": false, -(Runner pid=2088024) "kl_penalty": "low_var_kl", -(Runner pid=2088024) "kl_coef": 0.0, -(Runner pid=2088024) "kl_type": "fixed", -(Runner pid=2088024) "kl_horizon": 10000.0, -(Runner pid=2088024) "kl_target": 0.1, -(Runner pid=2088024) "oracle_injection": true, -(Runner pid=2088024) "oracle_injection_mode": "append", -(Runner pid=2088024) "oracle_builder": "orarl.rewards:build_oracle_response_from_ground_truth", -(Runner pid=2088024) "oracle_replace_index": -1, -(Runner pid=2088024) "oracle_log_exclude": true, -(Runner pid=2088024) "directional_gain": true, -(Runner pid=2088024) "directional_gain_gamma": 0.25, -(Runner pid=2088024) "directional_gain_positive_only": true, -(Runner pid=2088024) "directional_gain_recenter": true, -(Runner pid=2088024) "detached_oracle_advantage": true, -(Runner pid=2088024) "detached_oracle_advantage_scale": 2.0, -(Runner pid=2088024) "detached_oracle_use_directional_gain": false, -(Runner pid=2088024) "detached_oracle_match_best_ratio": 1.2, -(Runner pid=2088024) "detached_oracle_match_best_min": 0.05, -(Runner pid=2088024) "detached_oracle_match_best_max": 1.0, -(Runner pid=2088024) "oracle_reward_gate_beta": 2.0, -(Runner pid=2088024) "selection_prune_ratio": 0.5, -(Runner pid=2088024) "selection_keep_oracle": true, -(Runner pid=2088024) "selection_positive_quota": 1, -(Runner pid=2088024) "selection_negative_quota": 2, -(Runner pid=2088024) "selection_strict_sign_balance": true, -(Runner pid=2088024) "post_selection_recenter": true, -(Runner pid=2088024) "post_selection_rms_match": true, -(Runner pid=2088024) "post_selection_rms_min_scale": 0.25 -(Runner pid=2088024) }, -(Runner pid=2088024) "trainer": { -(Runner pid=2088024) "total_epochs": 1, -(Runner pid=2088024) "max_steps": 1, -(Runner pid=2088024) "project_name": "OraRL", -(Runner pid=2088024) "experiment_name": "smoke-orarl-9b", -(Runner pid=2088024) "logger": [ -(Runner pid=2088024) "console" -(Runner pid=2088024) ], -(Runner pid=2088024) "nnodes": 1, -(Runner pid=2088024) "n_gpus_per_node": 8, -(Runner pid=2088024) "critic_warmup": 0,(Runner pid=2088024) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. -(pid=2089852) /data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. -(pid=2089852) warnings.warn( -(pid=2090322) /data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. -(pid=2090322) warnings.warn( -(pid=2090324) /data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. -(pid=2090324) warnings.warn( -(WorkerDict pid=2090328) [W825 18:08:26.116194738 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator()) -(pid=2090327) /data/miniconda3/envs/orarl/lib/python3.11/site-packages/transformers/utils/import_utils.py:1006: FutureWarning: `is_flash_attn_greater_or_equal_2_10` is deprecated and will be removed in v5.8. Please use `is_flash_attn_greater_or_equal(library_version='2.1.0')` instead if needed. [repeated 5x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) -(pid=2090327) warnings.warn( [repeated 5x across cluster] -(WorkerDict pid=2090328) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. - -(Runner pid=2088024) "val_freq": -1, -(Runner pid=2088024) "val_before_train": false, -(Runner pid=2088024) "val_only": false, -(Runner pid=2088024) "val_generations_to_log": 0, -(Runner pid=2088024) "save_freq": 1, -(Runner pid=2088024) "save_limit": 3, -(Runner pid=2088024) "save_model_only": false, -(Runner pid=2088024) "keep_optim_only_latest": true, -(Runner pid=2088024) "save_checkpoint_path": "/mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175949/orarl", -(Runner pid=2088024) "load_checkpoint_path": null, -(Runner pid=2088024) "ray_timeline": null, -(Runner pid=2088024) "find_last_checkpoint": false, -(Runner pid=2088024) "keep_best_train_ckpt": false, -(Runner pid=2088024) "best_train_metric_key": "reward/overall", -(Runner pid=2088024) "best_train_smooth_window": 5, -(Runner pid=2088024) "best_train_min_step": 10 -(Runner pid=2088024) } -(Runner pid=2088024) } -(Runner pid=2088024) [TaskGroupedBatchSampler] 15 groups: image_sequence_mc_answer_only(10304), obj_appearance_order(64), object_abs_distance(1344), object_counting(960), object_rel_direction(448), object_rel_distance(640), object_size_estimation(1408), room_size_estimation(960), route_planning(960), segmentation(12032), spatial grounding(7040), spatial-temporal grounding(9536), temporal grounding(20096), tracking(13952), video_qa_mc(20288) | batch_size=8 | total_batches=12504 -(Runner pid=2088024) Size of train dataloader: 12504 -(Runner pid=2088024) Size of val dataloader: 32 -(Runner pid=2088024) KL is disabled, no KL metrics will be logged. Please set `kl_coef=0` to log KL metrics. -(Runner pid=2088024) [GRPO] Raw-centered advantages ON: A=r-mean (no std whitening). -(Runner pid=2088024) [OraRL] enabled | append annotation-as-rollout | directional_gain_gamma=0.25 | detached oracle | strict sign-balanced selection | post-selection moment correction -(Runner pid=2088024) Total training steps: 1 -(AutoRewardManager pid=2089612) Using reward function `compute_score` from `orarl.rewards`. -(AutoRewardManager pid=2089612) Reward name: orarl, reward type: batch. -(WorkerDict pid=2089852) actor OraRL selection active (P=0.5): scaling global_batch_size with k=4 instead of n=8. -(WorkerDict pid=2089852) actor will use global batch size 32. -(AutoRewardManager pid=2089614) Using reward function `compute_score` from `orarl.rewards`. -(AutoRewardManager pid=2089614) Reward name: orarl, reward type: batch. -(WorkerDict pid=2089852) Model config: Qwen3_5Config { -(WorkerDict pid=2089852) "architectures": [ -(WorkerDict pid=2089852) "Qwen3_5ForConditionalGeneration" -(WorkerDict pid=2089852) ], -(WorkerDict pid=2089852) "dtype": "bfloat16", -(WorkerDict pid=2089852) "image_token_id": 248056, -(WorkerDict pid=2089852) "model_type": "qwen3_5", -(WorkerDict pid=2089852) "text_config": { -(WorkerDict pid=2089852) "attention_bias": false, -(WorkerDict pid=2089852) "attention_dropout": 0.0, -(WorkerDict pid=2089852) "attn_output_gate": true, -(WorkerDict pid=2089852) "bos_token_id": null, -(WorkerDict pid=2089852) "dtype": "bfloat16", -(WorkerDict pid=2089852) "eos_token_id": 248044, -(WorkerDict pid=2089852) "full_attention_interval": 4, -(WorkerDict pid=2089852) "head_dim": 256, -(WorkerDict pid=2089852) "hidden_act": "silu", -(WorkerDict pid=2089852) "hidden_size": 4096, -(WorkerDict pid=2089852) "initializer_range": 0.02, -(WorkerDict pid=2089852) "intermediate_size": 12288, -(WorkerDict pid=2089852) "layer_types": [ -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "full_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "full_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "full_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "full_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "full_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "full_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "full_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "linear_attention", -(WorkerDict pid=2089852) "full_attention" -(WorkerDict pid=2089852) ], -(WorkerDict pid=2089852) "linear_conv_kernel_dim": 4, -(WorkerDict pid=2089852) "linear_key_head_dim": 128, -(WorkerDict pid=2089852) "linear_num_key_heads": 16, -(WorkerDict pid=2089852) "linear_num_value_heads": 32, -(WorkerDict pid=2089852) "linear_value_head_dim": 128, -(WorkerDict pid=2089852) "mamba_ssm_dtype": "float32", -(WorkerDict pid=2089852) "max_position_embeddings": 262144, -(WorkerDict pid=2089852) "mlp_only_layers": [], -(WorkerDict pid=2089852) "model_type": "qwen3_5_text", -(WorkerDict pid=2089852) "mtp_num_hidden_layers": 0, -(WorkerDict pid=2089852) "mtp_use_dedicated_embeddings": false, -(WorkerDict pid=2089852) "num_attention_heads": 16, -(WorkerDict pid=2089852) "num_hidden_layers": 32, -(WorkerDict pid=2089852) "num_key_value_heads": 4, -(WorkerDict pid=2089852) "pad_token_id": null, -(WorkerDict pid=2089852) "partial_rotary_factor": 0.25, -(WorkerDict pid=2089852) "rms_norm_eps": 1e-06, -(WorkerDict pid=2089852) "rope_parameters": { -(WorkerDict pid=2089852) "mrope_interleaved": true, -(WorkerDict pid=2089852) "mrope_section": [ -(WorkerDict pid=2089852) 11, -(WorkerDict pid=2089852) 11, -(WorkerDict pid=2089852) 10 -(WorkerDict pid=2089852) ], -(WorkerDict pid=2089852) "partial_rotary_factor": 0.25, -(WorkerDict pid=2089852) "rope_theta": 10000000, -(WorkerDict pid=2089852) "rope_type": "default" -(WorkerDict pid=2089852) }, -(WorkerDict pid=2089852) "tie_word_embeddings": false, -(WorkerDict pid=2089852) "use_cache": true, -(WorkerDict pid=2089852) "vocab_size": 248320 -(WorkerDict pid=2089852) }, -(WorkerDict pid=2089852) "tie_word_embeddings": false, -(WorkerDict pid=2089852) "transformers_version": "5.5.4", -(WorkerDict pid=2089852) "video_token_id": 248057, -(WorkerDict pid=2089852) "vision_config": { -(WorkerDict pid=2089852) "deepstack_visual_indexes": [],(WorkerDict pid=2090328) `torch_dtype` is deprecated! Use `dtype` instead! -(WorkerDict pid=2090327) [W825 18:08:27.154508901 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator()) [repeated 7x across cluster] -(WorkerDict pid=2090328) The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d -(WorkerDict pid=2090328) [rank7]:[W825 18:08:32.884323464 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group() -(WorkerDict pid=2089852) Loading weights: 0%| | 0/760 [00:00:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. -(WorkerDict pid=2090324) :1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. -(WorkerDict pid=2089852) `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead. [repeated 7x across cluster] -(WorkerDict pid=2089852) 2026-08-25 18:09:26,834 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ... -(WorkerDict pid=2089852) 2026-08-25 18:09:26,851 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends -(WorkerDict pid=2090325) :1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. [repeated 7x across cluster] -(WorkerDict pid=2090325) :1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. [repeated 7x across cluster] -(WorkerDict pid=2089852) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/51 [00:00, transformer_layer_cls={, }). -(WorkerDict pid=2089852) After FSDP module init: 15.42 GB / 95.00 GB. -(WorkerDict pid=2089852) After optimizer init: 15.42 GB / 95.00 GB. -(WorkerDict pid=2090327) Sampling params: {'max_tokens': 4096, 'detokenize': False, 'logit_bias': {248056: -100, 248057: -100}, 'n': 8, 'temperature': 1.0, 'top_p': 0.85, 'top_k': -1, 'ignore_eos': False}. -(Runner pid=2088024) Config -(Runner pid=2088024) algorithm: -(Runner pid=2088024) adv_estimator: grpo -(Runner pid=2088024) detached_oracle_advantage: true -(Runner pid=2088024) detached_oracle_advantage_scale: 2.0 -(Runner pid=2088024) detached_oracle_match_best_max: 1.0 -(Runner pid=2088024) detached_oracle_match_best_min: 0.05 -(Runner pid=2088024) detached_oracle_match_best_ratio: 1.2 -(Runner pid=2088024) detached_oracle_use_directional_gain: false -(Runner pid=2088024) directional_gain: true -(Runner pid=2088024) directional_gain_gamma: 0.25 -(Runner pid=2088024) directional_gain_positive_only: true -(Runner pid=2088024) directional_gain_recenter: true -(Runner pid=2088024) disable_kl: true -(Runner pid=2088024) gamma: 1.0 -(Runner pid=2088024) kl_coef: 0.0 -(Runner pid=2088024) kl_horizon: 10000.0 -(Runner pid=2088024) kl_penalty: low_var_kl -(Runner pid=2088024) kl_target: 0.1 -(Runner pid=2088024) kl_type: fixed -(Runner pid=2088024) lam: 1.0 -(Runner pid=2088024) name: orarl -(Runner pid=2088024) oracle_builder: orarl.rewards:build_oracle_response_from_ground_truth -(Runner pid=2088024) oracle_injection: true -(Runner pid=2088024) oracle_injection_mode: append -(Runner pid=2088024) oracle_log_exclude: true -(Runner pid=2088024) oracle_replace_index: -1 -(Runner pid=2088024) oracle_reward_gate_beta: 2.0 -(Runner pid=2088024) post_selection_recenter: true -(Runner pid=2088024) post_selection_rms_match: true -(Runner pid=2088024) post_selection_rms_min_scale: 0.25 -(Runner pid=2088024) scale_rewards: false -(Runner pid=2088024) selection_keep_oracle: true -(Runner pid=2088024) selection_negative_quota: 2 -(Runner pid=2088024) selection_positive_quota: 1 -(Runner pid=2088024) selection_prune_ratio: 0.5 -(Runner pid=2088024) selection_strict_sign_balance: true -(Runner pid=2088024) use_kl_loss: false -(Runner pid=2088024) data: -(Runner pid=2088024) answer_key: answer -(Runner pid=2088024) dataloader_num_workers: 0 -(Runner pid=2088024) enable_thinking: false -(Runner pid=2088024) filter_overlong_prompts: false -(Runner pid=2088024) filter_overlong_prompts_workers: 16 -(Runner pid=2088024) format_prompt: null -(Runner pid=2088024) group_by_task: true -(Runner pid=2088024) group_by_task_key: problem_type -(Runner pid=2088024) image_dir: / -(Runner pid=2088024) image_key: images -(Runner pid=2088024) image_max_pixels: 1048576 -(Runner pid=2088024) image_min_pixels: 4096 -(Runner pid=2088024) inline_video_tensors: true -(Runner pid=2088024) max_pixels: 4194304 -(Runner pid=2088024) max_prompt_length: 24576 -(Runner pid=2088024) max_response_length: 4096 -(Runner pid=2088024) min_pixels: 4096 -(Runner pid=2088024) mini_rollout_batch_size: null -(Runner pid=2088024) override_chat_template: null -(Runner pid=2088024) preprocessed_video_dir: null -(Runner pid=2088024) prompt_key: problem -(Runner pid=2088024) response_prefix: '' -(Runner pid=2088024) rollout_batch_size: 8 -(Runner pid=2088024) seed: 42 -(Runner pid=2088024) shuffle: true -(Runner pid=2088024) train_files: /mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_joint_100k_maskaware.jsonl -(Runner pid=2088024) use_preprocessed_videos: false -(Runner pid=2088024) val_batch_size: 16 -(Runner pid=2088024) val_files: /mnt/lyh/TempSamp/EasyVideoR1/data/joint_final/rl_phase3_49152.canary.jsonl -(Runner pid=2088024) val_preprocessed_video_dir: null -(Runner pid=2088024) val_video_fps: 2.0 -(Runner pid=2088024) val_video_max_frames: 128 -(Runner pid=2088024) val_video_max_pixels: 786432 -(Runner pid=2088024) val_video_min_pixels: 4096 -(Runner pid=2088024) val_video_source_mode: realtime_only -(Runner pid=2088024) val_video_total_pixels: 8388608 -(Runner pid=2088024) video_fps: 2.0 -(Runner pid=2088024) video_key: videos -(Runner pid=2088024) video_max_frames: 128 -(Runner pid=2088024) video_max_pixels: 786432 -(Runner pid=2088024) video_min_pixels: 4096 -(Runner pid=2088024) video_source_mode: realtime_only -(Runner pid=2088024) video_total_pixels: 8388608 -(Runner pid=2088024) trainer: -(Runner pid=2088024) best_train_metric_key: reward/overall -(Runner pid=2088024) best_train_min_step: 10 -(Runner pid=2088024) best_train_smooth_window: 5 -(Runner pid=2088024) critic_warmup: 0 -(Runner pid=2088024) experiment_name: smoke-orarl-9b -(Runner pid=2088024) find_last_checkpoint: false -(Runner pid=2088024) keep_best_train_ckpt: false -(Runner pid=2088024) keep_optim_only_latest: true -(Runner pid=2088024) load_checkpoint_path: null -(Runner pid=2088024) logger: -(Runner pid=2088024) - console -(Runner pid=2088024) max_steps: 1 -(Runner pid=2088024) n_gpus_per_node: 8 -(Runner pid=2088024) nnodes: 1 -(Runner pid=2088024) project_name: OraRL -(Runner pid=2088024) ray_timeline: null -(Runner pid=2088024) save_checkpoint_path: /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175949/orarl -(Runner pid=2088024) save_freq: 1 -(Runner pid=2088024) save_limit: 3 -(Runner pid=2088024) save_model_only: false -(Runner pid=2088024) total_epochs: 1 -(Runner pid=2088024) val_before_train: false -(Runner pid=2088024) val_freq: -1 -(Runner pid=2088024) val_generations_to_log: 0 -(Runner pid=2088024) val_only: false -(Runner pid=2088024) worker: -(Runner pid=2088024) actor: -(Runner pid=2088024) clip_ratio_dual: 3.0 -(Runner pid=2088024) clip_ratio_high: 0.2 -(Runner pid=2088024) clip_ratio_low: 0.2 -(Runner pid=2088024) disable_kl: true -(Runner pid=2088024) dynamic_batching: true -(Runner pid=2088024) fsdp: -(Runner pid=2088024) enable_cpu_offload: false -(Runner pid=2088024) enable_full_shard: true -(Runner pid=2088024) enable_rank0_init: true -(Runner pid=2088024) fsdp_size: -1 -(Runner pid=2088024) mp_buffer_dtype: fp32 -(Runner pid=2088024) mp_param_dtype: bf16 -(Runner pid=2088024) mp_reduce_dtype: fp32 -(Runner pid=2088024) torch_dtype: bf16 -(Runner pid=2088024) use_orig_params: false -(Runner pid=2088024) global_batch_size: 8 -(Runner pid=2088024) global_batch_size_per_device: -1 -(Runner pid=2088024) kl_coef: 0.0 -(Runner pid=2088024) kl_penalty: low_var_kl -(Runner pid=2088024) loss_avg_mode: token -(Runner pid=2088024) loss_type: default -(Runner pid=2088024) max_grad_norm: 1.0 -(Runner pid=2088024) max_token_len_per_gpu: 32768 -(Runner pid=2088024) micro_batch_size_per_device_for_experience: 1 -(Runner pid=2088024) micro_batch_size_per_device_for_update: 1 -(Runner pid=2088024) model: -(Runner pid=2088024) enable_gradient_checkpointing: true -(Runner pid=2088024) freeze_vision_tower: true -(Runner pid=2088024) model_path: /apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B -(Runner pid=2088024) override_config: {} -(Runner pid=2088024) tokenizer_path: /apdcephfs_gy7/share_305340263/hunyuan/yunhengli/checkpoint/Video-ORA-9B -(Runner pid=2088024) train_vision_merger: false -(Runner pid=2088024) trust_remote_code: false -(Runner pid=2088024) offload: -(Runner pid=2088024) offload_optimizer: false -(Runner pid=2088024) offload_params: false -(Runner pid=2088024) optim: -(Runner pid=2088024) betas: -(Runner pid=2088024) - 0.9 -(Runner pid=2088024) - 0.999 -(Runner pid=2088024) lr: 1.0e-06 -(Runner pid=2088024) lr_scheduler_type: constant -(Runner pid=2088024) lr_warmup_ratio: 0.0 -(Runner pid=2088024) lr_warmup_steps: null -(Runner pid=2088024) min_lr_ratio: null -(Runner pid=2088024) strategy: adamw -(Runner pid=2088024) training_steps: 1 -(Runner pid=2088024) weight_decay: 0.0 -(Runner pid=2088024) padding_free: true -(Runner pid=2088024) ppo_epochs: 1 -(Runner pid=2088024) selection_prune_ratio: 0.5 -(Runner pid=2088024) strategy: fsdp -(Runner pid=2088024) ulysses_size: 1 -(Runner pid=2088024) use_kl_loss: false -(Runner pid=2088024) use_torch_compile: false -(Runner pid=2088024) critic: -(Runner pid=2088024) cliprange_value: 0.5 -(Runner pid=2088024) dynamic_batching: true -(Runner pid=2088024) fsdp: -(Runner pid=2088024) enable_cpu_offload: false -(Runner pid=2088024) enable_full_shard: true -(Runner pid=2088024) enable_rank0_init: true -(Runner pid=2088024) fsdp_size: -1 -(Runner pid=2088024) mp_buffer_dtype: fp32 -(Runner pid=2088024) mp_param_dtype: bf16 -(Runner pid=2088024) mp_reduce_dtype: fp32 -(Runner pid=2088024) torch_dtype: null -(Runner pid=2088024) use_orig_params: false -(Runner pid=2088024) global_batch_size: 256 -(Runner pid=2088024) global_batch_size_per_device: -1 -(Runner pid=2088024) loss_avg_mode: token -(Runner pid=2088024) max_grad_norm: 1.0 -(Runner pid=2088024) max_token_len_per_gpu: null -(Runner pid=2088024) micro_batch_size_per_device_for_experience: 16 -(Runner pid=2088024) micro_batch_size_per_device_for_update: 4 -(Runner pid=2088024) model: -(Runner pid=2088024) enable_gradient_checkpointing: true -(Runner pid=2088024) freeze_vision_tower: false -(Runner pid=2088024) model_path: null -(Runner pid=2088024) override_config: {} -(Runner pid=2088024) tokenizer_path: null -(Runner pid=2088024) train_vision_merger: false -(Runner pid=2088024) trust_remote_code: true -(Runner pid=2088024) offload: -(Runner pid=2088024) offload_optimizer: false -(Runner pid=2088024) offload_params: false -(Runner pid=2088024) optim: -(Runner pid=2088024) betas: -(Runner pid=2088024) - 0.9 -(Runner pid=2088024) - 0.999 -(Runner pid=2088024) lr: 1.0e-06 -(Runner pid=2088024) lr_scheduler_type: constant -(Runner pid=2088024) lr_warmup_ratio: 0.0 -(Runner pid=2088024) lr_warmup_steps: null -(Runner pid=2088024) min_lr_ratio: null -(Runner pid=2088024) strategy: adamw -(Runner pid=2088024) training_steps: 1 -(Runner pid=2088024) weight_decay: 0.01 -(Runner pid=2088024) padding_free: false -(Runner pid=2088024) ppo_epochs: 1 -(Runner pid=2088024) strategy: fsdp -(Runner pid=2088024) ulysses_size: 1 -(Runner pid=2088024) hybrid_engine: true -(Runner pid=2088024) ref: -(Runner pid=2088024) dynamic_batching: true -(Runner pid=2088024) fsdp: -(Runner pid=2088024) enable_cpu_offload: false -(Runner pid=2088024) enable_full_shard: true -(Runner pid=2088024) enable_rank0_init: true -(Runner pid=2088024) fsdp_size: -1 -(Runner pid=2088024) mp_buffer_dtype: fp32 -(Runner pid=2088024) mp_param_dtype: bf16 -(Runner pid=2088024) mp_reduce_dtype: fp32 -(Runner pid=2088024) torch_dtype: bf16 -(Runner pid=2088024) use_orig_params: false -(Runner pid=2088024) max_token_len_per_gpu: 32768 -(Runner pid=2088024) micro_batch_size_per_device_for_experience: 1 -(Runner pid=2088024) offload: -(Runner pid=2088024) offload_optimizer: false -(Runner pid=2088024) offload_params: false -(Runner pid=2088024) padding_free: true -(Runner pid=2088024) strategy: fsdp -(Runner pid=2088024) ulysses_size: 1 -(Runner pid=2088024) use_torch_compile: false -(Runner pid=2088024) reward: -(Runner pid=2088024) num_cpus: 1 -(Runner pid=2088024) reward_function: orarl.rewards -(Runner pid=2088024) reward_function_is_module: true -(Runner pid=2088024) reward_function_kwargs: {} -(Runner pid=2088024) reward_function_name: compute_score -(Runner pid=2088024) skip_special_tokens: true -(Runner pid=2088024) rollout: -(Runner pid=2088024) calculate_log_probs: false -(Runner pid=2088024) collect_seq_logprob_for_filter: false -(Runner pid=2088024) disable_log_stats: true -(Runner pid=2088024) disable_tqdm: true -(Runner pid=2088024) dtype: bf16 -(Runner pid=2088024) enable_chunked_prefill: true -(Runner pid=2088024) enforce_eager: false -(Runner pid=2088024) gpu_memory_utilization: 0.45 -(Runner pid=2088024) ignore_eos: false -(Runner pid=2088024) kv_cache_dtype: auto -(Runner pid=2088024) limit_images: 10 -(Runner pid=2088024) max_model_len: null -(Runner pid=2088024) max_num_batched_tokens: 32768 -(Runner pid=2088024) n: 8 -(Runner pid=2088024) name: vllm -(Runner pid=2088024) prompt_length: 24576 -(Runner pid=2088024) response_length: 4096 -(Runner pid=2088024) seed: 42 -(Runner pid=2088024) temperature: 1.0 -(Runner pid=2088024) tensor_parallel_size: 1 -(Runner pid=2088024) top_k: -1 -(Runner pid=2088024) top_p: 0.85 -(Runner pid=2088024) trust_remote_code: false -(Runner pid=2088024) val_override_config: -(Runner pid=2088024) n: 1 -(Runner pid=2088024) temperature: 0.7 -(Runner pid=2088024) top_p: 0.95 -(Runner pid=2088024) - (pid=2088024) Running step: 0%| | 0.00/1.00 [00:00 /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175949/orarl/global_step_1 - -Smoke training finished. -Log: /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175949/smoke.log -GRPO output: /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175949/grpo -OraRL output: /mnt/lyh/TempSamp/EasyVideoR1/OraRL-official/runs/smoke-training-9b-20260825_175949/orarl