--- library_name: drq tags: - reinforcement-learning - humanoid - mujoco - humanoid-bench - locomotion - unitree-h1 - unitree-g1 datasets: - carlosferrazza/humanoid-bench base_model: dmux/DR.Q license: mit --- # HumanoidBench-DR.Q · 自训通关 checkpoints _Self-trained DR.Q checkpoints that **beat** the public dmux/DR.Q baseline on HumanoidBench locomotion tasks._ > 🛠 **训练源码 / Training source**: > 完整训练脚本、patches、eval harness、分析文档全在 GitHub 配套仓库。 > _Full training scripts, patches, eval harness, and analysis docs in the companion GitHub repo._ DR.Q 是 TD3 + model-based 表征学习的离策略 RL 算法（encoder + policy ~13 MB 推理）。 本仓库收录在 [HumanoidBench](https://github.com/carlosferrazza/humanoid-bench) 上**从零自训通关**的 checkpoints。 _DR.Q is a TD3-family off-policy RL algorithm with model-based representation learning (~13 MB encoder + policy at inference). This repo hosts checkpoints **trained from scratch** that pass the HumanoidBench locomotion bar._ --- ## 📊 性能 / Performance | Task | success_rate | mean_return | N | 对比公开 baseline | |---|---|---|---|---| | **`h1-walk-v0`** | **90%** | **801.05** | 10 ep × seed 0 | dmux/DR.Q seed 0: ~30% / mean ~530 | | **`g1-walk-v0`** | **70%** | **710.52** | 10 ep × seed 0 | torque baseline: 0% / mean ~100 (**7.1× 提升**) | `success_bar = 700` (HumanoidBench locomotion threshold). _All numbers from deterministic eval with `action_repeat=2`. Raw JSONL in `eval/`._ --- ## 🎬 演示 / Demos ### H1-walk-v0 (Unitree H1, 19 DoF) ### G1-walk-v0 (Unitree G1, 23 DoF with BlockedHands wrapper) --- ## 📦 仓库内容 / Repo layout ``` HumanoidBench-DR.Q/ ├── DRQ+HBench-h1-walk-v0+0/ # H1-walk self-trained ckpt (76 MB) │ ├── encoder.pt policy.pt agent_var.npy ← inference (~13 MB) │ ├── *_target.pt × 3 ← Q-learning targets │ ├── *_optimizer.pt × 3 ← Adam states (resume) │ ├── value.pt ← critic │ └── exp_var.npy ← exploration variance ├── DRQ+HBench-g1-walk-v0+0/ # G1-walk self-trained ckpt (76 MB) │ └── ... (same 11-file layout) ├── eval/ # Final eval JSONL (per-episode + summary row) └── assets/ # MP4 demos ``` **推理只需 3 个文件**：`encoder.pt` + `policy.pt` + `agent_var.npy`（共 ~13 MB）。 其余 8 个文件用于续训与 Q-learning target。 _Only 3 files are needed for inference; the rest are for resume-training and Q-learning targets._ --- ## 🚀 加载与推理 / Load & inference ```python # Minimal inference loader — see scripts/drq_viewer.py in companion repo import torch, numpy as np from huggingface_hub import snapshot_download ckpt_dir = snapshot_download( repo_id="wsagi/HumanoidBench-DR.Q", allow_patterns="DRQ+HBench-h1-walk-v0+0/*", ) var = np.load(f"{ckpt_dir}/DRQ+HBench-h1-walk-v0+0/agent_var.npy", allow_pickle=True).item() # encoder = Encoder(obs_dim, ...); encoder.load_state_dict(torch.load(.../encoder.pt)) # policy = Policy(...); policy.load_state_dict(torch.load(.../policy.pt)) ``` 完整加载链路参考配套仓库 [vitorcen/humanoid-training](https://github.com/vitorcen/humanoid-training) 的 `scripts/drq_viewer.py`。 ```bash git clone --recursive https://github.com/vitorcen/humanoid-training cd humanoid-training bash patches/apply.sh # apply DR.Q + HumanoidBench local patches DISPLAY=:0 python scripts/drq_viewer.py --task h1-walk-v0 --seed 0 DISPLAY=:0 python scripts/drq_viewer.py --task g1-walk-v0 --seed 0 ``` --- ## ⚠️ G1-walk 必备 patches / Required patches for G1 G1 通关**不是开箱即用**，需两层 patch（详见 [g1_training_strategies.html](https://github.com/vitorcen/humanoid-training/blob/main/docs/g1_training_strategies.html)）： | Patch | 作用 | |---|---| | `patches/g1-pos-control.patch` | G1 默认 torque control → **PD position control**（与 H1 一致），sample efficiency 4×↑ | | `patches/humanoid-bench-g1-blocked-hands.patch` | 扩展 `BlockedHandsLocoWrapper` 支持 G1，**屏蔽 14 维手指**（37D → 23D action），避免噪声污染 encoder dynamics loss | _G1 raw torque baseline trained 1M steps and stayed at 0% / mean ~100. The combined patches lift it to 70% / mean 711._ **根因（OpenCode deepseek-v4-pro 诊断）**：DR.Q 同方差 σ=0.2 exploration noise 在 37D action 上几乎每一步都扰动手指，encoder 的 dynamics loss 被迫学习无关的手指动力学 → 250k step 时 catastrophic forgetting。 _Root cause: isotropic σ=0.2 noise contaminates the encoder's dynamics loss with irrelevant 14-DoF finger motion, leading to catastrophic forgetting around 250k steps._ --- ## 🔧 训练配置 / Training config | | H1-walk | G1-walk | |---|---|---| | Algorithm | DR.Q (TD3 + zs encoder) | DR.Q + PD control + BlockedHands | | Env steps | 500,000 | 500,000 | | Wall time | 6.6 h | 3.0 h | | GPU | RTX 4090 | RTX 4090 | | `action_repeat` | 2 | 2 | | `save_freq` | 50,000 | 50,000 | | Watcher | slice-based auto-eval + early-stop (LeIsaac-inspired) | same | **训练流水线**（三个并行进程，详见配套仓库 README）： - A) DR.Q `main.py` 主训 - B) `scripts/train_watcher.py` — 分 10 slice 实时聚合 + PROGRESS/UNDERFIT/OVERFIT/DEAD 早停 - C) `scripts/ckpt_eval_loop.py` — 每出 ckpt 自动 mirror 到 HF cache + N=3 deterministic eval --- ## 📚 引用 / Citations ```bibtex @article{sferrazza2024humanoidbench, title={HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation}, author={Sferrazza, Carmelo and Huang, Dun-Ming and Lin, Xingyu and Lee, Youngwoon and Abbeel, Pieter}, journal={Robotics: Science and Systems}, year={2024} } @article{yarats2022mastering, title={Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning}, author={Yarats, Denis and Fergus, Rob and Lazaric, Alessandro and Pinto, Lerrel}, journal={ICLR}, year={2022} } ``` --- ## 📄 License MIT — same as base DR.Q and HumanoidBench. --- _Companion repository_: [github.com/vitorcen/humanoid-training](https://github.com/vitorcen/humanoid-training) — full training scripts, patches, eval harness, and analysis docs.