Bimanual Pose VAE — joint_mix40
双手姿态 VAE 的代码与权重:学习双手局部姿态、相对腕朝向和相对腕位置, 可用于姿态先验,以及从训练 GT 的 latent 邻域生成渲染增广候选。
This is the exact joint_mix40 checkpoint used by the VAE Pose Atlas visualization. The framewise model encodes 30 local finger rotations, one relative wrist rotation, and a relative wrist position into a 40-dimensional latent. It uses a width-384 MLP and a 189-dimensional input with row-based 6D rotation representations. Shared global rotation and translation are removed by expressing the wrist relation in the left wrist frame. RGB, camera parameters, hand shape, and temporal dynamics are outside the model.
Download and run
Python 3.11 is the tested environment. Download the repository, then run from its root:
pip install huggingface_hub
python -c "from huggingface_hub import snapshot_download; snapshot_download('WhynotGPT/bimanual-pose-vae', local_dir='bimanual-pose-vae')"
cd bimanual-pose-vae
pip install -r requirements.txt
python example.py --device cpu
sha256sum -c SHA256SUMS
The example uses synthetic inputs to verify loading, reconstruction, and input gradients through the frozen VAE. It does not measure accuracy.
import torch
from load_model import load_model
from bimanual_vae import pack
vae, metadata = load_model(device="cpu") # model.safetensors + config.json
# local_rot: [B,2,15,3,3]; root_rot: [B,2,3,3]; wrist: [B,2,3]
with torch.no_grad():
rot, pos = pack(local_rot, root_rot, wrist)
mu, logvar = vae.encode(rot, pos)
reconstructed_rot, reconstructed_pos = vae.decode(mu)
Inputs use left, right ordering and each hand's native MANO convention. Local
rotations represent the full pose, including the native hand mean where required
by the source convention. Wrist positions are actual joint positions in meters,
not necessarily the MANO transl parameter. Both hands must be valid.
Decoder rotations have shape [B,31,3,3]: left fingers 0:15, right fingers
15:30, and relative right-wrist orientation 30. Position [B,3] is the right
wrist in the left wrist frame. To place a pose in a scene, choose a left-wrist
anchor (R_L,t_L), then set R_R=R_L @ rot[:,30] and t_R=t_L+R_L @ pos.
The original API is also available:
from bimanual_vae import load_pair
vae, metadata = load_pair("joint_mix40.pt", device="cpu")
# Preserve gradients through the inputs when using a frozen prior:
loss_prior = vae.energy(local_rot, root_rot, wrist).mean()
Generate augmentation poses from training GT
The portable sampler follows the train-GT neighborhood method used in the Atlas.
It encodes only split=0 rows, retrieves the nearest training pose in the same
dataset as each query, and finds neighbors from 16 distinct other training recordings.
Queries can be difficult training/validation GT poses; test queries are rejected.
The caller selects difficult queries, for example using a model's validation PA error.
python augment.py encode --data /path/to/paired_motion.npz --output train_latents.npz
python augment.py sample --pool train_latents.npz --queries /path/to/difficult_gt.npz --output render_poses.npz
For each retrieved center, the sampler draws eight neighbor pairs with weights
w in [0.2,0.8] and reuses those endpoints at tau = 0.25, 0.5, 1.0:
endpoint = w * mu_neighbor_a + (1-w) * mu_neighbor_b
z = (1-tau) * mu_train_center + tau * endpoint
Each query yields 24 synthetic bimanual poses. Output includes full MANO axis angles and rotation matrices, actual wrist positions, zero shape coefficients, latent values, and training-source indices. The left wrist is at the origin with identity orientation. See DATA_FORMAT.md for all array conventions.
These outputs are pose labels for subsequent rendering. Geometry, scene placement, hand-object contact, and collision checks belong to the rendering pipeline. The model alone does not generate mesh vertices or guarantee physical plausibility. Augmentation gains on a downstream hand estimator have not been established by this release. The portable script implements the same sampling rule; it does not recreate the Atlas's source-specific query extraction or its exact random draw order.
Training
The released checkpoint was selected at epoch 50 of a 60-epoch run using validation reconstruction metrics. Training uses 48 updates per epoch, batch 192, AdamW at 0.001, dataset/recording-balanced random sampling, rotation and skeletal FK reconstruction, relative-position reconstruction, and KL regularization.
| Source | Train paired frames | Validation paired frames | Test paired frames |
|---|---|---|---|
| ARCTIC | 32,320 | 4,960 | 4,480 |
| H2O | 12,686 | 2,560 | 1,760 |
| GRAB | 104,825 | 10,460 | 11,834 |
ARCTIC/H2O use frozen recording splits with the experiment's benchmark recordings excluded. GRAB uses subjects s1–s8 / s9 / s10. Preparation retains at most 160 paired frames per recording; GRAB is first downsampled from 120 fps to 10 fps. These are VAE training counts, separate from the main estimator's SFT clip counts.
pip install -r requirements-training.txt
python train.py --data /path/to/paired_motion.npz \
--mano-dir /path/to/mano --output runs/joint_mix40 --device cuda
train.py preserves the original training algorithm, adding configurable paths,
device, and run lengths. It needs your prepared motion cache and MANO_LEFT.pkl /
MANO_RIGHT.pkl for the native skeleton FK loss. Raw data, the motion cache, MANO
model assets, and their derived mesh buffers are not distributed here. Obtain
those dependencies under their respective terms. The original experiment scripts
are archived in training/; their historical absolute paths
and external preprocessing dependencies are documented there.
The repository includes only the joint_mix40 trained weights. Training code also exposes the historical joint_ego40 and pose_mix40 variants.
Files and verification
joint_mix40.pt: byte-identical original checkpoint, with normalization and metadata.model.safetensors,config.json: equivalent tensor-only weights and JSON metadata.bimanual_vae.py,hand_vae.py: unchanged original model definitions and rotation helpers.load_model.py,example.py: loading and differentiable-prior examples.train.py,training_config.json,training/: training entry point and provenance.augment.py,DATA_FORMAT.md: train-GT encoding and neighborhood sampling.validation.json,provenance.json,SHA256SUMS: checks and file integrity records.
Original checkpoint SHA-256:
e27bd6bab2aee8780eafb3dbdf2a368ab328af526d2835f480819052cd53d096.
The original model and training work is from the MiroHand P3R4 bimanual VAE
experiment dated 2026-09-27; this package adds portable interfaces for release.
- Downloads last month
- 21