redredfirefire/piper_cocktail_expert

Piper Cocktail Expert

LingBot-VLA 2.0 post-trained for long-horizon cocktail pouring

Overview

Piper Cocktail Expert is an application-specific Vision-Language-Action policy for real-world, language-conditioned cocktail pouring with a PiPER six-axis single-arm robot. It retains the LingBot-VLA 2.0 architecture and adapts it with a dedicated PiPER embodiment mapping, a curated cocktail-manipulation dataset, and a task-specific post-training recipe.

The policy is designed for a progressively longer task curriculum: pouring one cup, two cups in sequence, or three cups in sequence. It jointly consumes two RGB views, robot proprioception, and a natural-language instruction, then predicts an action chunk for closed-loop execution.

This repository contains exported inference artifacts only: sharded model weights, configuration, tokenizer files, and image/video processor files. Optimizer, scheduler, and trainer state are not included.

Model at a glance

Item	Description
Model family	LingBot-VLA 2.0
Vision-language backbone	Qwen3-VL-4B-Instruct family
Robot embodiment	PiPER six-axis single arm with two-finger gripper
Primary task	Multi-stage cocktail pouring and cup placement
Visual observations	Main-view RGB + wrist-view RGB
Robot interface	6 arm joints + 1 gripper channel
Internal representation	55-dimensional canonical state/action space
Action generation	Flow matching with a sparse MoE action expert
Action horizon	50
Weight format	6 sharded safetensors files

Model design

The checkpoint follows the core design of LingBot-VLA 2.0:

• Unified action representation. The model operates in LingBot-VLA 2.0's 55-dimensional canonical space. For PiPER, the active source interface is a 7-dimensional vector containing six joint positions and one gripper position.
• Sparse MoE action expert. The action pathway uses 32 experts with top-4 routing, allowing task-specialized computation within the shared policy.
• Flow-matching policy. The model predicts an action chunk of length 50 and uses 10 denoising iterations during generation.
• Dual-query visual supervision. Current- and future-oriented visual queries are aligned with geometric and temporal teacher representations during training.
• Language conditioning. Natural-language task prompts specify the requested cup sequence and stopping condition.

For the PiPER mapping, arm targets are learned relative to the current joint state, while the gripper target remains absolute. Predictions are converted back to executable absolute PiPER commands by the robot feature transform.

Task-specific policy flow: PiPER 7-D control mapped into the LingBot-VLA 2.0 canonical space, with an MoE action expert and dual-query distillation.

Training dataset

Post-training used the public Cocktail Pouring — PiPER Multi-Stage Demonstrations release. It contains synchronized LeRobotDataset v3 data recorded at 15 FPS with main-view and wrist-view RGB cameras.

Curriculum stage	Released subset	Episodes	Frames
Red	cocktail_red_clean_trimmed_purged	32	31,322
Blue → Green	cocktail_blue_green_red_1	40	69,696
Blue → Green → Red	cocktail_blue_green_red_2_trimmed_purged	61	118,593
Total		133	219,611

Each raw observation/action vector has seven channels in the following order:

joint_1.pos, joint_2.pos, joint_3.pos, joint_4.pos,
joint_5.pos, joint_6.pos, gripper.pos

The three curriculum stages were mixed with a 4:2:1 task-sampling ratio for red, blue-to-green, and blue-to-green-to-red respectively. This ratio compensates for the different trajectory lengths: after weighting, the three stages contribute 125,288, 139,392, and 118,593 frame exposures, or approximately 32.7%, 36.4%, and 30.9% of a weighted epoch. Dataset-level mean and standard deviation statistics were used to normalize robot state and action features.

Dataset visuals

Wrist-view episode overview

The montage summarizes wrist-camera observations across the three curriculum stages. Released episode and frame counts in the table above are authoritative.

Main-camera demonstrations

Wrist-camera demonstrations

Training strategy

The model was initialized from PiPER-adapted LingBot-VLA 2.0 weights and post-trained for 9 epochs. Training focused on the action expert and state-projection pathway while retaining the pretrained vision-language capabilities.

The main action objective used flow matching. Future-depth and future-video representation objectives supplied additional geometric and temporal supervision. Patch-level matching and cosine feature alignment were enabled for visual consistency. Sparse routing used per-sequence auxiliary balancing and router z-loss regularization.

Training configuration

Configuration	Value
Released training data	133 episodes / 219,611 frames
Weighted frame exposures per epoch	383,273
Camera streams	2
Compute	4 × NVIDIA A100 80GB GPUs
Training duration	9 epochs
Distributed training	FSDP2
Optimizer	Muon
Learning rate	5e-5
Learning-rate schedule	Constant
Micro batch size	64 per GPU
Global batch size	256
Gradient accumulation	1
Batch-size relation	64 × 4 × 1 = 256
Action-expert precision	FP32
Gradient checkpointing	Enabled
Expert-only training	Enabled
State-projection training	Enabled
Image augmentation	Disabled
Depth-teacher input size	224 × 224
Video-teacher input size	256 × 256
Language token budget	72
Observation horizon	1
Action horizon	50
Maximum action/state dimension	55

The 5e-5 learning rate is intentional rather than unusually small. It follows the upstream real-robot post-training recipe, while the upstream RoboTwin recipe uses 1e-4 for a larger and more diverse simulation mixture. This checkpoint continues from PiPER-adapted weights and updates the expert/state pathway on a smaller real-world dataset, so the more conservative rate reduces the risk of unstable adaptation and catastrophic forgetting.

The batch-size arithmetic is explicit: 64 samples per GPU × 4 GPUs × 1 accumulation = 256. Two camera streams, expert-focused updates, FSDP2, and gradient checkpointing were used to support this memory profile on 80GB GPUs. Because adjacent video frames are correlated and the public dataset has no held-out split, the 9-epoch schedule should be interpreted together with real-robot evaluation rather than frame-level training loss alone.

MoE and auxiliary objectives

Configuration	Value
Routed experts	32
Experts selected per token	4
Router activation	Sigmoid
Routed scaling factor	4.0
Per-sequence auxiliary coefficient	1e-3
Router z-loss coefficient	1e-4
Current-depth loss weight	0.004
Future-depth loss weight	0.004
Future-video loss weight	0.004
Video feature MSE weight	1.0
Video feature cosine weight	0.2
Current patch alignment	Enabled
Warm-up frame conditioning	Enabled

Input and output contract

Modality	Dataset field	Model-side role
Main RGB	observation.images.image	camera_top
Wrist RGB	observation.images.wrist_image	camera_wrist_left
Robot state	observation.state	Six arm joints + gripper
Instruction	English task prompt	Language conditioning
Policy output	action	50 future 7-D robot commands after inverse mapping

The robot configuration and the normalization statistics used at inference must match the training data. A mismatched channel order, camera mapping, or normalization file can make otherwise valid weights unusable.

Download

With the Hugging Face CLI:

hf download redredfirefire/piper_cocktail_expert \
  --local-dir ./piper_cocktail_expert

Or with Python:

from huggingface_hub import snapshot_download

model_dir = snapshot_download(
    repo_id="redredfirefire/piper_cocktail_expert",
    local_dir="./piper_cocktail_expert",
)
print(model_dir)

Environment and deployment

The weights require the custom LingBot-VLA implementation; they cannot be loaded as a standalone stock transformers model.

Install the official codebase and environment:

git clone https://github.com/robbyant/lingbot-vla-v2.git
cd lingbot-vla-v2
bash tools/create_train_env.sh

Download the checkpoint and make the Qwen3-VL backbone available:

hf download redredfirefire/piper_cocktail_expert \
  --local-dir ./piper_cocktail_expert

export QWEN3VL_PATH=/path/to/Qwen3-VL-4B-Instruct

The upstream model-side server can be launched in the following form:

python -m deploy.lingbot_vla_v2_policy \
  --model_path ./piper_cocktail_expert \
  --use_compile \
  --use_length 25 \
  --port 8000

For physical PiPER deployment, add a compatible PiPER robot configuration, the matching normalization statistics, and an observation/action adapter. The adapter must preserve the camera names and 7-D joint/gripper ordering documented above. Hardware limits, collision checks, watchdog behavior, emergency stops, and action smoothing belong in the robot-side control stack.

The LingBot-Depth and DINO-Video teacher models are training dependencies; they are not bundled with this inference checkpoint.

Evaluation status

Preliminary real-robot result

Metric	Result
Evaluation trials	10
Successful trials	2
Overall success rate	20%

The reported result measures complete end-to-end task success in a small real-robot evaluation. Because it contains only 10 trials, it should be treated as an early diagnostic result rather than a statistically stable benchmark. Results published for upstream LingBot-VLA 2.0 models should not be interpreted as results for this PiPER-specialized checkpoint.

Possible causes of failure

The current result likely reflects a combination of data, optimization, and closed-loop control limitations. These are working hypotheses; no controlled ablation has yet isolated their individual contributions.

• Dataset coverage. The training set contains 133 demonstration episodes from one robot, one tabletop environment, two fixed camera viewpoints, and a limited set of cups and layouts. It provides relatively little variation in lighting, initial pose, object placement, pouring dynamics, and recovery behavior. Errors are especially likely to compound during the longer multi-cup sequences.
• Training configuration. Nine epochs on a small and visually consistent dataset, with image augmentation disabled, may encourage overfitting to the collection setup. The large global batch can also smooth rare corrective signals from difficult trajectory segments. In addition, expert-focused adaptation, a constant learning rate, and the selected action-versus-auxiliary loss weights may not provide the best balance for precise contact-rich control.
• Inference and robot control. Long action chunks reduce the frequency of visual feedback and replanning, so small joint, gripper, or cup-pose errors can accumulate. Performance is also sensitive to state normalization, robot calibration, camera timing, inference latency, action-queue behavior, command smoothing, and the conversion from relative arm predictions to absolute PiPER targets.

Useful next experiments include episode-level validation, broader camera and object randomization, recovery demonstrations, shorter execution chunks, more frequent replanning, learning-rate and auxiliary-loss sweeps, and per-failure logging for perception, grasping, pouring, placement, and stopping errors.

Limitations and safety

• The model is specialized to one PiPER embodiment, one tabletop setup, two fixed camera viewpoints, and a limited set of colored cups and pouring sequences.
• The dataset provides a training split only. Any evaluation split should be created at the episode level to avoid leakage across neighboring frames.
• The policy does not estimate force, liquid volume, calibrated uncertainty, or collision risk.
• Performance may degrade under camera movement, lighting changes, altered object geometry, unseen containers, or different robot calibration.
• Do not deploy around people, fragile objects, hot liquids, or hazardous materials without independent validation, workspace limits, emergency-stop procedures, and direct supervision.

Citation

If you use the architecture or upstream implementation, cite LingBot-VLA 2.0:

@article{lingbotvla2,
  title   = {From Foundation to Application: Improving VLA Models in Practice},
  author  = {Wei Wu and Fangjing Wang and Fan Lu and He Sun and Shi Liu and Yunnan Wang and Yibin Yan and Yong Wang and Shuailei Ma and Xinyang Wang and Yibin Liu and Shuai Yang and Tianxiang Zhou and Kejia Zhang and Lei Zhou and Cheng Su and Nan Xue and Bin Tan and Han Zhang and Youchao Zhang and Fei Liao and Xing Zhu and Yujun Shen and Kecheng Zheng},
  journal = {arXiv preprint arXiv:2607.06403},
  year    = {2026}
}

For the post-training data, cite the dataset repository together with the exact Hub revision used in your experiment.

License and acknowledgements

The upstream LingBot-VLA 2.0 codebase is released under Apache-2.0. The linked cocktail dataset currently does not declare a reuse license; public availability does not by itself grant commercial-use, redistribution, or derivative-work rights. Users are responsible for checking the terms that apply to the upstream weights, code, and dataset.

This checkpoint builds on LingBot-VLA 2.0, Qwen3-VL, LeRobot, LingBot-Depth, and DINO-Video.