SleepMastger/fruit-picking-lingbot

Fruit-picking LingBot-VA

Public archival bundle for the fruit-picking LingBot-VA model trained by Amin. This bundle contains the completed step 3,000 checkpoint. It has not yet been evaluated on a robot, and the physical meaning of the -1/+1 gripper polarity still needs confirmation.

Checkpoints mirror the existing WAM repository layout:

• checkpoints/weights/step_001000.safetensors — SHA-256 a491d011ab44b402aa6dcc7d059e4f8c39fa6423f2531b57e1648955aedca2b9
• checkpoints/weights/step_001600.safetensors — SHA-256 66815ef540cea9e57cfe0d94e0a606f2d933de371d4e176f6fd3a40ac8fea459
• checkpoints/weights/step_002400.safetensors — SHA-256 321341c6a195b8f5ab0b63d78aaaf63dffbfa4023ee2de7ab1d4bee6b17d573f
• checkpoints/weights/step_003000.safetensors — SHA-256 0c12052e8e95f7480d9dbe110b5b4d028cbf359387e90ea7c6b083ade9018797

Conditioning

Exact task text:

Lift the lid, put it aside, and pick the black plum.

• conditioning/text_embedding.pt: the exact UMT5 embedding used by every fruit-picking segment, shape [512, 4096], bfloat16.
• conditioning/empty_text_embedding.pt: the empty-string UMT5 embedding used for classifier-free guidance (cfg_prob: 0.1 during training).

Both were produced from the Wan2.2 Diffusers tokenizer/text encoder. The generation code is included in training_code/extract_latents.py.

Processing and masks

• Two 256x256 cameras are VAE-encoded independently and concatenated along latent width.
• Frames are converted from uint8 to [-1, 1].
• Video length is truncated to the largest 4k+1 frame count before temporal VAE compression.
• VAE latents are normalized with the Wan config's latents_mean/std.
• Actions use channels 0-6 in a 30-channel tensor. They are padded/aligned, normalized from fruit-picking q01/q99 to [-1, 1], clipped to [-1.5, 1.5], and accompanied by an exact boolean action mask.
• The FlexAttention mask separates clean/noisy latent and action streams, applies block-causal ordering, excludes same-block noisy-to-clean attention, limits attention to window 30, and uses frame chunk size 4.
• Cross-attention is sequence-isolated; each sample attends only to its own 512-token text embedding.

The exact config, dataset processor, latent extractor, model mask implementation, and state conversion are included under training_code/. No license is asserted here for the bundled upstream code; its original terms continue to apply.

Base components

This is a fine-tuned transformer checkpoint, not a standalone pipeline. It requires the matching LingBot-VA/Wan2.2 VAE and scheduler code. The transformer configuration is included as config.json.