---
license: apache-2.0
tags:
  - video
  - vae
  - autoencoder
  - tactile
  - visuo-tactile
  - world-model
library_name: pytorch
base_model: VideoVerses/VideoVAEPlus
pipeline_tag: video-to-video
---

# VideoVAE+ 16z — Tactile Finetune

A visuo-tactile finetune of [VideoVAE+](https://github.com/VideoVerses/VideoVAEPlus)
(the `sota-4-16z`, 16-latent-channel, text-free variant). Starting from the
released `sota-4-16z.ckpt`, the autoencoder was finetuned to encode/decode
tactile sensor video (left/right tactile streams) alongside RGB view frames.

## Model

- **Architecture:** `AutoencoderKL2plus1D_1dcnn` (factorized 2+1D KL autoencoder, 1D temporal CNN)
- **Latent channels (`z_channels`):** 16
- **Spatial compression:** 8× (`ch_mult=[1,2,4,4]`, 3 downsamples)
- **Temporal compression:** 4× (16 frames → 4 latent timesteps)
- **Text conditioning:** none (`caption_guide: False`)
- **Base checkpoint:** `sota-4-16z.ckpt` from VideoVerses/VideoVAEPlus
- **Finetune objective:** `LPIPSWithDiscriminator3D` (KL weight 1e-6, disc weight 0.5), base LR 5e-5

## Files

- `videovae_plus_16z_tactile.ckpt` — final finetuned weights (Lightning checkpoint, ~5 GB).
- Source code, configs, and inference scripts mirrored from the working repo.

## Usage

```bash
# reconstruct a video with the finetuned autoencoder
python inference_video.py \
    --config configs/inference/config_16z_infer_noloss.yaml \
    --ckpt   videovae_plus_16z_tactile.ckpt \
    --input  examples/videos/gt/0510_episode_000_tactile_left.mp4
```

See the included `configs/train/config_16z_tactile.yaml` for the exact finetuning
recipe. Load the checkpoint into the `AutoencoderKL2plus1D_1dcnn` model defined in
`src/models/autoencoder2plus1d_1dcnn.py`.

## License & attribution

Code and base model from [VideoVerses/VideoVAEPlus](https://github.com/VideoVerses/VideoVAEPlus)
(Apache-2.0). This repository adds tactile-finetuned weights and the corresponding
training/inference configs.