| --- |
| license: apache-2.0 |
| tags: |
| - video |
| - vae |
| - autoencoder |
| - tactile |
| - visuo-tactile |
| - world-model |
| library_name: pytorch |
| base_model: VideoVerses/VideoVAEPlus |
| pipeline_tag: video-to-video |
| --- |
| |
| # VideoVAE+ 16z — Tactile Finetune |
|
|
| A visuo-tactile finetune of [VideoVAE+](https://github.com/VideoVerses/VideoVAEPlus) |
| (the `sota-4-16z`, 16-latent-channel, text-free variant). Starting from the |
| released `sota-4-16z.ckpt`, the autoencoder was finetuned to encode/decode |
| tactile sensor video (left/right tactile streams) alongside RGB view frames. |
|
|
| ## Model |
|
|
| - **Architecture:** `AutoencoderKL2plus1D_1dcnn` (factorized 2+1D KL autoencoder, 1D temporal CNN) |
| - **Latent channels (`z_channels`):** 16 |
| - **Spatial compression:** 8× (`ch_mult=[1,2,4,4]`, 3 downsamples) |
| - **Temporal compression:** 4× (16 frames → 4 latent timesteps) |
| - **Text conditioning:** none (`caption_guide: False`) |
| - **Base checkpoint:** `sota-4-16z.ckpt` from VideoVerses/VideoVAEPlus |
| - **Finetune objective:** `LPIPSWithDiscriminator3D` (KL weight 1e-6, disc weight 0.5), base LR 5e-5 |
|
|
| ## Files |
|
|
| - `videovae_plus_16z_tactile.ckpt` — final finetuned weights (Lightning checkpoint, ~5 GB). |
| - Source code, configs, and inference scripts mirrored from the working repo. |
|
|
| ## Usage |
|
|
| ```bash |
| # reconstruct a video with the finetuned autoencoder |
| python inference_video.py \ |
| --config configs/inference/config_16z_infer_noloss.yaml \ |
| --ckpt videovae_plus_16z_tactile.ckpt \ |
| --input examples/videos/gt/0510_episode_000_tactile_left.mp4 |
| ``` |
|
|
| See the included `configs/train/config_16z_tactile.yaml` for the exact finetuning |
| recipe. Load the checkpoint into the `AutoencoderKL2plus1D_1dcnn` model defined in |
| `src/models/autoencoder2plus1d_1dcnn.py`. |
|
|
| ## License & attribution |
|
|
| Code and base model from [VideoVerses/VideoVAEPlus](https://github.com/VideoVerses/VideoVAEPlus) |
| (Apache-2.0). This repository adds tactile-finetuned weights and the corresponding |
| training/inference configs. |
|
|
