File size: 2,427 Bytes
86d7061 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | ---
license: other
library_name: ltx
tags:
- spatial-audio
- audio-visual
- stereo
- binaural
- video-editing
- ltx
- ltx-2
---
# SpatialAV2AV — Weights
Full fine-tuned transformer weights for **spatial (binaural / stereo) audio-video editing**
on top of **LTX-2.3 (22B)**. The model learns
`source (pre-edit sound field + video) + camera-trajectory instruction → edit (target sound field + video)`,
producing **real 2-channel stereo** audio (L≠R) whose spatial image follows the camera motion.
Trained on the [`BingoG/LTX`](https://huggingface.co/datasets/BingoG/LTX) dataset
(source→edit pairs, same clip rendered under different camera trajectories via Binamix HRTF).
## Repository layout
Each checkpoint lives in its own `step_XXXXX/` folder with an `INFO.md` recording the
training run, step, loss, format, and inference usage.
| Folder | Step | Notes |
|--------|------|-------|
| [`step_04000/`](step_04000) | 4000 / 20000 | **Latest & recommended** — best tested checkpoint (run `2026.06.19-17.33.11`) |
## ⚠️ Important: the file is torch-pickle, NOT safetensors
The weight file is named `model_weights_step_04000.safetensors` **but it is actually a
`torch.save` (zip/pickle) file**, not a real safetensors container. This is a known quirk of
the trainer (it uses `accelerator.save`, which writes torch format under a `.safetensors` name).
- ❌ `safetensors.torch.load_file(...)` → **fails with `HeaderTooLarge`**.
- ✅ Load with `torch.load(...)` and inject into the base LTX-2.3 transformer via
`load_state_dict(..., strict=False)` (verified `missing=0 unexpected=0`).
The project's `SpatialAV2AV_inference.py --full_weights <file>` path already does this correctly.
## Usage (inference)
```bash
# base model + text encoder are the stock LTX-2.3 assets (not in this repo)
python packages/ltx-trainer/scripts/SpatialAV2AV_inference.py \
--checkpoint <LTX-2.3/ltx-2.3-22b-dev.safetensors> \
--full_weights step_04000/model_weights_step_04000.safetensors \
--text_encoder_path <gemma-3-12b-it-...> \
--meta_json <...>+pan_right.json \
--output out_pan_right.mp4 \
--num_frames 113 --min_size 288 --steps 20 --guidance_scale 4.0 --seed 42
# prints "双声道 L≠R? True" on success
```
Download just one checkpoint folder:
```python
from huggingface_hub import snapshot_download
d = snapshot_download("BingoG/SpatialAV2AV", allow_patterns="step_04000/*")
```
|