README.md

---
license: other
library_name: ltx
tags:
  - spatial-audio
  - audio-visual
  - stereo
  - binaural
  - video-editing
  - ltx
  - ltx-2
---

# SpatialAV2AV — Weights

Full fine-tuned transformer weights for **spatial (binaural / stereo) audio-video editing**
on top of **LTX-2.3 (22B)**. The model learns
`source (pre-edit sound field + video) + camera-trajectory instruction → edit (target sound field + video)`,
producing **real 2-channel stereo** audio (L≠R) whose spatial image follows the camera motion.

Trained on the [`BingoG/LTX`](https://huggingface.co/datasets/BingoG/LTX) dataset
(source→edit pairs, same clip rendered under different camera trajectories via Binamix HRTF).

## Repository layout

Each checkpoint lives in its own `step_XXXXX/` folder with an `INFO.md` recording the
training run, step, loss, format, and inference usage.

| Folder | Step | Notes |
|--------|------|-------|
| [`step_04000/`](step_04000) | 4000 / 20000 | **Latest & recommended** — best tested checkpoint (run `2026.06.19-17.33.11`) |

## ⚠️ Important: the file is torch-pickle, NOT safetensors

The weight file is named `model_weights_step_04000.safetensors` **but it is actually a
`torch.save` (zip/pickle) file**, not a real safetensors container. This is a known quirk of
the trainer (it uses `accelerator.save`, which writes torch format under a `.safetensors` name).

- ❌ `safetensors.torch.load_file(...)` → **fails with `HeaderTooLarge`**.
- ✅ Load with `torch.load(...)` and inject into the base LTX-2.3 transformer via
  `load_state_dict(..., strict=False)` (verified `missing=0 unexpected=0`).

The project's `SpatialAV2AV_inference.py --full_weights <file>` path already does this correctly.

## Usage (inference)

```bash
# base model + text encoder are the stock LTX-2.3 assets (not in this repo)
python packages/ltx-trainer/scripts/SpatialAV2AV_inference.py \
  --checkpoint  <LTX-2.3/ltx-2.3-22b-dev.safetensors> \
  --full_weights step_04000/model_weights_step_04000.safetensors \
  --text_encoder_path <gemma-3-12b-it-...> \
  --meta_json <...>+pan_right.json \
  --output out_pan_right.mp4 \
  --num_frames 113 --min_size 288 --steps 20 --guidance_scale 4.0 --seed 42
# prints "双声道 L≠R? True" on success
```

Download just one checkpoint folder:

```python
from huggingface_hub import snapshot_download
d = snapshot_download("BingoG/SpatialAV2AV", allow_patterns="step_04000/*")
```