README.md

---
license: apache-2.0
language:
- en
base_model:
- Qwen/Qwen3-VL-4B-Instruct
pipeline_tag: video-text-to-text
tags:
- Transition Detection
- Video Transition
- Video Processing
---

<div style="display:flex;align-items:center;justify-content:space-between;gap:16px"><span style="font-size:2em;font-weight:600;line-height:1.25">TransVLM</span><picture style="flex:none"><source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/HeyGenAI/TransVLM-Qwen3-VL-4B-Instruct/resolve/main/assets/heygen-logo-white.png"><img src="https://huggingface.co/HeyGenAI/TransVLM-Qwen3-VL-4B-Instruct/resolve/main/assets/heygen-logo-black.png" alt="HeyGen Research" style="height:44px;display:block"></picture></div>

<p align="center">
  <a href="https://arxiv.org/abs/2604.27975"><img src="https://img.shields.io/badge/arXiv-2604.27975-b31b1b.svg" alt="arXiv"></a>
  <a href="https://heygen-com.github.io/TransVLM/"><img src="https://img.shields.io/badge/Project-Page-1f6feb.svg" alt="Project Page"></a>
  <a href="https://github.com/heygen-com/TransVLM"><img src="https://img.shields.io/badge/GitHub-heygen--com%2FTransVLM-181717.svg" alt="GitHub"></a>
  <a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/License-Apache%202.0-3da639.svg" alt="License"></a>
</p>

<p align="center">
  <b>A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions</b>
</p>

<p align="center">
  <b>Accepted to ECCV 2026 (Poster)</b>
</p>

> [!IMPORTANT]
> **This is not a drop-in Qwen3-VL checkpoint.** Its vision tower takes **6 channels**: RGB
> concatenated with an optical-flow visualisation (`config.json` → `vision_config.in_channels: 6`).
> A standard `AutoProcessor` / `Qwen3VLForConditionalGeneration` pipeline supplies 3 channels and
> will not produce correct results. The weights load, but the input is wrong.
>
> Run it through the released inference code: **https://github.com/heygen-com/TransVLM**

## Model Description

Traditional Shot Boundary Detection (SBD) looks for isolated cut *points*, which breaks down on
gradual transitions and frequently yields corrupted shots. **TransVLM** targets Shot Transition
Detection (STD) instead: it predicts the continuous temporal *segments* over which a transition
happens.

The model is Qwen3-VL-4B-Instruct with one architectural change. Its vision patch embedding is
widened from 3 to 6 input channels (zero-padded at initialisation), so an optical-flow
visualisation enters the model alongside RGB at the input stage. Motion is what separates a
dissolve from a camera pan, and a model that only sees appearance has to infer it. Because the
flow is fused *before* patchification, the language backbone carries no additional visual tokens,
so temporal awareness comes for free at the token budget.

Training used a scalable FFmpeg-based data engine covering 59 transition effects, which addresses
the severe class imbalance in public shot-boundary data.

## Model Details

| | |
|---|---|
| Base model | [Qwen/Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) |
| Architecture | `Qwen3VLForConditionalGeneration` |
| Precision | `bfloat16` |
| Vision input channels | **6** (RGB + optical-flow visualisation, concatenated on the channel axis) |
| Sampling frame rate | 25 fps |
| Optical flow | [NeuFlow v2](https://huggingface.co/Study-is-happy/neuflow-v2), computed at inference time. Its weights (~30 MB) come from the Hub on first use, so expect a second download |
| Input | One video; flow is computed for you |
| Output | Transition segments as start/end times in seconds |
| Saved with | `transformers` 4.57.3 |

## Intended Use and Limitations

**Intended for** detecting shot transitions in videos, both hard cuts and gradual effects, and
as a baseline on the STD benchmark.

**Limitations**

- Not a frame-level cut-point classifier. The output is a time span per transition, not a single
  boundary frame.
- Not a general video-chat model. It was fine-tuned on one task with one prompt; the prompt ships
  with the inference code and changing it changes the task.
- Long videos should be split before inference. Cost is linear in duration, and the flow
  visualisation is normalised over its whole input, so a long video and its parts are not the same
  signal.

## Usage

No standalone `transformers` snippet is given here on purpose: a 3-channel one would run and
return nonsense. Use the inference package, which needs Python 3.12, a CUDA GPU, and `ffmpeg`
on `PATH`:

```bash
git clone https://github.com/heygen-com/TransVLM
cd TransVLM/inference

uv venv && source .venv/bin/activate
uv sync --group cu130 --group dev        # cu128 if your driver is older than 570

# Do not skip this. PyTorch 2.9.1 has a Conv3d bug below cuDNN 9.15, and the 6-channel
# patch embed IS a Conv3d, so it is on the hot path of every forward pass. `uv sync`
# resolves cuDNN back down, so re-run this after every sync. Activate the venv first:
# unlike `uv sync`, `uv pip` installs into whatever environment is active.
uv pip install nvidia-cudnn-cu13==9.16.0.29        # nvidia-cudnn-cu12 for cu128 / cu126
python -c "import torch; print(torch.backends.cudnn.version())"   # must print 91600

hf download HeyGenAI/TransVLM-Qwen3-VL-4B-Instruct --local-dir ./pretrained/TransVLM-v1

python infer_video.py \
  --video /path/to/video.mp4 \
  --ckpt-dir ./pretrained/TransVLM-v1 \
  --output-jsonl out.jsonl
```

The cuDNN package name tracks the CUDA flavour, and installing the wrong one is a silent
no-op: it lands an unused package while torch keeps loading the other family. That is why
the check prints a version number instead of trusting the install.

Each output line carries `segments` as `{start_time, end_time}` pairs in seconds on the original
video's timeline, plus timings and the full configuration that produced them.

Three backends are supported: HuggingFace (default), vLLM and SGLang. Environment setup, every
option and the output schema are documented in
[`inference/README.md`](https://github.com/heygen-com/TransVLM/blob/main/inference/README.md).

## Evaluation

Qualitative results and the comparison against baselines are on the
[project page](https://heygen-com.github.io/TransVLM/).

The scorer that produced the paper's metric is released at
[`evaluation/`](https://github.com/heygen-com/TransVLM/tree/main/evaluation). Note that the
published figures cannot be reproduced bit-for-bit from the released artifacts alone. The
benchmark ground truth and the pre-computed optical flow used for that run are not part of this
release. The reasons are spelled out in
[`evaluation/README.md`](https://github.com/heygen-com/TransVLM/blob/main/evaluation/README.md).

## Release Progress

- [x] Model weights
- [x] Inference code
- [x] Evaluation code
- [ ] Data engine code
- [ ] STD benchmark data
- [ ] Re-annotated dataset labels
- [ ] Leaderboard

🚧 The remaining items are being prepared for release.

## Citation

```bibtex
@inproceedings{chen2026transvlm,
  title={TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions},
  author={Chen, Ce and Ren, Yi and Li, Yuanming and Goriachko, Viktor and
          Ye, Zhenhui and Guo, Zujin and Hong, Zhibin and Gong, Mingming},
  booktitle={European Conference on Computer Vision},
  year={2026},
  organization={Springer}
}
```

## License

Apache License 2.0, inherited from [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct).

### Scope of this release

<sub>TransVLM was developed as academic research and is published at ECCV 2026. It is not used in
any HeyGen product; HeyGen's production system for this task is a different model, trained
separately from this one.</sub>

### Training data

<sub>Training combined HeyGen's own generated and annotated footage, which accounts for the
majority of the sampling distribution, with four public shot-boundary datasets:
<a href="https://github.com/wentaozhu/AutoShot">AutoShot</a>,
<a href="https://github.com/Tangshitao/ClipShots">ClipShots</a>,
MovieShots2 (<a href="https://zweipa.github.io/TMM_SCTSNet/">SCTSNet</a>, whose cut annotations
build on the <a href="https://www.robots.ox.ac.uk/~vgg/data/condensed-movies/">Condensed Movies
Dataset</a>), and <a href="https://huggingface.co/datasets/MCG-NJU/SportsShot">SportsShot</a>.
Together these four are roughly 42% of the training sampling distribution, of which SportsShot is
about 7.5%.</sub>

<sub><b>SportsShot is licensed CC BY-NC 4.0</b> and is credited here accordingly. The other three
distribute annotations rather than footage, so consult each dataset's own terms before relying on
them. No dataset is redistributed by this repository, and no dataset footage is contained in these
weights.</sub>