Add model card

The card was three lines of front matter. It now describes what the checkpoint is, and leads with
the thing a reader has to know before anything else: the vision tower takes six channels, RGB
concatenated with an optical-flow visualisation, so a standard AutoProcessor pipeline feeds three
and returns nonsense. The weights load either way, which is what makes that failure worth a
warning rather than a footnote.

No standalone transformers snippet is given, on purpose. A three-channel one would run. Instead the
usage section is the real path end to end, including the cuDNN pin that a uv sync silently undoes
and that a Conv3d patch embed cannot do without.

Metrics are not quoted here. Comparisons live on the project page with the protocol attached, and
the evaluation README states plainly which released artifacts are missing for a bit-exact
reproduction of the published figures.

README.md

@@ -9,4 +9,168 @@ tags:
 - Transition Detection
 - Video Transition
 - Video Processing
----
+---
+
+<p align="center">
+  <a href="https://www.heygen.com/research">
+    <picture>
+      <source media="(prefers-color-scheme: dark)" srcset="assets/heygen-full-white-color-logo.svg">
+      <img src="assets/heygen-full-black-color-logo.svg" width="240" alt="HeyGen Research">
+    </picture>
+  </a>
+</p>
+
+<h1 align="center">TransVLM</h1>
+
+<p align="center">
+  <b>A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions</b>
+</p>
+
+<p align="center">
+  University of Melbourne &nbsp;·&nbsp; HeyGen Research &nbsp;·&nbsp; Nanyang Technological University
+</p>
+
+<p align="center">
+  <a href="https://arxiv.org/abs/2604.27975"><img src="https://img.shields.io/badge/arXiv-2604.27975-b31b1b.svg" alt="arXiv"></a>
+  <a href="https://heygen-com.github.io/TransVLM/"><img src="https://img.shields.io/badge/Project-Page-1f6feb.svg" alt="Project Page"></a>
+  <a href="https://github.com/heygen-com/TransVLM"><img src="https://img.shields.io/badge/GitHub-heygen--com%2FTransVLM-181717.svg" alt="GitHub"></a>
+  <a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/License-Apache%202.0-3da639.svg" alt="License"></a>
+</p>
+
+<p align="center">
+  <b>Accepted to ECCV 2026 (Poster)</b>
+</p>
+
+> [!IMPORTANT]
+> **This is not a drop-in Qwen3-VL checkpoint.** Its vision tower takes **6 channels**: RGB
+> concatenated with an optical-flow visualisation (`config.json` → `vision_config.in_channels: 6`).
+> A standard `AutoProcessor` / `Qwen3VLForConditionalGeneration` pipeline supplies 3 channels and
+> will not produce correct results. The weights load, but the input is wrong.
+>
+> Run it through the released inference code: **https://github.com/heygen-com/TransVLM**
+
+## Model Description
+
+Traditional Shot Boundary Detection (SBD) looks for isolated cut *points*, which breaks down on
+gradual transitions and frequently yields corrupted shots. **TransVLM** targets Shot Transition
+Detection (STD) instead: it predicts the continuous temporal *segments* over which a transition
+happens.
+
+The model is Qwen3-VL-4B-Instruct with one architectural change. Its vision patch embedding is
+widened from 3 to 6 input channels (zero-padded at initialisation), so an optical-flow
+visualisation enters the model alongside RGB at the input stage. Motion is what separates a
+dissolve from a camera pan, and a model that only sees appearance has to infer it. Because the
+flow is fused *before* patchification, the language backbone carries no additional visual tokens,
+so temporal awareness comes for free at the token budget.
+
+Training used a scalable FFmpeg-based data engine covering 59 transition effects, which addresses
+the severe class imbalance in public shot-boundary data.
+
+## Model Details
+
+| | |
+|---|---|
+| Base model | [Qwen/Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) |
+| Architecture | `Qwen3VLForConditionalGeneration` |
+| Precision | `bfloat16` |
+| Vision input channels | **6** (RGB + optical-flow visualisation, concatenated on the channel axis) |
+| Sampling frame rate | 25 fps |
+| Optical flow | [NeuFlow v2](https://huggingface.co/Study-is-happy/neuflow-v2), computed at inference time. Its weights (~30 MB) come from the Hub on first use, so expect a second download |
+| Input | One video; flow is computed for you |
+| Output | Transition segments as start/end times in seconds |
+| Saved with | `transformers` 4.57.3 |
+
+## Intended Use and Limitations
+
+**Intended for** detecting shot transitions in videos, both hard cuts and gradual effects, and
+as a baseline on the STD benchmark.
+
+**Limitations**
+
+- Not a frame-level cut-point classifier. The output is a time span per transition, not a single
+  boundary frame.
+- Not a general video-chat model. It was fine-tuned on one task with one prompt; the prompt ships
+  with the inference code and changing it changes the task.
+- Long videos should be split before inference. Cost is linear in duration, and the flow
+  visualisation is normalised over its whole input, so a long video and its parts are not the same
+  signal.
+
+## Usage
+
+No standalone `transformers` snippet is given here on purpose: a 3-channel one would run and
+return nonsense. Use the inference package, which needs Python 3.12, a CUDA GPU, and `ffmpeg`
+on `PATH`:
+
+```bash
+git clone https://github.com/heygen-com/TransVLM
+cd TransVLM/inference
+
+uv venv && source .venv/bin/activate
+uv sync --group cu130 --group dev        # cu128 if your driver is older than 570
+
+# Do not skip this. PyTorch 2.9.1 has a Conv3d bug below cuDNN 9.15, and the 6-channel
+# patch embed IS a Conv3d, so it is on the hot path of every forward pass. `uv sync`
+# resolves cuDNN back down, so re-run this after every sync. Activate the venv first:
+# unlike `uv sync`, `uv pip` installs into whatever environment is active.
+uv pip install nvidia-cudnn-cu13==9.16.0.29        # nvidia-cudnn-cu12 for cu128 / cu126
+python -c "import torch; print(torch.backends.cudnn.version())"   # must print 91600
+
+hf download HeyGenAI/TransVLM-Qwen3-VL-4B-Instruct --local-dir ./pretrained/TransVLM-v1
+
+python infer_video.py \
+  --video /path/to/video.mp4 \
+  --ckpt-dir ./pretrained/TransVLM-v1 \
+  --output-jsonl out.jsonl
+```
+
+The cuDNN package name tracks the CUDA flavour, and installing the wrong one is a silent
+no-op: it lands an unused package while torch keeps loading the other family. That is why
+the check prints a version number instead of trusting the install.
+
+Each output line carries `segments` as `{start_time, end_time}` pairs in seconds on the original
+video's timeline, plus timings and the full configuration that produced them.
+
+Three backends are supported: HuggingFace (default), vLLM and SGLang. Environment setup, every
+option and the output schema are documented in
+[`inference/README.md`](https://github.com/heygen-com/TransVLM/blob/main/inference/README.md).
+
+## Evaluation
+
+Qualitative results and the comparison against baselines are on the
+[project page](https://heygen-com.github.io/TransVLM/).
+
+The scorer that produced the paper's metric is released at
+[`evaluation/`](https://github.com/heygen-com/TransVLM/tree/main/evaluation). Note that the
+published figures cannot be reproduced bit-for-bit from the released artifacts alone. The
+benchmark ground truth and the pre-computed optical flow used for that run are not part of this
+release. The reasons are spelled out in
+[`evaluation/README.md`](https://github.com/heygen-com/TransVLM/blob/main/evaluation/README.md).
+
+## Release Progress
+
+- [x] Model weights
+- [x] Inference code
+- [x] Evaluation code
+- [ ] Data engine code
+- [ ] STD benchmark data
+- [ ] Re-annotated dataset labels
+- [ ] Leaderboard
+
+🚧 The remaining items are being prepared for release.
+
+## Citation
+
+```bibtex
+@inproceedings{chen2026transvlm,
+  title={TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions},
+  author={Chen, Ce and Ren, Yi and Li, Yuanming and Goriachko, Viktor and
+          Ye, Zhenhui and Guo, Zujin and Hong, Zhibin and Gong, Mingming},
+  booktitle={European Conference on Computer Vision},
+  year={2026},
+  organization={Springer}
+}
+```
+
+## License
+
+Apache License 2.0, inherited from [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct).