I3D-ViT

Introduction

I3D-ViT is the standalone vision encoder obtained after the vision encoder pre-training stage (Stage 0) described in the paper. It contains the VideoChat3VisionModel implementation, vision configuration, and the ViT weights. This vision encoder is subsequently used to initialize the visual backbone of MCG-NJU/VideoChat3-4B.

📄 Paper · 🌐 Homepage · 💻 GitHub · 🤗 Paper Page

Example usage

For image input:

from PIL import Image
import torch
from transformers import AutoImageProcessor, AutoModel

model_path = "MCG-NJU/I3D-ViT"
image_path = "example.jpg"

model = AutoModel.from_pretrained(
    model_path,
    trust_remote_code=True,
    dtype="auto",
    device_map="auto",
).eval()
processor = AutoImageProcessor.from_pretrained(model_path)

image = Image.open(image_path).convert("RGB")
inputs = processor(images=image, return_tensors="pt")

pixel_values = inputs["pixel_values"].to(
    device=model.device,
    dtype=model.dtype,
)
grid_thws = inputs["image_grid_thw"].to(model.device)

with torch.inference_mode():
    image_features: list = model(
        pixel_values=pixel_values,
        grid_thws=grid_thws,
    )

print(image_features[0].shape)

For video input:

import torch
from transformers import AutoModel, AutoVideoProcessor


model_path = "MCG-NJU/I3D-ViT"
video_path = "example.mp4"
num_frames = 4

model = AutoModel.from_pretrained(
    model_path,
    trust_remote_code=True,
    dtype="auto",
    device_map="auto",
).eval()
processor = AutoVideoProcessor.from_pretrained(
    model_path,
    trust_remote_code=True,
)

inputs = processor(
    videos=video_path,
    fps=None,
    num_frames=num_frames,
    return_tensors="pt",
)

pixel_values = inputs["pixel_values_videos"].to(
    device=model.device,
    dtype=model.dtype,
)
grid_thws = inputs["video_grid_thw"].to(model.device)

with torch.inference_mode():
    video_features: list = model(
        pixel_values=pixel_values,
        grid_thws=grid_thws,
    )

print(video_features[0].shape)

Citation

@misc{videochat3,
      title={VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding}, 
      author={Xinhao Li and Yuhan Zhu and Xiangyu Zeng and Yuhao Dong and Haoning Wu and Zhiqiu Zhang and Yuandong Yang and Changlian Ma and Qingyu Zhang and Yansong Shi and Xinyu Chen and Haoran Chen and Zizheng Huang and Jun Zhang and Kun Ouyang and Lin Sui and Ziang Yan and Yicheng Xu and Chenting Wang and Yinan He and Hongjie Zhang and Yi Wang and Yu Qiao and Yali Wang and Ziwei Liu and Kai Chen and Limin Wang},
      year={2026},
      eprint={2607.14935},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.14935}, 
}