Hyun1A/DCAT-LLaMA-Vision2Video

DCAT · LLaMA-3.1-8B · Vision → Video

Project Page  |  Paper (PDF)  |  GitHub

DCAT (Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs, NeurIPS 2026 Spotlight) transfers cross-modal alignment from a strong donor MLLM to a weaker recipient MLLM with a training-free, closed-form merge that uses only a few hundred unlabeled calibration samples. This checkpoint is a LLaVA-Video (LLaMA-3.1-8B) video MLLM after DCAT merging with a LLaVA LLaMA-3.1-8B vision MLLM.

This repository contains the full merged weights (LLaVA-NeXT format). Loading needs LLaVA-NeXT with the small patch shipped in the GitHub repository (third_party/llava_next_dcat.patch), which adds the aapool_mlp projector used by this checkpoint.

Usage

git clone https://github.com/Hyun1A/DCAT.git && cd DCAT
pip install -r requirements_llama.txt
git clone https://github.com/LLaVA-VL/LLaVA-NeXT.git third_party/LLaVA-NeXT
git -C third_party/LLaVA-NeXT checkout bce12e4
git -C third_party/LLaVA-NeXT apply ../llava_next_dcat.patch
pip install -e third_party/LLaVA-NeXT --no-deps

python quick_start/inference.py --setting llama-video --model Hyun1A/DCAT-LLaMA-Vision2Video \
    --media sample.mp4 --question "What is the main content of this video?" --choices "option 1" "option 2" "option 3" "option 4"

See the GitHub repository for installation details, the inference code, and the scripts that reproduce the merge.

Citation

@inproceedings{seo2026dcat,
  title     = {Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs},
  author    = {Seo, Hoigi and Lee, Byung Hyun and Kim, Minjun and
               Mah, Dohyun and Lee, Jongho and Chun, Se Young},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2610.04580}
}