README.md

---
license: apache-2.0
language:
- en
pipeline_tag: image-text-to-text
tags:
- vision-language-model
- vlm
- multimodal
- llava-onevision
- cross-layer-injection
- eccv2026
library_name: transformers
base_model:
- lmms-lab/llava-onevision-qwen2-0.5b-mid-stage-a4
- google/siglip-so400m-patch14-384
---

# CLI-0.5B: Cross-Layer Injection for Deep Vision-Language Fusion

Official model checkpoint of **"From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion"** (ECCV 2026).

CLI-0.5B is the 0.5B-parameter variant of our **Cross-Layer Injection (CLI)** framework, built on top of [LLaVA-OneVision-Qwen2-0.5B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-0.5b-mid-stage-a4) (Qwen2-0.5B LLM + SigLIP-so400m-patch14-384 vision encoder). It replaces the conventional one-to-one vision–language bridge with a dynamic **many-to-many** connection, allowing every designated LLM decoder layer to query the full visual hierarchy on demand.

> **Abstract.** Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection that links only the output of the vision encoder to the input of the large language model (LLM). This static architecture fundamentally limits the ability of LLMs to achieve comprehensive alignment with hierarchical visual knowledge, compromising their capacity to accurately integrate local details with global semantics into coherent reasoning. To resolve this, we introduce **Cross-Layer Injection (CLI)**, a novel framework that forges a dynamic "many-to-many" bridge between the two modalities. CLI consists of two synergistic, parameter-efficient components: an **Adaptive Multi-Projection (AMP)** module that harmonizes features from diverse vision layers, and an **Adaptive Gating Fusion (AGF)** mechanism that empowers the LLM to selectively inject the most relevant visual information based on its real-time decoding context. We validate the effectiveness and versatility of CLI by integrating it into LLaVA-OneVision and LLaVA-1.5. Extensive experiments on 28 diverse benchmarks demonstrate significant performance improvements, establishing CLI as a scalable paradigm that unlocks deeper multimodal understanding by granting LLMs on-demand access to the full visual hierarchy. Code is available at https://github.com/codefuse-ai/CLI.

## Key Features

- **Many-to-Many Fusion** — CLI replaces the conventional one-to-one vision–language bridge with a dynamic many-to-many architecture; each LLM decoder layer can query the full visual hierarchy on demand.
- **Adaptive Multi-Projection (AMP)** — Parameter-efficient LoRA-based projectors that harmonize features from diverse vision encoder layers into a shared semantic space.
- **Adaptive Gating Fusion (AGF)** — A query-based attention gate that dynamically selects and injects the most relevant visual information based on the LLM's real-time decoding context.
- **Architecture-Agnostic** — Validated on both LLaVA-OneVision (0.5B/7B) and LLaVA-1.5-7B. This checkpoint is the **0.5B** variant.
- **Minimal Overhead** — Only ~1.3% inference memory increase with marginal latency impact.

## Model Details

| | |
|:---|:---|
| **Base VLM** | LLaVA-OneVision-Qwen2-0.5B (mid-stage) |
| **LLM** | Qwen2-0.5B (24 decoder layers) |
| **Vision Encoder** | SigLIP-so400m-patch14-384 (28 layers) |
| **Projector** | `mlp2x_gelu` |
| **LLM injection layers** | `RANGE-1-24-4` → layers 1, 5, 9, 13, 17, 21 |
| **Vision extraction layers** | `RANGE-1-28-4` → layers 1, 5, 9, 13, 17, 21, 25 |
| **Image resolution** | `anyres_max_9` (spatial unpad, up to 6×6 grid) |
| **Model max length** | 32768 |
| **Precision** | bf16 |
| **License** | Apache 2.0 |

This model card is a model checkpoint hosted here for convenience. To actually load and run CLI-0.5B you need the custom model implementation (AMP + AGF) shipped with the [`codefuse-ai/CLI`](https://github.com/codefuse-ai/CLI) codebase — a plain `transformers` LlavaOneVision load will **not** reproduce CLI behavior.

## How to Use

CLI requires the custom architecture from the [`codefuse-ai/CLI`](https://github.com/codefuse-ai/CLI) repository. The layer configuration passed at load time **must match** the training configuration above.

### 1. Install

```bash
git clone https://github.com/codefuse-ai/CLI.git
cd CLI

conda create -n cli python=3.10 -y
conda activate cli

pip install -e ".[train]"
pip install flash-attn --no-build-isolation
```

### 2. Download this checkpoint

```bash
# Option A: huggingface-cli
huggingface-cli download codefuse-ai/CLI-0.5B --local-dir ./checkpoints/CLI-0.5B

# Option B: git-lfs
git clone https://huggingface.co/codefuse-ai/CLI-0.5B ./checkpoints/CLI-0.5B
```

### 3. Inference

The way to run CLI-0.5B is through the repo's evaluation/inference entry point, which instantiates the AMP + AGF modules and wires the cross-layer connections using the `RANGE-1-24-4` / `RANGE-1-28-4` configuration:

```bash
python run_eval_cli.py \
    --model_path ./checkpoints/CLI-0.5B \
    --eval_tasks mme,mmmu,ai2d \
    --vlm_exp_layers RANGE-1-24-4 \
    --vision_exp_layers RANGE-1-28-4 \
    --num_gpus 1
```

> ⚠️ **Important:** The `--vlm_exp_layers RANGE-1-24-4` and `--vision_exp_layers RANGE-1-28-4` flags are mandatory — they rebuild the CLI architecture (which LLM layers receive injections, which vision layers are extracted). Loading with the default one-to-one projector path or with `transformers.LlavaOnevisionForConditionalGeneration` will not reproduce CLI.

For single-turn visual chat / custom inference, use the LLaVA-OneVision conversation template (`qwen_1_5` prompt version) with `anyres_max_9` image preprocessing. See `run_eval_cli.py` and `llava/eval/` in the repo for reference implementations.

## Evaluation

We evaluate on 28 benchmarks across three categories using the [LMMs-Eval](https://github.com/EvolvingLMMs-Lab/lmms-eval) framework:

- **Chart / Diagram / Document:** `ai2d`, `chartqa`, `docvqa_val/test`, `infovqa_val/test`
- **Perception & Reasoning:** `mme`, `mmbench_en_dev`, `mmvet`, `mmmu`, `mmstar`, `mathvista_testmini`, `mathverse_*`, `gqa`, `ok_vqa`, `scienceqa_img`, `seedbench`, `pope`
- **Real-world & Visual Chat:** `realworldqa`, `llava_in_the_wild`

Run all benchmarks in one command:

```bash
python run_eval_cli.py \
    --model_path ./checkpoints/CLI-0.5B \
    --eval_tasks ai2d,chartqa,docvqa_val,docvqa_test,infovqa_val,infovqa_test,mme,mmbench_en_dev,mmvet,mmmu,mmstar,mathvista_testmini,gqa,ok_vqa,scienceqa_img,seedbench,pope,realworldqa,llava_in_the_wild \
    --vlm_exp_layers RANGE-1-24-4 \
    --vision_exp_layers RANGE-1-28-4 \
    --num_gpus 1
```

## Citation

If you find this work useful, please cite:

```bibtex
@inproceedings{chen2026cli,
    title={From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion},
    author={Chen, Cheng and Guo, Yuyu and Zeng, Pengpeng and Song, Jingkuan and Di, Peng and Yu, Hang and Gao, Lianli},
    booktitle={European Conference on Computer Vision (ECCV)},
    year={2026}
}
```

## Acknowledgements

This checkpoint is built upon [LLaVA-OneVision-Qwen2-0.5B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-0.5b-mid-stage-a4), [SigLIP](https://huggingface.co/google/siglip-so400m-patch14-384), and the [LLaVA-NeXT](https://github.com/LLaVA-VL/LLaVA-NeXT) framework, and is evaluated with [LMMs-Eval](https://github.com/EvolvingLMMs-Lab/lmms-eval).

## License

Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).