README.md

---
base_model: Qwen/Qwen2.5-7B-Instruct
library_name: peft
license: apache-2.0
tags:
- multimodal
- vision-language
- lora
- ssval
- vapi
- mllm
- distillation
---

# SSVAL-7B

This is the released checkpoint of **SSVAL (Spatial-Spectral Visual
Anchor Learning)** built on top of Qwen2.5-7B-Instruct with a CLIP
ViT-L/14-336 vision tower. It is the artefact produced by the
two-stage pipeline described in the paper:

1. **Projector pretraining** on LLaVA-Pretrain LCS-558K.
2. **Visual instruction tuning** on LLaVA-665K with SPARA + SPERA losses.

The checkpoint ships as a **PEFT LoRA adapter + mm-projector**: it is
*not* a standalone model. To use it you must pair it with the base
LLM `Qwen/Qwen2.5-7B-Instruct` and the vision tower
`openai/clip-vit-large-patch14-336`.

## Files

| File | Size | What it is |
|---|---|---|
| `adapter_model.safetensors` | ~620 MB | LoRA adapter (rank 128, alpha 256) over the LLM's attention and MLP projections |
| `non_lora_trainables.bin` | ~170 MB | Multimodal projector and any other non-LoRA trainables saved separately |
| `adapter_config.json` | <1 KB | PEFT adapter configuration |
| `config.json` | ~4 KB | Full model configuration including vision-side and CPD settings |
| `trainer_state.json` | ~3 MB | Training-step / loss history (optional, kept for reproducibility) |

## Required base dependencies

```python
# pip install: transformers==4.37.2, torch>=2.1, peft>=0.10,
#              Pillow, accelerate, einops
```

See the companion code repository
[`msls38/SSVAL_code`](https://github.com/msls38/SSVAL_code) for the
load_pretrained_model helper, evaluation scripts, and full dependency
pins.

## Vision encoder and VFM teacher

| Component | Choice |
|---|---|
| Vision tower (`mm_vision_tower`) | `clip-vit-large-patch14-336` |
| Patch selection layer (`mm_vision_select_layer`) | -2 (second-to-last) |
| Feature type (`mm_vision_select_feature`) | `patch` |
| Image aspect ratio | `pad` |
| Patch merge | `flat` |
| MM projector type | `mlp2x_gelu` |
| Image-token wrapping | `<image>` only (no `<im_start>` / `<im_end>`) |

**VFM teacher (training-time only, not needed at inference).**

| Field | Value |
|---|---|
| `vfm_target` | `dinov2-vit-b` |
| `num_vision_prompts` | 4 |
| `num_vfm_prompts` | 4 |
| `cpd_target_layers` | `[14]` |
| Multi-scale factors | `[1, 2, 4]` |

## How to load and use

The companion repo's evaluation entry points expect the following on-disk
layout:

```
checkpoints/
├── Qwen2.5-7B-Instruct/         # base LLM, from Qwen/Qwen2.5-7B-Instruct
└── clip-vit-large-patch14-336/  # vision tower, from openai/clip-vit-large-patch14-336
```

Then, for single-image inference:

```bash
python SSVAL/src/inference.py \
    --model-path ./SSVAL-7B \
    --model-base ./checkpoints/Qwen2.5-7B-Instruct \
    --image-path ./my_image.jpg \
    --prompt "Describe this image."
```

For benchmark evaluation, e.g. MME:

```bash
python SSVAL/src/eval_mme.py \
    --model-path ./SSVAL-7B \
    --model-base ./checkpoints/Qwen2.5-7B-Instruct \
    --mme-path ./playground/benchmarks/MME/MME_Benchmark
```

The benchmark scripts run end-to-end without modification once the
companion code is installed and the dataset / benchmark directories are
populated as described in the repository README.

## Citation

If you use this checkpoint, please cite the SSVAL paper (full bibtex to
be added once the paper is publicly available).

## License

- The LoRA weights and configuration in this repository are released
  under the **Apache License 2.0** (matching the SSVAL code repository).
- The base LLM `Qwen2.5-7B-Instruct` is subject to its own license
  ([Qwen2.5 LICENSE](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)).
- The vision tower is subject to the **OpenAI CLIP LICENSE**.