|
|
--- |
|
|
license: apache-2.0 |
|
|
library_name: transformers |
|
|
pipeline_tag: image-classification |
|
|
--- |
|
|
|
|
|
# Model Merging with Functional Dual Anchors |
|
|
|
|
|
This repository is the official PyTorch implementation of the paper "[Model Merging with Functional Dual Anchors](https://huggingface.co/papers/2510.21223)", by Kexuan Shi, Yandong Wen, Weiyang Liu. |
|
|
|
|
|
**Functional Dual Anchors (FDAs)** propose a novel framework for efficiently integrating knowledge from multiple fine-tuned checkpoints of a shared foundation model. Unlike existing methods that operate in the parameter space, FDAs model knowledge in the input-representation space. FDAs are synthetic inputs whose induced gradients align with task vectors, capturing task-specific functional shifts relative to the pre-trained model. This perspective bridges joint multi-task training and post-hoc merging, offering both robustness and flexibility across various tasks, including vision, natural language processing, and natural language generation. |
|
|
|
|
|
<p align="center"> |
|
|
<img src="https://github.com/Sphere-AI-Lab/fda/raw/main/docs/assets/framework_trajectory.png" width="90%" /> |
|
|
</p> |
|
|
|
|
|
You can find more details on the [project page](https://spherelab.ai/fda/) and in the [official GitHub repository](https://github.com/Sphere-AI-Lab/fda/tree/main). |
|
|
|
|
|
## 🚀 Quick Start |
|
|
|
|
|
The official GitHub repository provides detailed instructions for setting up the environment, downloading checkpoints and corresponding FDAs, and running adaptation/construction scripts. |
|
|
|
|
|
For vision, NLP, and NLG tasks, the framework leverages base models such as `RoBERTa` and `Llama-2` from Hugging Face. |
|
|
|
|
|
### Checkpoints and Corresponding FDAs |
|
|
|
|
|
The checkpoints for vision, NLP, and NLG tasks and their corresponding FDAs are available for download via the [official GitHub repository](https://github.com/Sphere-AI-Lab/fda/tree/main). Specifically, vision and NLU FDAs are hosted on Hugging Face: [fda_for_vision](https://huggingface.co/datasets/SphereLab/FDA_for_Vision) and [fda_for_nlu](https://huggingface.co/datasets/SphereLab/FDA_for_NLU/tree/main). |
|
|
|
|
|
### Environment |
|
|
|
|
|
For Vision and NLP tasks, the environment can be installed by: |
|
|
```bash |
|
|
cd FDA/Vision #cd FDA/NLU |
|
|
# Create conda environment |
|
|
conda env create -f environment.yaml |
|
|
# Activate environment |
|
|
conda activate fda |
|
|
``` |
|
|
For NLG tasks, please use: ```NLG/environment.yaml``` |
|
|
|
|
|
### Adapt by FDAs |
|
|
|
|
|
Please follow the path comments in the code file ```adapt.py```, replace them with the paths to your local checkpoints and FDAs, and then run the following commands to reproduce the FDA adaptation results: |
|
|
```bash |
|
|
cd FDA/Vision #cd FDA/NLU cd FDA/NLG |
|
|
sh adapt.sh |
|
|
``` |
|
|
|
|
|
For models in NLG tasks, please split the model first: |
|
|
```bash |
|
|
cd FDA/NLG |
|
|
python split_model.py |
|
|
``` |
|
|
|
|
|
### Construct FDAs |
|
|
|
|
|
If you want to construct FDAs for your finetuned checkpoint, please follow the path comments in the code file ```construct_fda.py```, replace them with the paths to your finetuned checkpoints. Then, |
|
|
```bash |
|
|
sh construct.sh |
|
|
``` |
|
|
|
|
|
## Citation |
|
|
If you find this work useful, please consider citing: |
|
|
|
|
|
```bibtex |
|
|
@article{shi2025modelmergingfunctionaldual, |
|
|
title = {Model Merging with Functional Dual Anchors}, |
|
|
author = {Shi, Kexuan and Wen, Yandong and Liu, Weiyang}, |
|
|
year = {2025}, |
|
|
journal = {arXiv preprint arXiv:2510.21223}, |
|
|
archivePrefix = {arXiv}, |
|
|
primaryClass = {cs.LG}, |
|
|
url = {https://arxiv.org/abs/2510.21223} |
|
|
} |
|
|
``` |
