Model Merging with Functional Dual Anchors

This repository is the official PyTorch implementation of “Model Merging with Functional Dual Anchors”, by Kexuan Shi, Yandong Wen, Weiyang Liu.

Introduction

Model Merging has been an intriguing post-training strategy for integrating knowledge from mutliple finetuned checkpoints of a shared foundation model. Existing methods focuses on the operation in the parameter space, i.e, combing task vectors to mitgate knowledge confilctsm, thereby remain constrained by the complexity of the parameter space. In this work, we propose Functional Dual Anchors (FDAs), a framework (Figure 1(a)) that instead models the knowledge in the input-representation space. Specifically, FDAs are synthetic inputs whose induced gradients align with task vectors, capturing task-specific functional shifts relative to the pretrained model. Then, we use the FDAs to adapt the pretrained model. Comparing with the task vectors, FDAs can provide more robust and flexible trajectory for model merging, as shown in the Figure 1(b). FDAs provide an alternative perspective on model merging by extending input-space modeling to this setting and bridges joint multi-task training and post-hoc merging.

🚀 Quick Start

Checkpoints and Corresponding FDAs

To help you quickly get started with Functional Dual Anchors (FDAs), we provide download links for the checkpoints used in the paper, along with the corresponding FDAs.

Vision Tasks

We directly use the checkpoints provided at the following link: Vision Checkpoints (Google Drive). For convenience, you can download all vision models of our experiment in our hugging face page.

NLP Tasks

We adopt the pretrained RoBERTa-base and RoBERTa-large models from Hugging Face – RoBERTa Large. Then, we use the finetuning scripts from DARE to obtain the checkpoints on eight GLUE benchmarks. For convenience, you can download all NLP models of our experiment in our hugging face page.

NLG Tasks

• Base Model: Llama-2-13B (Meta)
• Expert Models: WizardMath-13B-V1.0; Llama-2-13B-Code-Alpaca

FDAs

The FDAs corresponding to the above checkpoints can be downloaded from: fda_for_vision and fda_for_nlu.

Please follow the path comments in the code and replace them with your local paths for checkpoints and FDAs,
then run the provided commands to reproduce the FDA adaptation results.

---

Environment

For Vision and NLP tasks, we use the same environment. It can be installed by:

cd FDA/Vision #cd FDA/NLU
# Create conda environment
conda env create -f environment.yaml
# Activate environment
conda activate fda

For NLU tasks, please use: NLG/environment.yaml

---

Adapt by FDAs

Please follow the path comments in the code file adapt.py, replace them with the paths to your local checkpoints and FDAs, and then run the following commands to reproduce the FDA adaptation results:

cd FDA/Vision #cd FDA/NLU cd FDA/NLG
sh adapt.sh

For models in NLG tasks, please split the model first:

cd FDA/NLG
python split_model.py

---

Construct FDAs

If you want to construct FDAs for your finetuned checkpoint, please follow the path comments in the code file construct_fda.py, replace them with the paths to your finetuned checkpoints. Then,

sh construct.sh

Acknowledgement

This repository uses codes and resources from Task Arithmetic, DARE, TSVM, WUDI, Prodistill.

Citation

If you find this work useful, please consider citing:

@article{shi2025modelmergingfunctionaldual,
  title     = {Model Merging with Functional Dual Anchors},
  author    = {Shi, Kexuan and Wen, Yandong and Liu, Weiyang},
  year      = {2025},
  journal   = {arXiv preprint arXiv:2510.21223},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url       = {https://arxiv.org/abs/2510.21223}
}