README.md

metadata

license: apache-2.0
library_name: pytorch
tags:
  - memsft
  - memory-decoder
  - router
  - qwen3
  - biology
  - biology-instructions
  - openswi
  - law

MemSFT-Qwen3-Routers

📄 Paper • 💻 GitHub • 🤗 HF Collection

Overview

This repository contains the learned token-level routers for the Qwen3 MemSFT release. A router predicts how the frozen backbone and an external domain memory should be combined at each decoding step.

The router checkpoints are auxiliary components, not standalone language models. Each router must be used with its matching backbone and memory checkpoint through the MemSFT implementation.

Available Routers

Domain	Backbone	Memory	Router directory
Biology-Instructions	Qwen3-8B	MemSFT-Qwen3-Bio-Memory-8B	Qwen3-8B-Bio-M8B-Router
Biology-Instructions	Qwen3-14B	MemSFT-Qwen3-Bio-Memory-8B	Qwen3-14B-Bio-M8B-Router
Biology-Instructions	Qwen3-32B	MemSFT-Qwen3-Bio-Memory-8B	Qwen3-32B-Bio-M8B-Router
Biology-Instructions	Qwen3-235B-A22B	MemSFT-Qwen3-Bio-Memory-8B	Qwen3-235B-A22B-Bio-M8B-Router
Biology-Instructions	Qwen3-14B	MemSFT-Qwen3-Bio-Memory-1.7B	Qwen3-14B-Bio-M1.7B-Router
Biology-Instructions	Qwen3-14B	MemSFT-Qwen3-Bio-Memory-4B	Qwen3-14B-Bio-M4B-Router
OpenSWI	Qwen3-8B	MemSFT-Qwen3-OpenSWI-Memory-8B	Qwen3-8B-OpenSWI-M8B-Router
OpenSWI	Qwen3-14B	MemSFT-Qwen3-OpenSWI-Memory-8B	Qwen3-14B-OpenSWI-M8B-Router
OpenSWI	Qwen3-32B	MemSFT-Qwen3-OpenSWI-Memory-8B	Qwen3-32B-OpenSWI-M8B-Router
OpenSWI	Qwen3-235B-A22B	MemSFT-Qwen3-OpenSWI-Memory-8B	Qwen3-235B-A22B-OpenSWI-M8B-Router
Law	Qwen3-14B	MemSFT-Qwen3-Law-Memory-8B	Qwen3-14B-Law-M8B-Router

Quick Start

Download the router that matches the selected backbone and memory:

from pathlib import Path

from huggingface_hub import snapshot_download

router_subdir = "Qwen3-14B-Bio-M8B-Router"
router_root = snapshot_download(
    repo_id="Jiarui-Wang/MemSFT-Qwen3-Routers",
    revision="v1.0",
    allow_patterns=[f"{router_subdir}/*"],
)
router_path = str(Path(router_root) / router_subdir)

Pass router_path together with the loaded backbone and memory to MemSFT:

from router.adaptive_memdec import AdaptiveMemoryDecoder

model = AdaptiveMemoryDecoder(
    base_lm=base,
    knn_generator=memory,
    router_path=router_path,
    router_device=device,
).eval()
model.set_tokenizer(tokenizer)

See the MemSFT repository and the model card for the selected memory checkpoint for complete installation, loading, and generation examples.

Intended Use and Limitations

These checkpoints are intended for reproducing the evaluated MemSFT Qwen3 configurations listed above. A router should not be used with a different backbone, memory size, or domain unless that combination is separately trained and evaluated. Performance outside the listed combinations has not been established.

License

The MemSFT router checkpoints are released under the Apache License 2.0. Upstream models, software, and datasets remain subject to their respective licenses and terms.

Citation

If you find MemSFT helpful in your research, please consider citing:

@misc{wang2026memsftmitigatingalignmenttax,
      title={MemSFT: Mitigating Alignment Tax with an External Parametric Memory},
      author={Jiarui Wang and Xiang Shi and Jiaqi Cao and Rubin Wei and Xiquan Wang and Hao Sun and Jingzhi Wang and Zhiqi Yang and Qipeng Guo and Bowen Zhou and Zhouhan Lin},
      year={2026},
      eprint={2607.25614},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2607.25614},
}

Contact

For questions and discussions, feel free to email wangjiarui1@sjtu.edu.cn.