Intern-MemDec-4B

💻 GitHub Repo • 🤗 Model Collections • 📖 MemSFT Paper

Introduction

We introduce Intern-MemDec-4B, a memory decoder designed to extend Intern-S2 backbone models with specialized domain knowledge and capabilities. It is an auxiliary model component rather than a standalone language model: during inference, an Intern-S2 backbone and the memory decoder process the same context in parallel, while a lightweight token-level router dynamically combines their predictions.

Scientific knowledge is long-tailed and continuously evolving. A fixed post-trained foundation model cannot fully cover every specialized subfield, task protocol, or newly emerging discovery. Updating the entire backbone for each domain can also disturb its general reasoning, instruction-following, multimodal, and agentic capabilities. Intern-MemDec-4B instead provides a modular domain-extension mechanism. The Intern-S2 backbone remains intact and serves as the primary reasoning engine, while the attached memory supplies additional biological knowledge and recurring task patterns when they are useful.

Intern-MemDec-4B is trained by compressing retrieval-based evidence from domain data into a reusable parametric module. At inference time, it does not require access to the original retrieval datastore.

Important: Intern-MemDec-4B is not intended to be loaded or used as a standalone chat model. It must be deployed together with a compatible Intern-S2 backbone and the corresponding fusion configuration or router.

Features

• Modular Domain Extension. Intern-MemDec-4B adds specialized domain knowledge and task capabilities without updating the Intern-S2 backbone parameters. By turning domain adaptation into memory attachment rather than backbone rewriting, it provides a practical way to extend a general scientific foundation model with expertise in DNA, RNA, proteins, and biomolecular interactions while keeping the original backbone unchanged.
• Adaptive Fusion While Preserving General Capabilities. The Intern-S2 backbone and the biological memory process the same decoding context in parallel, and a lightweight token-level router dynamically determines their contributions to each next-token prediction. The memory branch can strengthen specialized predictions on biology-related inputs, while the backbone remains dominant for general reasoning, instruction following, factual knowledge, multimodal understanding, and agentic tasks, reducing interference with the model's original capabilities.
• Reusable and Extensible Parametric Memory. Domain evidence is compressed into a reusable parametric module, so online inference does not require the original retrieval datastore. Once trained, the memory can be attached to compatible Intern-S2 backbones using a matching router or fusion configuration. The same approach can also be applied to up-to-date data from other target domains, enabling independently trained memories to continually introduce new knowledge and capabilities as scientific fields evolve.

How It Works

For every decoding step, the Intern-S2 backbone and Intern-MemDec-4B receive the same input context and independently produce next-token predictions. A lightweight router examines signals from both branches and determines how much each should contribute to the final prediction.

On biology-related inputs, the router can increase the contribution of the memory branch to strengthen specialized predictions. On general or unrelated inputs, it can keep the backbone dominant so that the original reasoning, instruction-following, factual, multimodal, and agentic behavior is retained. New domain capabilities can therefore be added as independently trained, plug-and-play memories rather than through repeated full-model fine-tuning.

Performance

We evaluate Intern-MemDec-4B with Intern-S2-Preview-397B on all 21 tasks in Biology-Instructions. The attached biological memory improves the average score from 56.92 to 60.32 (+3.40 points). The evaluation in the Technical Report also shows a broadly comparable capability profile on general and multimodal benchmarks, supporting the intended use of Memory Decoder as a domain-extension interface that preserves the foundation model's general behavior.

(a) Biology-Instructions category performance

(b) Cross-domain capability profile

Biology-Instructions

All scores are reported on a 0–100 scale.

Biology-Instructions task	Intern-S2-Preview-397B	+ Intern-MemDec-4B
DNA-cpd	63.11	72.57
DNA-emp	19.95	27.25
DNA-enhancer activity	53.68	60.71
DNA-pd	84.40	89.12
DNA-tf-h	56.57	55.99
DNA-tf-m	56.96	67.09
Multi-sequence antibody-antigen	40.24	36.44
Multi-sequence promoter-enhancer interaction	22.46	38.47
Multi-sequence RNA-protein interaction	84.74	87.34
Multi-sequence siRNA efficiency	63.05	60.63
Protein-Fluorescence	70.48	72.23
Protein-FunctionEC	61.88	60.10
Protein-Solubility	68.60	68.00
Protein-Stability	69.67	67.80
Protein-Thermostability	58.44	53.97
RNA-CRISPROnTarget	6.61	17.18
RNA-Isoform	82.65	84.81
RNA-MeanRibosomeLoading	56.20	59.71
RNA-Modification	59.64	60.48
RNA-NoncodingRNAFamily	78.80	85.70
RNA-ProgrammableRNASwitches	37.13	41.23
Average score	56.92	60.32

Domain Coverage

This release covers the biology domain only. Its training and evaluation focus on biological sequence understanding and prediction tasks involving DNA, RNA, proteins, and biomolecular interactions. Representative task families include:

• DNA property, enhancer activity, and transcription-factor prediction;
• RNA modification, isoform, regulatory, and interaction prediction;
• protein function, solubility, stability, fluorescence, and thermostability prediction;
• antibody-antigen and other multi-sequence biomolecular interaction tasks.

Intern-MemDec-4B should not be interpreted as a general memory covering every scientific discipline. To extend Intern-S2 to another domain, a separate memory decoder can be trained on up-to-date, high-quality data from that target domain. The resulting memory can inject new domain knowledge and specialized capabilities into the backbone without rewriting the backbone parameters. This makes the memory interface suitable for continual domain extension as scientific data and task requirements evolve.

Quick Start

We recommend using LMDeploy (>= 0.15.0) to deploy the model suite — the internlm/Intern-S2-Preview-397B-FP8 LLM and the internlm/Intern-MemDec-4B memory model.

pip install "lmdeploy>=0.15.0"

On 4× H200 GPUs, launch the service with:

lmdeploy serve api_server internlm/Intern-S2-Preview-397B-FP8 \
  --backend pytorch \
  --tp 4 \
  --trust-remote-code \
  --hf-overrides '{"memory_model_path": "internlm/Intern-MemDec-4B"}'

The service exposes OpenAI-compatible APIs at http://0.0.0.0:23333 by default. Query it with:

curl http://localhost:23333/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "internlm/Intern-S2-Preview-397B-FP8",
    "messages": [{"role": "user", "content": "<PROTEIN> GSSGSSGPSKFIEGLRNEEATEGDTATLWCELSKAAPVEWRKGHETLRDGDRHSLRQDGSRCELQIRGLAVVDAGEYSCVCGQERTSATLTVRALPARFIESGPSSG </PROTEIN> What is the EC number associated with the enzymatic function of this protein? Please put the final enzyme within \\boxed{} using an EC number such as ECx.x.x.x, and separate multiple entries with commas."}]
  }'

For this example, the model returns the predicted enzyme classes:

\boxed{EC2.7.11.-,EC2.7.11.1}

Compatibility

• Intern-MemDec-4B must be used with a supported Intern-S2 backbone through a Memory Decoder-enabled inference runtime. Currently, Intern-S2-Preview-397B is the only supported backbone.
• Although the memory checkpoint is designed to be reusable across compatible backbone scales, adaptive routers depend on the selected backbone-memory pair. Use only a router or fusion configuration explicitly released for that pair.

Citation

@misc{wang2026memsftmitigatingalignmenttax,
      title={MemSFT: Mitigating Alignment Tax with an External Parametric Memory}, 
      author={Jiarui Wang and Xiang Shi and Jiaqi Cao and Rubin Wei and Xiquan Wang and Hao Sun and Jingzhi Wang and Zhiqi Yang and Qipeng Guo and Bowen Zhou and Zhouhan Lin},
      year={2026},
      eprint={2607.25614},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2607.25614}, 
}
@article{cao2026memory,
  title={Memory decoder: A pretrained, plug-and-play memory for large language models},
  author={Cao, Jiaqi and Wang, Jiarui and Wei, Rubin and Guo, Qipeng and Chen, Kai and Zhou, Bowen and Lin, Zhouhan},
  journal={Advances in Neural Information Processing Systems},
  volume={38},
  pages={115487--115510},
  year={2026}
}