---
license: other
license_name: apache-2.0-and-sam-license
license_link: LICENSE
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - multimodal
  - scientific
  - protein
  - rna
  - dna
  - molecule
  - weather
  - medical-imaging
base_model:
  - Qwen/Qwen3-VL-8B-Instruct
extra_gated_heading: You need to agree to Meta's SAM License to use the medical-image segmentation weights
extra_gated_description: >-
  The bulk of this model is Apache-2.0. The medical-image segmentation branch
  embeds SAM 3 weights, which are governed by Meta's SAM License (see
  SAM_LICENSE.txt). By accessing these weights you agree to that license,
  including its acceptable-use restrictions.
---

<div align="center">

[🤗 Model](https://huggingface.co/sais-org/Polaris_Pro) &nbsp;•&nbsp; [💻 GitHub](https://github.com/Shanghai-Academy-of-AI-For-Science/Polaris-Pro) &nbsp;•&nbsp; [📜 Technical Report (coming soon)](#) &nbsp;•&nbsp; [⚖️ License: Apache-2.0 + SAM License](https://github.com/Shanghai-Academy-of-AI-For-Science/Polaris-Pro/blob/main/LICENSE)

</div>

# Polaris-Pro

**Polaris-Pro is a unified scientific multimodal foundation model** that
supports scientific **understanding and generation** across Earth science,
proteins, RNA, DNA, and small molecules within a single **8B** model. Native
scientific encoders/decoders wrap a shared **Qwen3-VL-8B-Instruct** backbone, so
heterogeneous scientific data (sequences, molecular graphs, gridded physical
fields, medical images) are reasoned about and generated in one representation
space — natural language in and out, no per-task fine-tuning.

> 📜 **Technical report coming soon.**

## Key features

- **Unified understanding *and* generation** across 7 modalities through one
  natural-language interface.
- **Seven modalities, one 8B backbone** (protein / RNA / DNA / molecule /
  weather / medical-image / text) via a modality router.
- **Native scientific encoders/decoders** (ESM-2, RNA/DNA ConvFormers, molecular
  graph encoder, Swin-ViT weather tower, SAM-based image path) preserve domain
  structure a generic tokenizer would destroy.

## Capabilities

| Modality      | Understanding | Generation |
|:--------------|:-------------:|:----------:|
| Protein       | ✅            | —          |
| RNA           | ✅            | ✅         |
| DNA           | ✅            | —          |
| Molecule      | ✅            | ✅         |
| Weather       | —             | ✅         |
| Medical image | —             | ✅         |
| Text          | ✅            | ✅         |

<sub>**Understanding** = classification / regression / scientific QA. **Generation**: RNA sequence design · Molecule text → SMILES · Weather 10-day global ERA5 0.25° forecast · Medical-image text-prompted segmentation (SAM 3-based; Meta SAM License).</sub>

## Benchmarks

**Polaris-Pro** (**8B**) vs **Biology-Instructions** (Llama-3.1-**8B**, text-token,
no scientific encoders) and **Intern-S1-Pro** (**~1T** MoE scientific model).
**Bold** = best; <u>underline</u> = second-best.

### Biological sequence understanding

| Task | Metric | Polaris-Pro (8B) | Biology-Instructions (8B) | Intern-S1-Pro (~1T) |
|:-----|:------:|:----------------:|:-------------------------:|:-------------------:|
| DNA · Epigenetic marks (EMP) | MCC | **71.99** | 3.64 | <u>14.02</u> |
| DNA · Promoter det. 300bp (PD300) | MCC | **91.17** | 58.18 | <u>82.65</u> |
| DNA · Core-promoter (CPD) | MCC | **66.35** | 44.54 | <u>54.60</u> |
| DNA · Enhancer activity (EA) | PCC | 52.64 | <u>53.28</u> | **55.16** |
| RNA · ncRNA function | Acc | **91.46** | <u>63.09</u> | 34.50 |
| RNA · Modification | AUC | **96.03** | <u>59.06</u> | 57.77 |
| RNA · APA isoform | R² | <u>79.87</u> | 59.01 | **82.95** |
| RNA · CRISPR on-target | Spearman ρ | **28.76** | -0.02 | <u>15.69</u> |
| Protein · Stability | Spearman ρ | **70.63** | 60.25 | <u>60.82</u> |
| Protein · Fluorescence | Spearman ρ | <u>70.12</u> | 2.57 | **78.14** |
| Protein · Enzyme Commission | Fmax | <u>68.65</u> | 19.79 | **72.70** |
| Protein · Solubility | Acc | <u>67.26</u> | 63.02 | **67.60** |
| Cross-modal · RPI (RNA–protein) | MCC | **76.49** | <u>74.26</u> | 58.51 |
| Cross-modal · AAN (antibody–antigen) | MCC | <u>42.96</u> | 1.06 | **44.76** |
| Cross-modal · EPI (enhancer–promoter) | MCC | <u>-0.03</u> | **3.37** | -1.30 |

<sub>Aggregate over 20 biological-understanding benchmarks: Polaris-Pro matches or beats the ~1T Intern-S1-Pro on 10/20 and the same-scale 8B text-token baseline on 16/20.</sub>

### Molecule understanding (SMolInstruct)

| Task | Metric | Polaris-Pro (8B) | LlaSMol |
|:-----|:------:|:----------------:|:-------:|
| BBBP | Acc | **96.95** | 74.60 |
| HIV | Acc | **97.00** | 96.70 |
| SIDER | Acc | **71.00** | 70.70 |
| ClinTox | Acc | 92.36 | **93.10** |
| ESOL | RMSE ↓ | **0.550** | 1.150 |
| Lipophilicity | RMSE ↓ | **0.628** | 1.010 |

### Earth-science forecasting — vs ECMWF HRES (day-10, global ERA5 0.25°)

| Variable | Metric | Polaris-Pro (8B) | ECMWF HRES (NWP) |
|:---------|:------:|:----------------:|:----------------:|
| Z500 | RMSE ↓ | **≈740** | ≈810 |
| T2M | RMSE ↓ (K) | **≈2.65** | ≈2.90 |
| MSL | RMSE ↓ (Pa) | **≈680** | ≈745 |

<sub>Polaris-Pro tracks or beats the operational physics-based HRES system, with the advantage growing at longer lead times.</sub>

### Medical-image segmentation

Mean Dice (%) on the BiomedParse test splits, 102,855 image–prompt pairs across
nine imaging modalities, versus six modality-native segmentation specialists.

| Modality | # Samples | Polaris-Pro | BiomedParse | MedSAM | SAM | SAM3 | DINO+MedSAM | DINO+SAM |
|:---------|----------:|:-----------:|:-----------:|:------:|:---:|:----:|:-----------:|:--------:|
| **All**    | 102,855 | **91.20** | <u>90.73</u> | 83.55 | 71.29 | 35.40 | 15.37 | 15.10 |
| CT         |  45,306 | **93.36** | <u>92.25</u> | 83.87 | 74.10 | 28.93 |  9.59 | 10.34 |
| MRI        |  30,990 | **85.29** | <u>85.25</u> | 75.90 | 68.34 | 53.64 | 13.28 | 12.39 |
| OCT        |     283 | <u>85.31</u> | **86.63** | 56.26 | 55.99 |  8.69 |  6.68 |  6.98 |
| X-ray      |  13,840 | <u>98.02</u> | **98.28** | 97.75 | 81.35 | 39.96 | 37.22 | 30.63 |
| Dermoscopy |      65 | **98.08** | 97.11 | <u>97.35</u> | 88.23 | 51.47 | 81.28 | 78.29 |
| Endoscopy  |     410 | **97.39** | 96.77 | <u>97.05</u> | 92.88 | 38.82 | 25.01 | 24.54 |
| Fundus     |     800 | <u>91.33</u> | **91.50** | 88.06 | 57.16 | 18.58 |  3.19 |  2.73 |
| Pathology  |     977 | **87.29** | <u>81.57</u> | 43.44 | 42.06 | 26.08 | 25.38 | 24.69 |
| Ultrasound |  10,184 | <u>90.54</u> | **91.03** | 89.76 | 57.47 |  5.23 | 17.12 | 22.91 |

<sub>Best overall Dice (All), and best on CT, MRI, pathology, dermoscopy, and endoscopy; on X-ray, Fundus, and Ultrasound the gap to BiomedParse is ≤ 0.5 Dice, and on the smallest split (OCT) it is 1.3.</sub>

## Usage

Runs via the accompanying code repository (custom multimodal architecture).

```bash
git clone https://github.com/Shanghai-Academy-of-AI-For-Science/Polaris-Pro && cd Polaris-Pro
pip install -r requirements.txt          # Python 3.10; transformers==5.0.0
hf download sais-org/Polaris_Pro --local-dir ./model

export PYTHONPATH=$PWD/code
python code/inference.py --model_path model --greedy --max_new_tokens 64 \
  --rna "GGATGCGATCATGTCTGCACTAACACACCGGATCCCATCAGAACTCCGAAGTTAAGCGTGCTTGGGCGGGAGTAGTACTAGGATGGGCGACCCCTTAGGAAGTACTCGTGTTGCATCCC" \
  --system "You are a non-coding RNA family classifier. Output only the family name, no other text." \
  --prompt $'<rna>\nWhich family does this non-coding RNA sequence belong to?'
```

All weights are contained in `model.safetensors`: the scientific
encoders/decoders (ESM-2, the Suiren molecular graph encoder, the RNA/DNA
ConvFormers, the Swin-ViT weather tower) and the fine-tuned SAM 3 branch used
for medical-image segmentation.

Each task has a specific `--system` prompt that fixes the output format; see
`run_examples.sh` in the repository for per-task examples, weather, and segmentation.

## License

**Composite license.** Polaris-Pro's own components — the code, and all weights
except the SAM 3 branch — are **Apache-2.0**, built on Qwen3-VL (Apache-2.0) and
including merged ESM-2 (MIT) and Polaris/Suiren-derived encoders.

The **medical-image segmentation branch embeds SAM 3 weights**, which are
governed by **Meta's SAM License** (`SAM_LICENSE.txt`, shipped alongside these
weights). SAM 3 use is subject to that license, including its acceptable-use
restrictions (no military / weapons / illegal uses; Trade-Control compliance).
See `THIRD_PARTY_LICENSES.md` / `NOTICE` for the full third-party breakdown.

## Citation

```bibtex
@misc{polarispro2026,
  title  = {Polaris-Pro: A Unified Scientific Multimodal Foundation Model},
  author = {Hesen Chen and Xinyu Su and Xiaomeng Yang and Yuetan Lin and Zixiong Yang and Zhiyu Tan and Hao Li},
  year   = {2026},
  note   = {https://huggingface.co/sais-org/Polaris_Pro}
}
```