Instructions to use sais-org/Polaris_Pro with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sais-org/Polaris_Pro with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="sais-org/Polaris_Pro")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("sais-org/Polaris_Pro")
model = AutoModelForMultimodalLM.from_pretrained("sais-org/Polaris_Pro")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use sais-org/Polaris_Pro with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sais-org/Polaris_Pro"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sais-org/Polaris_Pro",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/sais-org/Polaris_Pro

SGLang

How to use sais-org/Polaris_Pro with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "sais-org/Polaris_Pro" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sais-org/Polaris_Pro",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "sais-org/Polaris_Pro" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sais-org/Polaris_Pro",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use sais-org/Polaris_Pro with Docker Model Runner:
```
docker model run hf.co/sais-org/Polaris_Pro
```

Polaris_Pro / README.md

Descartes

Add SAM License + SAM3 config; composite-license model card

dd5ae9f verified 2 days ago

preview code

Raw

History Blame Contribute Delete

9.06 kB

metadata

license: other
license_name: apache-2.0-and-sam-license
license_link: LICENSE
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - multimodal
  - scientific
  - protein
  - rna
  - dna
  - molecule
  - weather
  - medical-imaging
base_model:
  - Qwen/Qwen3-VL-8B-Instruct
extra_gated_heading: >-
  You need to agree to Meta's SAM License to use the medical-image segmentation
  weights
extra_gated_description: >-
  The bulk of this model is Apache-2.0. The medical-image segmentation branch
  embeds SAM 3 weights, which are governed by Meta's SAM License (see
  SAM_LICENSE.txt). By accessing these weights you agree to that license,
  including its acceptable-use restrictions.

🤗 Model • 💻 GitHub • 📜 Technical Report (coming soon) • ⚖️ License: Apache-2.0 + SAM License

Polaris-Pro

Polaris-Pro is a unified scientific multimodal foundation model that supports scientific understanding and generation across Earth science, proteins, RNA, DNA, and small molecules within a single 8B model. Native scientific encoders/decoders wrap a shared Qwen3-VL-8B-Instruct backbone, so heterogeneous scientific data (sequences, molecular graphs, gridded physical fields, medical images) are reasoned about and generated in one representation space — natural language in and out, no per-task fine-tuning.

📜 Technical report coming soon.

Key features

Unified understanding and generation across 7 modalities through one natural-language interface.
Seven modalities, one 8B backbone (protein / RNA / DNA / molecule / weather / medical-image / text) via a modality router.
Native scientific encoders/decoders (ESM-2, RNA/DNA ConvFormers, molecular graph encoder, Swin-ViT weather tower, SAM-based image path) preserve domain structure a generic tokenizer would destroy.

Capabilities

Modality	Understanding	Generation
Protein	✅	—
RNA	✅	✅
DNA	✅	—
Molecule	✅	✅
Weather	—	✅
Medical image	—	✅
Text	✅	✅

_{Understanding = classification / regression / scientific QA. Generation: RNA sequence design · Molecule text → SMILES · Weather 10-day global ERA5 0.25° forecast · Medical-image text-prompted segmentation (SAM 3-based; Meta SAM License).}

Benchmarks

Polaris-Pro (8B) vs Biology-Instructions (Llama-3.1-8B, text-token, no scientific encoders) and Intern-S1-Pro (~1T MoE scientific model). Bold = best; underline = second-best.

Biological sequence understanding

Task	Metric	Polaris-Pro (8B)	Biology-Instructions (8B)	Intern-S1-Pro (~1T)
DNA · Epigenetic marks (EMP)	MCC	71.99	3.64	14.02
DNA · Promoter det. 300bp (PD300)	MCC	91.17	58.18	82.65
DNA · Core-promoter (CPD)	MCC	66.35	44.54	54.60
DNA · Enhancer activity (EA)	PCC	52.64	53.28	55.16
RNA · ncRNA function	Acc	91.46	63.09	34.50
RNA · Modification	AUC	96.03	59.06	57.77
RNA · APA isoform	R²	79.87	59.01	82.95
RNA · CRISPR on-target	Spearman ρ	28.76	-0.02	15.69
Protein · Stability	Spearman ρ	70.63	60.25	60.82
Protein · Fluorescence	Spearman ρ	70.12	2.57	78.14
Protein · Enzyme Commission	Fmax	68.65	19.79	72.70
Protein · Solubility	Acc	67.26	63.02	67.60
Cross-modal · RPI (RNA–protein)	MCC	76.49	74.26	58.51
Cross-modal · AAN (antibody–antigen)	MCC	42.96	1.06	44.76
Cross-modal · EPI (enhancer–promoter)	MCC	-0.03	3.37	-1.30

_{Aggregate over 20 biological-understanding benchmarks: Polaris-Pro matches or beats the ~1T Intern-S1-Pro on 10/20 and the same-scale 8B text-token baseline on 16/20.}

Molecule understanding (SMolInstruct)

Task	Metric	Polaris-Pro (8B)	LlaSMol
BBBP	Acc	96.95	74.60
HIV	Acc	97.00	96.70
SIDER	Acc	71.00	70.70
ClinTox	Acc	92.36	93.10
ESOL	RMSE ↓	0.550	1.150
Lipophilicity	RMSE ↓	0.628	1.010

Earth-science forecasting — vs ECMWF HRES (day-10, global ERA5 0.25°)

Variable	Metric	Polaris-Pro (8B)	ECMWF HRES (NWP)
Z500	RMSE ↓	≈740	≈810
T2M	RMSE ↓ (K)	≈2.65	≈2.90
MSL	RMSE ↓ (Pa)	≈680	≈745

_{Polaris-Pro tracks or beats the operational physics-based HRES system, with the advantage growing at longer lead times.}

Medical-image segmentation

Mean Dice (%) on the BiomedParse test splits, 102,855 image–prompt pairs across nine imaging modalities, versus six modality-native segmentation specialists.

Modality	# Samples	Polaris-Pro	BiomedParse	MedSAM	SAM	SAM3	DINO+MedSAM	DINO+SAM
All	102,855	91.20	90.73	83.55	71.29	35.40	15.37	15.10
CT	45,306	93.36	92.25	83.87	74.10	28.93	9.59	10.34
MRI	30,990	85.29	85.25	75.90	68.34	53.64	13.28	12.39
OCT	283	85.31	86.63	56.26	55.99	8.69	6.68	6.98
X-ray	13,840	98.02	98.28	97.75	81.35	39.96	37.22	30.63
Dermoscopy	65	98.08	97.11	97.35	88.23	51.47	81.28	78.29
Endoscopy	410	97.39	96.77	97.05	92.88	38.82	25.01	24.54
Fundus	800	91.33	91.50	88.06	57.16	18.58	3.19	2.73
Pathology	977	87.29	81.57	43.44	42.06	26.08	25.38	24.69
Ultrasound	10,184	90.54	91.03	89.76	57.47	5.23	17.12	22.91

_{Best overall Dice (All), and best on CT, MRI, pathology, dermoscopy, and endoscopy; on X-ray, Fundus, and Ultrasound the gap to BiomedParse is ≤ 0.5 Dice, and on the smallest split (OCT) it is 1.3.}

Usage

Runs via the accompanying code repository (custom multimodal architecture).

git clone https://github.com/Shanghai-Academy-of-AI-For-Science/Polaris-Pro && cd Polaris-Pro
pip install -r requirements.txt          # Python 3.10; transformers==5.0.0
hf download sais-org/Polaris_Pro --local-dir ./model

export PYTHONPATH=$PWD/code
python code/inference.py --model_path model --greedy --max_new_tokens 64 \
  --rna "GGATGCGATCATGTCTGCACTAACACACCGGATCCCATCAGAACTCCGAAGTTAAGCGTGCTTGGGCGGGAGTAGTACTAGGATGGGCGACCCCTTAGGAAGTACTCGTGTTGCATCCC" \
  --system "You are a non-coding RNA family classifier. Output only the family name, no other text." \
  --prompt $'<rna>\nWhich family does this non-coding RNA sequence belong to?'

All weights are contained in model.safetensors: the scientific encoders/decoders (ESM-2, the Suiren molecular graph encoder, the RNA/DNA ConvFormers, the Swin-ViT weather tower) and the fine-tuned SAM 3 branch used for medical-image segmentation.

Each task has a specific --system prompt that fixes the output format; see run_examples.sh in the repository for per-task examples, weather, and segmentation.

License

Composite license. Polaris-Pro's own components — the code, and all weights except the SAM 3 branch — are Apache-2.0, built on Qwen3-VL (Apache-2.0) and including merged ESM-2 (MIT) and Polaris/Suiren-derived encoders.

The medical-image segmentation branch embeds SAM 3 weights, which are governed by Meta's SAM License (SAM_LICENSE.txt, shipped alongside these weights). SAM 3 use is subject to that license, including its acceptable-use restrictions (no military / weapons / illegal uses; Trade-Control compliance). See THIRD_PARTY_LICENSES.md / NOTICE for the full third-party breakdown.

Citation

@misc{polarispro2026,
  title  = {Polaris-Pro: A Unified Scientific Multimodal Foundation Model},
  author = {Hesen Chen and Xinyu Su and Xiaomeng Yang and Yuetan Lin and Zixiong Yang and Zhiyu Tan and Hao Li},
  year   = {2026},
  note   = {https://huggingface.co/sais-org/Polaris_Pro}
}