Instructions to use sais-org/Polaris_Pro with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sais-org/Polaris_Pro with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="sais-org/Polaris_Pro")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("sais-org/Polaris_Pro")
model = AutoModelForMultimodalLM.from_pretrained("sais-org/Polaris_Pro")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use sais-org/Polaris_Pro with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sais-org/Polaris_Pro"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sais-org/Polaris_Pro",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/sais-org/Polaris_Pro

SGLang

How to use sais-org/Polaris_Pro with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "sais-org/Polaris_Pro" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sais-org/Polaris_Pro",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "sais-org/Polaris_Pro" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sais-org/Polaris_Pro",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use sais-org/Polaris_Pro with Docker Model Runner:
```
docker model run hf.co/sais-org/Polaris_Pro
```

Polaris_Pro / README.md

Descartes

Add SAM License + SAM3 config; composite-license model card

dd5ae9f verified 4 days ago

preview code

Raw

History Blame Contribute Delete

9.06 kB

	---
	license: other
	license_name: apache-2.0-and-sam-license
	license_link: LICENSE
	library_name: transformers
	pipeline_tag: image-text-to-text
	tags:
	- multimodal
	- scientific
	- protein
	- rna
	- dna
	- molecule
	- weather
	- medical-imaging
	base_model:
	- Qwen/Qwen3-VL-8B-Instruct
	extra_gated_heading: You need to agree to Meta's SAM License to use the medical-image segmentation weights
	extra_gated_description: >-
	The bulk of this model is Apache-2.0. The medical-image segmentation branch
	embeds SAM 3 weights, which are governed by Meta's SAM License (see
	SAM_LICENSE.txt). By accessing these weights you agree to that license,
	including its acceptable-use restrictions.
	---

	<div align="center">

	[🤗 Model](https://huggingface.co/sais-org/Polaris_Pro)  •  [💻 GitHub](https://github.com/Shanghai-Academy-of-AI-For-Science/Polaris-Pro)  •  [📜 Technical Report (coming soon)](#)  •  [⚖️ License: Apache-2.0 + SAM License](https://github.com/Shanghai-Academy-of-AI-For-Science/Polaris-Pro/blob/main/LICENSE)

	</div>

	# Polaris-Pro

	Polaris-Pro is a unified scientific multimodal foundation model that
	supports scientific understanding and generation across Earth science,
	proteins, RNA, DNA, and small molecules within a single 8B model. Native
	scientific encoders/decoders wrap a shared Qwen3-VL-8B-Instruct backbone, so
	heterogeneous scientific data (sequences, molecular graphs, gridded physical
	fields, medical images) are reasoned about and generated in one representation
	space — natural language in and out, no per-task fine-tuning.

	> 📜 Technical report coming soon.

	## Key features

	- *Unified understanding and* generation** across 7 modalities through one
	natural-language interface.
	- Seven modalities, one 8B backbone (protein / RNA / DNA / molecule /
	weather / medical-image / text) via a modality router.
	- Native scientific encoders/decoders (ESM-2, RNA/DNA ConvFormers, molecular
	graph encoder, Swin-ViT weather tower, SAM-based image path) preserve domain
	structure a generic tokenizer would destroy.

	## Capabilities

	\| Modality \| Understanding \| Generation \|
	\|:--------------\|:-------------:\|:----------:\|
	\| Protein \| ✅ \| — \|
	\| RNA \| ✅ \| ✅ \|
	\| DNA \| ✅ \| — \|
	\| Molecule \| ✅ \| ✅ \|
	\| Weather \| — \| ✅ \|
	\| Medical image \| — \| ✅ \|
	\| Text \| ✅ \| ✅ \|

	<sub>Understanding = classification / regression / scientific QA. Generation: RNA sequence design · Molecule text → SMILES · Weather 10-day global ERA5 0.25° forecast · Medical-image text-prompted segmentation (SAM 3-based; Meta SAM License).</sub>

	## Benchmarks

	Polaris-Pro (8B) vs Biology-Instructions (Llama-3.1-8B, text-token,
	no scientific encoders) and Intern-S1-Pro (~1T MoE scientific model).
	Bold = best; <u>underline</u> = second-best.

	### Biological sequence understanding

	\| Task \| Metric \| Polaris-Pro (8B) \| Biology-Instructions (8B) \| Intern-S1-Pro (~1T) \|
	\|:-----\|:------:\|:----------------:\|:-------------------------:\|:-------------------:\|
	\| DNA · Epigenetic marks (EMP) \| MCC \| 71.99 \| 3.64 \| <u>14.02</u> \|
	\| DNA · Promoter det. 300bp (PD300) \| MCC \| 91.17 \| 58.18 \| <u>82.65</u> \|
	\| DNA · Core-promoter (CPD) \| MCC \| 66.35 \| 44.54 \| <u>54.60</u> \|
	\| DNA · Enhancer activity (EA) \| PCC \| 52.64 \| <u>53.28</u> \| 55.16 \|
	\| RNA · ncRNA function \| Acc \| 91.46 \| <u>63.09</u> \| 34.50 \|
	\| RNA · Modification \| AUC \| 96.03 \| <u>59.06</u> \| 57.77 \|
	\| RNA · APA isoform \| R² \| <u>79.87</u> \| 59.01 \| 82.95 \|
	\| RNA · CRISPR on-target \| Spearman ρ \| 28.76 \| -0.02 \| <u>15.69</u> \|
	\| Protein · Stability \| Spearman ρ \| 70.63 \| 60.25 \| <u>60.82</u> \|
	\| Protein · Fluorescence \| Spearman ρ \| <u>70.12</u> \| 2.57 \| 78.14 \|
	\| Protein · Enzyme Commission \| Fmax \| <u>68.65</u> \| 19.79 \| 72.70 \|
	\| Protein · Solubility \| Acc \| <u>67.26</u> \| 63.02 \| 67.60 \|
	\| Cross-modal · RPI (RNA–protein) \| MCC \| 76.49 \| <u>74.26</u> \| 58.51 \|
	\| Cross-modal · AAN (antibody–antigen) \| MCC \| <u>42.96</u> \| 1.06 \| 44.76 \|
	\| Cross-modal · EPI (enhancer–promoter) \| MCC \| <u>-0.03</u> \| 3.37 \| -1.30 \|

	<sub>Aggregate over 20 biological-understanding benchmarks: Polaris-Pro matches or beats the ~1T Intern-S1-Pro on 10/20 and the same-scale 8B text-token baseline on 16/20.</sub>

	### Molecule understanding (SMolInstruct)

	\| Task \| Metric \| Polaris-Pro (8B) \| LlaSMol \|
	\|:-----\|:------:\|:----------------:\|:-------:\|
	\| BBBP \| Acc \| 96.95 \| 74.60 \|
	\| HIV \| Acc \| 97.00 \| 96.70 \|
	\| SIDER \| Acc \| 71.00 \| 70.70 \|
	\| ClinTox \| Acc \| 92.36 \| 93.10 \|
	\| ESOL \| RMSE ↓ \| 0.550 \| 1.150 \|
	\| Lipophilicity \| RMSE ↓ \| 0.628 \| 1.010 \|

	### Earth-science forecasting — vs ECMWF HRES (day-10, global ERA5 0.25°)

	\| Variable \| Metric \| Polaris-Pro (8B) \| ECMWF HRES (NWP) \|
	\|:---------\|:------:\|:----------------:\|:----------------:\|
	\| Z500 \| RMSE ↓ \| ≈740 \| ≈810 \|
	\| T2M \| RMSE ↓ (K) \| ≈2.65 \| ≈2.90 \|
	\| MSL \| RMSE ↓ (Pa) \| ≈680 \| ≈745 \|

	<sub>Polaris-Pro tracks or beats the operational physics-based HRES system, with the advantage growing at longer lead times.</sub>

	### Medical-image segmentation

	Mean Dice (%) on the BiomedParse test splits, 102,855 image–prompt pairs across
	nine imaging modalities, versus six modality-native segmentation specialists.

	\| Modality \| # Samples \| Polaris-Pro \| BiomedParse \| MedSAM \| SAM \| SAM3 \| DINO+MedSAM \| DINO+SAM \|
	\|:---------\|----------:\|:-----------:\|:-----------:\|:------:\|:---:\|:----:\|:-----------:\|:--------:\|
	\| All \| 102,855 \| 91.20 \| <u>90.73</u> \| 83.55 \| 71.29 \| 35.40 \| 15.37 \| 15.10 \|
	\| CT \| 45,306 \| 93.36 \| <u>92.25</u> \| 83.87 \| 74.10 \| 28.93 \| 9.59 \| 10.34 \|
	\| MRI \| 30,990 \| 85.29 \| <u>85.25</u> \| 75.90 \| 68.34 \| 53.64 \| 13.28 \| 12.39 \|
	\| OCT \| 283 \| <u>85.31</u> \| 86.63 \| 56.26 \| 55.99 \| 8.69 \| 6.68 \| 6.98 \|
	\| X-ray \| 13,840 \| <u>98.02</u> \| 98.28 \| 97.75 \| 81.35 \| 39.96 \| 37.22 \| 30.63 \|
	\| Dermoscopy \| 65 \| 98.08 \| 97.11 \| <u>97.35</u> \| 88.23 \| 51.47 \| 81.28 \| 78.29 \|
	\| Endoscopy \| 410 \| 97.39 \| 96.77 \| <u>97.05</u> \| 92.88 \| 38.82 \| 25.01 \| 24.54 \|
	\| Fundus \| 800 \| <u>91.33</u> \| 91.50 \| 88.06 \| 57.16 \| 18.58 \| 3.19 \| 2.73 \|
	\| Pathology \| 977 \| 87.29 \| <u>81.57</u> \| 43.44 \| 42.06 \| 26.08 \| 25.38 \| 24.69 \|
	\| Ultrasound \| 10,184 \| <u>90.54</u> \| 91.03 \| 89.76 \| 57.47 \| 5.23 \| 17.12 \| 22.91 \|

	<sub>Best overall Dice (All), and best on CT, MRI, pathology, dermoscopy, and endoscopy; on X-ray, Fundus, and Ultrasound the gap to BiomedParse is ≤ 0.5 Dice, and on the smallest split (OCT) it is 1.3.</sub>

	## Usage

	Runs via the accompanying code repository (custom multimodal architecture).

	```bash
	git clone https://github.com/Shanghai-Academy-of-AI-For-Science/Polaris-Pro && cd Polaris-Pro
	pip install -r requirements.txt # Python 3.10; transformers==5.0.0
	hf download sais-org/Polaris_Pro --local-dir ./model

	export PYTHONPATH=$PWD/code
	python code/inference.py --model_path model --greedy --max_new_tokens 64 \
	--rna "GGATGCGATCATGTCTGCACTAACACACCGGATCCCATCAGAACTCCGAAGTTAAGCGTGCTTGGGCGGGAGTAGTACTAGGATGGGCGACCCCTTAGGAAGTACTCGTGTTGCATCCC" \
	--system "You are a non-coding RNA family classifier. Output only the family name, no other text." \
	--prompt $'<rna>\nWhich family does this non-coding RNA sequence belong to?'
	```

	All weights are contained in `model.safetensors`: the scientific
	encoders/decoders (ESM-2, the Suiren molecular graph encoder, the RNA/DNA
	ConvFormers, the Swin-ViT weather tower) and the fine-tuned SAM 3 branch used
	for medical-image segmentation.

	Each task has a specific `--system` prompt that fixes the output format; see
	`run_examples.sh` in the repository for per-task examples, weather, and segmentation.

	## License

	Composite license. Polaris-Pro's own components — the code, and all weights
	except the SAM 3 branch — are Apache-2.0, built on Qwen3-VL (Apache-2.0) and
	including merged ESM-2 (MIT) and Polaris/Suiren-derived encoders.

	The medical-image segmentation branch embeds SAM 3 weights, which are
	governed by Meta's SAM License (`SAM_LICENSE.txt`, shipped alongside these
	weights). SAM 3 use is subject to that license, including its acceptable-use
	restrictions (no military / weapons / illegal uses; Trade-Control compliance).
	See `THIRD_PARTY_LICENSES.md` / `NOTICE` for the full third-party breakdown.

	## Citation

	```bibtex
	@misc{polarispro2026,
	title = {Polaris-Pro: A Unified Scientific Multimodal Foundation Model},
	author = {Hesen Chen and Xinyu Su and Xiaomeng Yang and Yuetan Lin and Zixiong Yang and Zhiyu Tan and Hao Li},
	year = {2026},
	note = {https://huggingface.co/sais-org/Polaris_Pro}
	}
	```