Image-Text-to-Text
Transformers
Safetensors
English
Chinese
qwen3_5_moe
exomind
scientific-reasoning
scientific-research
agentic
tool-use
multimodal
vision-language
qwen3.5
conversational
Instructions to use AI4SGI/ExoMind with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use AI4SGI/ExoMind with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="AI4SGI/ExoMind") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("AI4SGI/ExoMind") model = AutoModelForMultimodalLM.from_pretrained("AI4SGI/ExoMind", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use AI4SGI/ExoMind with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AI4SGI/ExoMind" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AI4SGI/ExoMind", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/AI4SGI/ExoMind
- SGLang
How to use AI4SGI/ExoMind with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "AI4SGI/ExoMind" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AI4SGI/ExoMind", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "AI4SGI/ExoMind" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AI4SGI/ExoMind", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use AI4SGI/ExoMind with Docker Model Runner:
docker model run hf.co/AI4SGI/ExoMind
Add ExoMind model card
Browse files
README.md
CHANGED
|
@@ -1,3 +1,214 @@
|
|
| 1 |
---
|
|
|
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
library_name: transformers
|
| 3 |
license: apache-2.0
|
| 4 |
+
base_model: Qwen/Qwen3.5-35B-A3B
|
| 5 |
+
base_model_relation: finetune
|
| 6 |
+
pipeline_tag: image-text-to-text
|
| 7 |
+
language:
|
| 8 |
+
- en
|
| 9 |
+
- zh
|
| 10 |
+
tags:
|
| 11 |
+
- exomind
|
| 12 |
+
- scientific-reasoning
|
| 13 |
+
- scientific-research
|
| 14 |
+
- agentic
|
| 15 |
+
- tool-use
|
| 16 |
+
- multimodal
|
| 17 |
+
- vision-language
|
| 18 |
+
- qwen3.5
|
| 19 |
+
- safetensors
|
| 20 |
---
|
| 21 |
+
|
| 22 |
+
<div align="center">
|
| 23 |
+
|
| 24 |
+
<img src="./assets/ExoMind.png" alt="ExoMind" width="560">
|
| 25 |
+
|
| 26 |
+
# ExoMind
|
| 27 |
+
|
| 28 |
+
**Democratizing Scientific Intelligence via an Extended-Mind-Inspired Agentic System**
|
| 29 |
+
|
| 30 |
+
**ExoMind Team · Shanghai Artificial Intelligence Laboratory**
|
| 31 |
+
|
| 32 |
+
[Project Page](https://ai4sgi.github.io/ExoMind/) ·
|
| 33 |
+
[GitHub](https://github.com/AI4SGI/ExoMind) ·
|
| 34 |
+
[Technical Report](https://github.com/AI4SGI/ExoMind/blob/main/Paper.pdf) ·
|
| 35 |
+
[ModelScope](https://modelscope.cn/models/AI4SGI/ExoMind/summary)
|
| 36 |
+
|
| 37 |
+
</div>
|
| 38 |
+
|
| 39 |
+
## Overview
|
| 40 |
+
|
| 41 |
+
ExoMind is an extended-mind-inspired agentic system for scientific reasoning
|
| 42 |
+
and research. It organizes a language model, typed interaction objects, and
|
| 43 |
+
autonomous interaction processes as one system, allowing scientific
|
| 44 |
+
specialization to scale through both model training and structured interaction
|
| 45 |
+
with external tools.
|
| 46 |
+
|
| 47 |
+
This repository contains the main ExoMind checkpoint, fine-tuned from
|
| 48 |
+
[Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B). The ExoMind
|
| 49 |
+
project combines training-value-aware data engineering, a scientific
|
| 50 |
+
interaction framework, and progressive Chain-of-Interaction training. Using a
|
| 51 |
+
few thousand high-quality trajectories and 1–2 days of full-parameter SFT on
|
| 52 |
+
eight NVIDIA H200 GPUs, the reported system raises its average score across
|
| 53 |
+
eight scientific benchmarks from **36.2 to 67.5** and improves over its base
|
| 54 |
+
model on all six evaluated general-capability benchmarks.
|
| 55 |
+
|
| 56 |
+
<p align="center">
|
| 57 |
+
<a href="https://ai4sgi.github.io/ExoMind/#results">
|
| 58 |
+
<img src="./assets/fig1-benchmark.png" alt="ExoMind scientific intelligence evaluation" width="100%">
|
| 59 |
+
</a>
|
| 60 |
+
</p>
|
| 61 |
+
|
| 62 |
+
> The headline results evaluate the complete ExoMind system under the report's
|
| 63 |
+
> interaction and evaluation setup. The checkpoint alone does not reproduce
|
| 64 |
+
> tool-augmented results unless it is connected to compatible interaction
|
| 65 |
+
> objects and prompts.
|
| 66 |
+
|
| 67 |
+
## Model details
|
| 68 |
+
|
| 69 |
+
| Item | Value |
|
| 70 |
+
| --- | --- |
|
| 71 |
+
| Base model | Qwen3.5-35B-A3B |
|
| 72 |
+
| Architecture | Multimodal causal LM with vision encoder and sparse MoE language model |
|
| 73 |
+
| Parameters | 35B total, 3B activated |
|
| 74 |
+
| Hidden size / layers | 2,048 / 40 |
|
| 75 |
+
| Experts | 256 total; 8 routed experts plus 1 shared expert activated per token |
|
| 76 |
+
| Native context length | 262,144 tokens |
|
| 77 |
+
| Weight format | BF16 safetensors |
|
| 78 |
+
| Default behavior | Thinking mode, with reasoning enclosed in `<think>...</think>` |
|
| 79 |
+
| Primary capabilities | Scientific reasoning, source-grounded research, code-assisted verification, tool use, and image-text input |
|
| 80 |
+
|
| 81 |
+
The architecture and context specifications follow the upstream Qwen3.5 model.
|
| 82 |
+
Runtime support for the full context window depends on the inference framework
|
| 83 |
+
and available memory.
|
| 84 |
+
|
| 85 |
+
## Why ExoMind
|
| 86 |
+
|
| 87 |
+
- **Extended-mind-inspired scientific intelligence:** unifies the LLM,
|
| 88 |
+
interaction objects, and autonomous interaction processes instead of treating
|
| 89 |
+
tool use as an afterthought.
|
| 90 |
+
- **Training-value-aware data engineering:** estimates problem difficulty and
|
| 91 |
+
interaction benefit before trajectory generation, followed by quality
|
| 92 |
+
filtering, difficulty selection, and capability routing.
|
| 93 |
+
- **Deep scientific interaction:** represents source discovery, source
|
| 94 |
+
grounding, executable verification, and observation integration as typed
|
| 95 |
+
objects under a unified action–observation contract.
|
| 96 |
+
- **Progressive Chain-of-Interaction training:** jointly develops intrinsic
|
| 97 |
+
reasoning and basic interaction capability before strengthening interaction
|
| 98 |
+
reasoning with higher-quality trajectories.
|
| 99 |
+
|
| 100 |
+
<p align="center">
|
| 101 |
+
<a href="https://ai4sgi.github.io/ExoMind/#overview">
|
| 102 |
+
<img src="./assets/fig2-paradigm.png" alt="The extended-mind-inspired ExoMind paradigm" width="92%">
|
| 103 |
+
</a>
|
| 104 |
+
</p>
|
| 105 |
+
|
| 106 |
+
## Quick start
|
| 107 |
+
|
| 108 |
+
Qwen3.5 support is evolving quickly. Use current releases or development builds
|
| 109 |
+
of the serving framework and consult the upstream Qwen3.5 instructions when an
|
| 110 |
+
older installation does not recognize the architecture.
|
| 111 |
+
|
| 112 |
+
### Serve with vLLM
|
| 113 |
+
|
| 114 |
+
The following command exposes an OpenAI-compatible endpoint and enables the
|
| 115 |
+
Qwen reasoning and tool-call parsers:
|
| 116 |
+
|
| 117 |
+
```bash
|
| 118 |
+
vllm serve AI4SGI/ExoMind \
|
| 119 |
+
--port 8000 \
|
| 120 |
+
--tensor-parallel-size 8 \
|
| 121 |
+
--max-model-len 262144 \
|
| 122 |
+
--reasoning-parser qwen3 \
|
| 123 |
+
--enable-auto-tool-choice \
|
| 124 |
+
--tool-call-parser qwen3_coder
|
| 125 |
+
```
|
| 126 |
+
|
| 127 |
+
If memory is limited, reduce `--max-model-len`. For text-only deployments,
|
| 128 |
+
vLLM's `--language-model-only` option skips vision-encoder profiling and leaves
|
| 129 |
+
more memory for KV cache.
|
| 130 |
+
|
| 131 |
+
### Call the endpoint
|
| 132 |
+
|
| 133 |
+
```python
|
| 134 |
+
from openai import OpenAI
|
| 135 |
+
|
| 136 |
+
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
| 137 |
+
|
| 138 |
+
response = client.chat.completions.create(
|
| 139 |
+
model="AI4SGI/ExoMind",
|
| 140 |
+
messages=[
|
| 141 |
+
{
|
| 142 |
+
"role": "user",
|
| 143 |
+
"content": "Develop and verify a rigorous solution to this scientific problem: ...",
|
| 144 |
+
}
|
| 145 |
+
],
|
| 146 |
+
temperature=1.0,
|
| 147 |
+
top_p=0.95,
|
| 148 |
+
extra_body={"top_k": 20},
|
| 149 |
+
)
|
| 150 |
+
|
| 151 |
+
print(response.choices[0].message.content)
|
| 152 |
+
```
|
| 153 |
+
|
| 154 |
+
For the complete scientific interaction workflow, use the framework and prompt
|
| 155 |
+
contracts in the [ExoMind GitHub repository](https://github.com/AI4SGI/ExoMind).
|
| 156 |
+
|
| 157 |
+
## Evaluation
|
| 158 |
+
|
| 159 |
+
<p align="center">
|
| 160 |
+
<a href="https://ai4sgi.github.io/ExoMind/#results">
|
| 161 |
+
<img src="./assets/fig8-benchmarks.png" alt="ExoMind performance across eight scientific benchmarks" width="100%">
|
| 162 |
+
</a>
|
| 163 |
+
</p>
|
| 164 |
+
|
| 165 |
+
The [interactive evaluation explorer](https://ai4sgi.github.io/ExoMind/#results)
|
| 166 |
+
provides the complete model comparisons, benchmark scopes, settings, and
|
| 167 |
+
rankings. Parameter counts marked with `~` are IKP-based estimates; see the
|
| 168 |
+
technical report for the methodology and limitations.
|
| 169 |
+
|
| 170 |
+
## Intended use
|
| 171 |
+
|
| 172 |
+
ExoMind is intended for research and development in scientific question
|
| 173 |
+
answering, literature-grounded investigation, mathematical and computational
|
| 174 |
+
reasoning, code-assisted verification, and agentic workflows that expose tools
|
| 175 |
+
through a compatible action–observation interface.
|
| 176 |
+
|
| 177 |
+
It should not be treated as an autonomous authority for medical, legal, safety-
|
| 178 |
+
critical, or other high-stakes decisions. Users should independently verify
|
| 179 |
+
citations, calculations, generated code, and experimental conclusions.
|
| 180 |
+
|
| 181 |
+
## Limitations
|
| 182 |
+
|
| 183 |
+
- The model can produce incorrect reasoning, fabricated references, unsafe
|
| 184 |
+
code, or conclusions unsupported by the retrieved evidence.
|
| 185 |
+
- Reported system-level results depend on prompts, tools, observation formats,
|
| 186 |
+
inference settings, and benchmark-specific evaluators in addition to weights.
|
| 187 |
+
- Very long contexts are memory-intensive and do not guarantee faithful use of
|
| 188 |
+
every supplied token.
|
| 189 |
+
- Tool-call syntax and reasoning extraction can vary across inference
|
| 190 |
+
frameworks and versions.
|
| 191 |
+
- Performance outside the reported tasks, languages, and domains has not been
|
| 192 |
+
established by the published evaluation.
|
| 193 |
+
|
| 194 |
+
## License and attribution
|
| 195 |
+
|
| 196 |
+
The distributed model checkpoint and upstream Qwen3.5 materials are provided
|
| 197 |
+
under the Apache License 2.0 included in this repository. ExoMind software is
|
| 198 |
+
also Apache-2.0 where expressly identified by the project licensing overview.
|
| 199 |
+
The technical report, scientific figures and results, and ExoMind brand assets
|
| 200 |
+
are subject to the separate [ExoMind Research Content and Brand
|
| 201 |
+
Terms](./CONTENT_RIGHTS.md). Third-party materials remain subject to their
|
| 202 |
+
respective terms; see [NOTICE.md](./NOTICE.md).
|
| 203 |
+
|
| 204 |
+
## Citation
|
| 205 |
+
|
| 206 |
+
```bibtex
|
| 207 |
+
@misc{exomind2026,
|
| 208 |
+
title = {ExoMind: Democratizing Scientific Intelligence via Extended-Mind-Inspired Agentic System},
|
| 209 |
+
author = {Peng Ye and Zhuo Liu and Jingqi Ye and Fangchen Yu and Shengji Tang and Yichen Jiang and Haonan He and Zongsheng Cao and Tao Chen and Bo Zhang and Wanli Ouyang and Bowen Zhou and Lei Bai},
|
| 210 |
+
year = {2026},
|
| 211 |
+
note = {Technical report},
|
| 212 |
+
url = {https://github.com/AI4SGI/ExoMind/blob/main/Paper.pdf}
|
| 213 |
+
}
|
| 214 |
+
```
|