WIA-9B / README.md
LightningCreeper's picture
Set model generation example to 16384 tokens
1186b90 verified
|
Raw History Blame Contribute Delete
8.19 kB
---
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- world-model
- digital-agents
- qwen3.5
- wia
---
# WIA-9B
**WIA-9B** is a 9B-parameter text world model from the World Intelligence Agent (WIA) project. It predicts action-conditioned observations across Android, MCP, desktop OS, Search, SWE, Terminal, and Web. It is intended for world-model evaluation, synthetic agent training, and test-time action lookahead; it is not a general-purpose chat assistant.
The model is initialized from a Qwen3.5 9B backbone and trained with WIA-CPT, observation-only WIA-SFT, and GSPO-based world-model RL (WIA-WM).
## Evaluation
Results below are from the WIA technical report. Scores are on a 0--100 scale, evaluated by `DeepSeek-V4-Pro`; `Avg` is sample-weighted and `MAvg` is the macro average over seven domains.
### WIA-Bench OOD
The complete OOD comparison from the technical report is reproduced below. Scores use the `DeepSeek-V4-Pro` judge; `Avg` is sample-weighted and `MAvg` is the macro average over seven domains.
| Group | Model | Android | MCP | OS | Search | SWE | Terminal | Web | Avg | MAvg |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Proprietary | GPT-5.6-luna | 48.37 | 44.51 | 54.96 | 40.01 | 55.88 | 62.42 | 53.87 | 51.91 | 51.43 |
| Proprietary | GPT-5.6-terra | 48.34 | 43.55 | 56.14 | 40.46 | 57.68 | 61.79 | 54.57 | 52.38 | 51.79 |
| Proprietary | GPT-5.6-sol | 49.39 | 45.75 | 59.70 | 39.99 | 58.61 | 63.63 | 55.00 | 53.58 | 53.15 |
| Proprietary | Claude-Sonnet-5 | 47.76 | 49.85 | 63.22 | 35.26 | 44.58 | 46.65 | 54.24 | 48.03 | 48.79 |
| Proprietary | Claude-Opus-4.8 | 48.98 | 53.97 | 63.02 | 39.10 | 58.15 | 57.93 | 55.26 | 53.40 | 53.77 |
| Open-source | GLM-5.3 | 47.20 | 50.23 | 55.58 | 31.10 | 32.68 | 53.88 | 51.03 | 44.98 | 45.96 |
| Open-source | MiMo-v2.5-pro | 45.80 | 46.18 | 59.78 | 38.20 | 49.73 | 53.43 | 51.50 | 49.09 | 49.23 |
| Open-source | Qwen3.8-27B | 41.08 | 39.40 | 53.34 | 34.88 | 48.47 | 51.84 | 47.51 | 45.46 | 45.22 |
| Open-source | Qwen3.8-max | 47.60 | 45.32 | 57.97 | 36.82 | 57.62 | 63.42 | 55.48 | 52.40 | 52.03 |
| Open-source | DeepSeek-V4-Flash | 46.74 | 44.46 | 58.93 | 38.40 | 50.51 | 55.53 | 50.77 | 49.40 | 49.33 |
| Open-source | DeepSeek-V4-Pro | 47.87 | 43.92 | 61.93 | 38.77 | 51.14 | 57.77 | 52.63 | 50.73 | 50.57 |
| Qwen3.5 | Qwen3.5-4B | 39.92 | 38.38 | 55.29 | 30.44 | 37.26 | 39.50 | 45.24 | 40.46 | 40.86 |
| Qwen3.5 | Qwen3.5-9B | 43.00 | 41.37 | 58.19 | 30.99 | 44.05 | 43.81 | 47.42 | 43.76 | 44.12 |
| Qwen3.5 | Qwen3.5-27B | 47.84 | 42.56 | 64.29 | 33.75 | 48.10 | 55.09 | 52.78 | 49.15 | 49.20 |
| Qwen3.5 | Qwen3.5-35B-A3B | 47.09 | 42.69 | 56.59 | 32.03 | 45.70 | 49.80 | 49.55 | 46.01 | 46.21 |
| Qwen3.5 | Qwen3.5-122B-A10B* | 43.92 | 32.43 | 41.86 | 34.93 | 48.65 | 55.70 | 44.15 | 44.07 | 43.09 |
| Qwen3.5 | Qwen-AgentWorld | 47.43 | 42.63 | 64.21 | 45.02 | 56.31 | 57.22 | 53.68 | 52.87 | 52.36 |
| WIA-WM | WIA-WM-4B | 27.19 | 44.10 | 48.40 | 36.84 | 60.06 | 55.87 | 49.27 | 46.34 | 45.96 |
| WIA-WM | WIA-WM-9B | 46.82 | 52.12 | 62.70 | 46.06 | 60.77 | 59.38 | 53.43 | 54.49 | 54.47 |
| WIA-WM | WIA-WM-27B | 50.51 | 52.53 | 63.42 | 46.36 | 59.92 | 60.73 | 54.52 | 55.45 | 55.43 |
| WIA-WM | WIA-WM-35B-A3B | 48.76 | 43.72 | 63.86 | 51.33 | 59.19 | 62.52 | 54.95 | 55.72 | 54.90 |
| WIA-WM | WIA-WM-122B-A10B* | 50.59 | 40.00 | 64.71 | 58.18 | 60.46 | 64.62 | 53.59 | 57.35 | 56.02 |
### WIA-Bench ID
The complete ID comparison from the technical report is reproduced below.
| Group | Model | Android | MCP | OS | Search | SWE | Terminal | Web | Avg | MAvg |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Proprietary | GPT-5.6-sol | 43.62 | 61.02 | 33.08 | 50.66 | 53.11 | 59.96 | 62.38 | 52.04 | 51.98 |
| Proprietary | GPT-5.6-terra | 41.57 | 61.98 | 33.08 | 47.32 | 52.81 | 59.25 | 60.94 | 51.05 | 50.99 |
| Proprietary | GPT-5.6-luna | 41.29 | 60.03 | 32.43 | 45.40 | 51.29 | 55.28 | 58.33 | 49.11 | 49.15 |
| Proprietary | Claude-Opus-4.8 | 41.59 | 59.08 | 33.00 | 63.50 | 56.46 | 65.91 | 66.21 | 55.76 | 55.11 |
| Proprietary | Claude-Sonnet-5 | 40.08 | 60.28 | 32.27 | 58.46 | 38.72 | 60.40 | 63.96 | 50.75 | 50.59 |
| Open-source | GLM-5.3 | 41.20 | 58.63 | 33.20 | 54.74 | 32.09 | 63.22 | 64.52 | 49.64 | 49.66 |
| Open-source | MiMo-v2.5-pro | 40.93 | 59.61 | 32.39 | 51.34 | 47.36 | 54.35 | 64.16 | 50.12 | 50.02 |
| Open-source | Qwen3.8-27B | 36.69 | 55.67 | 30.75 | 50.87 | 43.50 | 52.67 | 61.15 | 47.59 | 47.33 |
| Open-source | Qwen3.8-max | 41.18 | 58.32 | 32.39 | 61.00 | 51.06 | 65.09 | 65.82 | 54.06 | 53.55 |
| Open-source | DeepSeek-V4-Flash | 40.64 | 59.23 | 32.39 | 56.41 | 44.87 | 56.97 | 64.40 | 50.90 | 50.70 |
| Open-source | DeepSeek-V4-Pro | 41.54 | 59.06 | 32.31 | 55.72 | 45.65 | 55.51 | 64.51 | 50.76 | 50.61 |
| Qwen3.5 | Qwen3.5-4B | 35.61 | 54.57 | 30.96 | 45.63 | 33.14 | 40.22 | 56.33 | 42.20 | 42.35 |
| Qwen3.5 | Qwen3.5-9B | 36.80 | 55.09 | 31.81 | 38.29 | 37.16 | 38.97 | 60.23 | 42.39 | 42.62 |
| Qwen3.5 | Qwen3.5-27B | 41.24 | 57.26 | 32.09 | 45.92 | 43.77 | 57.37 | 64.72 | 48.91 | 48.91 |
| Qwen3.5 | Qwen3.5-35B-A3B | 37.50 | 56.75 | 32.21 | 39.15 | 41.76 | 50.03 | 61.36 | 45.47 | 45.54 |
| Qwen3.5 | Qwen-AgentWorld | 39.03 | 56.99 | 32.49 | 63.68 | 45.60 | 61.33 | 64.54 | 52.51 | 51.95 |
| WIA-WM | WIA-WM-4B | 27.12 | 65.26 | 30.76 | 82.59 | 57.52 | 75.76 | 63.24 | 59.17 | 57.46 |
| WIA-WM | WIA-WM-9B | 41.61 | 63.89 | 32.79 | 84.31 | 60.87 | 73.96 | 68.69 | 61.99 | 60.87 |
| WIA-WM | WIA-WM-27B | 43.21 | 67.74 | 35.28 | 88.54 | 66.24 | 80.85 | 73.84 | 66.41 | 65.10 |
| WIA-WM | WIA-WM-35B-A3B | 43.49 | 67.45 | 34.39 | 89.30 | 64.18 | 76.36 | 71.86 | 65.04 | 63.86 |
| WIA-WM | WIA-WM-122B-A10B* | 41.44 | 64.33 | 33.01 | 86.66 | 57.74 | 71.43 | 66.21 | 61.16 | 60.12 |
*The 122B-A10B configuration differs from the other WIA-WM scales; it is RL-only and uses a different RL data and reward setup.
### Stage ablation
OOD-only ablation under the `DeepSeek-V4-Pro` judge:
| Stage | Android | MCP | OS | Search | SWE | Terminal | Web | Avg |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| Qwen3.5-9B-Base | 37.12 | 36.88 | 53.45 | 27.27 | 39.56 | 39.08 | 40.80 | 38.80 |
| WIA-CPT-9B | 42.94 | 34.23 | 50.69 | 34.17 | 35.91 | 38.54 | 47.38 | 40.62 |
| WIA-SFT-9B | 39.45 | 43.69 | 57.51 | 37.82 | 50.59 | 51.96 | 49.45 | 47.23 |
| **WIA-WM-9B** | **46.82** | **52.12** | **62.70** | **46.06** | **60.77** | **59.38** | **53.43** | **54.49** |
World-model RL adds 7.26 OOD points and 4.44 ID points over WIA-SFT-9B. The report identifies the largest improvements in Search and SWE, with the 9B model already exceeding the strongest reported OOD baselines on weighted average.
## Download
```bash
pip install -U "transformers>=5.8.0" huggingface_hub torch
hf download darwinagent/WIA-9B --local-dir ./WIA-9B
```
## Usage
WIA-9B predicts an observation after an action. Keep the WIA prompt format and append `###[Observation]` before generation.
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "darwinagent/WIA-9B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
messages = [
{"role": "system", "content": "Operate as a language world model and report only the resulting observation."},
{"role": "user", "content": "Task: inspect the repository.\nCurrent observation: the shell is open.\nAction: shell_exec(command='pwd', cwd='/workspace', timeout_sec=30)\n###[Observation]"},
]
inputs = tokenizer.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
with torch.inference_mode():
output = model.generate(inputs, max_new_tokens=16384, do_sample=False)
print(tokenizer.decode(output[0, inputs.shape[-1]:], skip_special_tokens=True))
```
Use a GPU configuration appropriate for the model and context length. Practical memory use depends on dtype, context length, batching, and attention implementation.
## Citation
Please cite the WIA technical report when using this model.