LightningCreeper commited on
Commit
355d0f3
·
verified ·
1 Parent(s): ba94d59

Update WIA model card tables and evaluation details

Browse files
Files changed (1) hide show
  1. README.md +15 -13
README.md CHANGED
@@ -20,23 +20,25 @@ The model is initialized from a Qwen3.5 27B backbone and trained with WIA-CPT, o
20
 
21
  Results below are from the WIA technical report. Scores are on a 0--100 scale, evaluated by `DeepSeek-V4-Pro`; `Avg` is sample-weighted and `MAvg` is the macro average over seven domains.
22
 
23
- ### WIA-Bench
24
 
25
- | Split | Android | MCP | OS | Search | SWE | Terminal | Web | Avg | MAvg |
26
  |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
27
- | OOD | 50.51 | 52.53 | 63.42 | 46.36 | 59.92 | 60.73 | 54.52 | **55.45** | 55.43 |
28
- | ID | 43.21 | 67.74 | 35.28 | 88.54 | 66.24 | 80.85 | 73.84 | **66.41** | 65.10 |
 
 
 
29
 
30
- ### Stage ablation
31
 
32
- | Stage | OOD Avg | ID Avg |
33
- |---|---:|---:|
34
- | Qwen3.5-27B | 49.15 | 48.91 |
35
- | WIA-CPT-27B | 44.19 | 43.89 |
36
- | WIA-SFT-27B | 46.36 | 61.96 |
37
- | **WIA-WM-27B** | **55.45** | **66.41** |
38
-
39
- The 27B run is a useful reminder that intermediate stages need not improve every aggregate: CPT and SFT are below the base OOD average, while world-model RL supplies the final OOD gain of 9.09 points from WIA-SFT-27B. The final model has the highest reported ID weighted average in the WIA-WM family.
40
 
41
  ## Download
42
 
 
20
 
21
  Results below are from the WIA technical report. Scores are on a 0--100 scale, evaluated by `DeepSeek-V4-Pro`; `Avg` is sample-weighted and `MAvg` is the macro average over seven domains.
22
 
23
+ ### WIA-Bench OOD
24
 
25
+ | Model | Android | MCP | OS | Search | SWE | Terminal | Web | Avg | MAvg |
26
  |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
27
+ | WIA-WM-4B | 27.19 | 44.10 | 48.40 | 36.84 | 60.06 | 55.87 | 49.27 | 46.34 | 45.96 |
28
+ | WIA-WM-9B | 46.82 | 52.12 | 62.70 | 46.06 | 60.77 | 59.38 | 53.43 | 54.49 | 54.47 |
29
+ | WIA-WM-27B | 50.51 | 52.53 | 63.42 | 46.36 | 59.92 | 60.73 | 54.52 | 55.45 | 55.43 |
30
+ | WIA-WM-35B-A3B | 48.76 | 43.72 | 63.86 | 51.33 | 59.19 | 62.52 | 54.95 | 55.72 | 54.90 |
31
+ | WIA-WM-122B-A10B | 50.59 | 40.00 | 64.71 | 58.18 | 60.46 | 64.62 | 53.59 | 57.35 | 56.02 |
32
 
33
+ ### WIA-Bench ID
34
 
35
+ | Model | Android | MCP | OS | Search | SWE | Terminal | Web | Avg | MAvg |
36
+ |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
37
+ | WIA-WM-4B | 27.12 | 65.26 | 30.76 | 82.59 | 57.52 | 75.76 | 63.24 | 59.17 | 57.46 |
38
+ | WIA-WM-9B | 41.61 | 63.89 | 32.79 | 84.31 | 60.87 | 73.96 | 68.69 | 61.99 | 60.87 |
39
+ | WIA-WM-27B | 43.21 | 67.74 | 35.28 | 88.54 | 66.24 | 80.85 | 73.84 | 66.41 | 65.10 |
40
+ | WIA-WM-35B-A3B | 43.49 | 67.45 | 34.39 | 89.30 | 64.18 | 76.36 | 71.86 | 65.04 | 63.86 |
41
+ | WIA-WM-122B-A10B | 41.44 | 64.33 | 33.01 | 86.66 | 57.74 | 71.43 | 66.21 | 61.16 | 60.12 |
 
42
 
43
  ## Download
44