Hugging Face's logo

OraRL
/
Video-ORA-9B

Image-Text-to-Text
Transformers
Safetensors
qwen3_5
vllm
video
multimodal
reinforcement-learning
temporal-grounding
object-tracking
video-segmentation
visual-question-answering
spatial-reasoning
qwen3.5
conversational
Model card Files
xet
 Community
lyhisme commited on
Commit
25ed8ba
·
verified ·
1 Parent(s): 15c04f3

Use detailed benchmark comparison

Browse files
Files changed (1) hide show
  1. README.md +23 -113
README.md CHANGED
@@ -55,119 +55,29 @@ chain-of-thought.
55
 
56
  ## Results
57
 
58
- All values are percentages. Results use direct answer generation without
59
- chain-of-thought and the exact preprocessing and aggregation protocols reported
60
- in the paper.
61
-
62
- ### Temporal grounding
63
-
64
- | TimeLens split | R1@0.3 | R1@0.5 | R1@0.7 | mIoU |
65
- | --- | ---: | ---: | ---: | ---: |
66
- | Charades-STA | **84.5** | **71.6** | **42.4** | **61.8** |
67
- | ActivityNet Captions | **77.4** | **69.5** | **53.3** | **63.6** |
68
- | QVHighlights | **84.9** | **77.6** | **64.0** | **72.5** |
69
-
70
- ### Spatial grounding
71
-
72
- | Benchmark split | R@0.5 |
73
- | --- | ---: |
74
- | RefCOCO testA | **94.6** |
75
- | RefCOCO testB | **90.6** |
76
- | RefCOCO val | **93.4** |
77
- | RefCOCO+ testA | **92.7** |
78
- | RefCOCO+ testB | **84.4** |
79
- | RefCOCO+ val | **89.7** |
80
- | RefCOCOg test | **91.0** |
81
- | RefCOCOg val | **90.8** |
82
-
83
- ### Segmentation
84
-
85
- | Benchmark | cIoU | J | F | J&F |
86
- | --- | ---: | ---: | ---: | ---: |
87
- | RefCOCO val | **79.4** | — | — | — |
88
- | RefCOCO+ val | **72.6** | — | — | — |
89
- | RefCOCOg val | **75.1** | — | — | — |
90
- | MeViS | — | **57.7** | **64.8** | **61.3** |
91
- | ReasonVOS | — | **59.8** | **67.6** | **63.7** |
92
-
93
- ### Visual tracking
94
-
95
- | Benchmark | Frames | AO | R@0.3 | R@0.5 | R@0.7 |
96
- | --- | ---: | ---: | ---: | ---: | ---: |
97
- | GOT-10k | 32 | **78.2** | **94.1** | **87.8** | **75.3** |
98
-
99
- ### Spatial-temporal grounding
100
-
101
- | Benchmark | Frames | tIoU@0.5 | mean tIoU | sIoU@0.5 | mean sIoU |
102
- | --- | ---: | ---: | ---: | ---: | ---: |
103
- | STVG | 128 | **40.0** | **41.7** | **27.4** | **28.4** |
104
-
105
- ### Video question answering
106
-
107
- | Benchmark | Accuracy |
108
- | --- | ---: |
109
- | VideoMME | **76.7** |
110
- | VideoMME-v2 | **32.0** |
111
- | MV-Bench | **72.5** |
112
- | MMVU (multiple choice) | **77.0** |
113
- | VideoHolmes | **65.5** |
114
- | LongVideoBench val | **66.2** |
115
- | MLVU-MC dev | **77.7** |
116
- | Macro average | **66.8** |
117
-
118
- ### Spatial intelligence
119
-
120
- VSI-Bench numerical tasks use mean relative accuracy (MRA); its
121
- multiple-choice tasks and the remaining benchmarks use accuracy.
122
-
123
- | VSI-Bench subtask | Score |
124
- | --- | ---: |
125
- | Object count | **76.1** |
126
- | Absolute distance | **58.3** |
127
- | Object size | **78.2** |
128
- | Room size | **72.1** |
129
- | Relative distance | **75.2** |
130
- | Relative direction | **86.9** |
131
- | Route planning | **47.4** |
132
- | Appearance order | **90.6** |
133
- | Average | **73.1** |
134
-
135
- | MMSI-Bench subtask | Accuracy |
136
- | --- | ---: |
137
- | Camera-camera relation | **41.9** |
138
- | Object-object relation | **30.9** |
139
- | Region-region relation | **30.9** |
140
- | Camera-object relation | **57.0** |
141
- | Object-region relation | **41.2** |
142
- | Camera-region relation | **50.6** |
143
- | Measurement attribute | **39.1** |
144
- | Appearance attribute | **21.2** |
145
- | Camera motion | **36.5** |
146
- | Object motion | **23.7** |
147
- | Multi-step reasoning | **38.4** |
148
- | Average | **37.9** |
149
-
150
- | MindCube-Tiny subtask | Accuracy |
151
- | --- | ---: |
152
- | Rotation | **44.0** |
153
- | Among | **57.2** |
154
- | Around | **68.8** |
155
- | Average | **57.4** |
156
-
157
- | ReVSI subtask | 64 frames | 128 frames |
158
- | --- | ---: | ---: |
159
- | Object count | **46.4** | **48.8** |
160
- | Absolute distance | **65.3** | **69.1** |
161
- | Object size | **70.4** | **72.7** |
162
- | Room size | **50.2** | **51.1** |
163
- | Relative distance | **62.3** | **66.3** |
164
- | Relative direction | **47.4** | **49.9** |
165
- | Route planning | **48.8** | **49.8** |
166
- | Average | **55.8** | **58.2** |
167
-
168
- See the [project page](https://orarl.github.io/) and
169
- [paper](https://arxiv.org/abs/2608.20492) for complete benchmark protocols,
170
- baseline comparisons, and aggregation details.
171
 
172
  ## Quick start
173
 
 
55
 
56
  ## Results
57
 
58
+ ### Dataset-level comparison
59
+
60
+ <a href="https://github.com/HVision-NKU/OraRL#results">
61
+ <picture>
62
+ <source media="(prefers-color-scheme: dark)"
63
+ srcset="assets/video_ora_benchmark_matrix_dark.svg">
64
+ <source media="(prefers-color-scheme: light)"
65
+ srcset="assets/video_ora_benchmark_matrix_light.svg">
66
+ <img src="assets/video_ora_benchmark_matrix_light.svg"
67
+ alt="Dataset-level benchmark matrix comparing Video-ORA with multimodal baselines"
68
+ width="100%">
69
+ </picture>
70
+ </a>
71
+
72
+ Video-ORA-9B leads the matched seven-family comparison without chain-of-thought
73
+ decoding. Best and second-best values are highlighted per row; `†` denotes an
74
+ original-report value whose frame, prompt, split, or decoding settings may
75
+ differ. Averages require complete family coverage.
76
+
77
+ See the [OraRL repository](https://github.com/HVision-NKU/OraRL),
78
+ [project page](https://orarl.github.io/), and
79
+ [paper](https://arxiv.org/abs/2608.20492) for complete benchmark protocols and
80
+ source attribution.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
 
82
  ## Quick start
83