Zhan-Lun commited on
Commit
ad14ac3
·
verified ·
1 Parent(s): be43068

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +19 -17
README.md CHANGED
@@ -81,10 +81,12 @@ as synthetically generated data.
81
 
82
  ## 📊 Evaluation results
83
 
84
- The table reports NDCG@10 scores on the eight **public** ViDoRe V3 tasks as
85
- percentages rather than values between 0 and 1 (for example, 0.80 is shown as
86
- 80.00). Each task value is the mean of its six language subsets; the public
87
- average is the unweighted mean of the eight public task values.
 
 
88
 
89
  The evaluation software versions and setup are documented under
90
  [Reproducing the evaluation environment](#reproducing-the-evaluation-environment).
@@ -103,18 +105,18 @@ All table values are shown to two decimal places. ColVec1.1 values are rounded
103
  from the submitted artifacts using round-half-up. Comparator values were read
104
  from the live
105
  [ViDoRe V3 MTEB leaderboard](https://mteb-leaderboard.hf.space/benchmark/ViDoRe%28v3%29)
106
- on July 22, 2026.
107
-
108
- | Model | Computer Science | Energy | FinanceEn | FinanceFr | HR | Industrial | Pharmaceuticals | Physics | **Avg. public** |
109
- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
110
- | **webAI-ColVec1.1-8b (this model)** | 80.08 | 70.12 | **71.90** | **54.87** | 68.55 | **57.65** | 67.88 | 51.50 | **65.32** |
111
- | [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | 57.41 | **68.19** | **51.73** | 64.72 |
112
- | [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 67.32 | 48.38 | 64.45 |
113
- | **[webAI-ColVec1.1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b)** | 80.34 | 69.50 | 69.18 | 53.13 | 66.90 | 56.36 | 67.25 | 51.24 | 64.24 |
114
- | [VultronRetriever Core](https://huggingface.co/vultr/VultronRetrieverCore-Qwen3.5-4.5B) | 79.77 | 69.19 | 68.93 | 52.02 | 66.10 | 56.11 | 67.45 | 50.18 | 63.72 |
115
- | [Nemotron ColEmbed VL 8B V2](https://huggingface.co/nvidia/nemotron-colembed-vl-8b-v2) | 79.29 | 69.82 | 67.29 | 51.54 | 66.32 | 56.03 | 67.19 | 50.84 | 63.54 |
116
- | [webAI-ColVec1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1-4b) | 79.84 | 68.70 | 68.49 | 51.11 | 67.40 | 55.73 | 65.68 | 50.15 | 63.39 |
117
- | [Tomoro ColQwen3 Embed 8B](https://huggingface.co/TomoroAI/tomoro-colqwen3-embed-8b) | 75.35 | 68.41 | 65.08 | 49.10 | 63.98 | 54.41 | 66.36 | 50.13 | 61.60 |
118
 
119
  The current MTEB leaderboard entries named `webAI-ColVec1-4b` and
120
  `webAI-ColVec1-9b` refer to the previous ColVec1 release, not these ColVec1.1
@@ -307,7 +309,7 @@ The requirements include FlashAttention 2 as an optional supported backend;
307
  its presence does not change the SDPA configuration used for the reported
308
  scores.
309
 
310
- The public ViDoRe V3 scores used the following core software versions:
311
 
312
  ```text
313
  Python 3.12
 
81
 
82
  ## 📊 Evaluation results
83
 
84
+ The table reports NDCG@10 scores on the ViDoRe V3 tasks as percentages rather
85
+ than values between 0 and 1 (for example, 0.80 is shown as 80.00). Each task
86
+ value is the mean of its six language subsets. Mean (Public) is the unweighted
87
+ mean of the eight public task values, Mean (Private) covers the Nuclear and
88
+ Telecom tasks, and the final mean is the unweighted mean across all ten tasks.
89
+ Models are sorted by the final mean.
90
 
91
  The evaluation software versions and setup are documented under
92
  [Reproducing the evaluation environment](#reproducing-the-evaluation-environment).
 
105
  from the submitted artifacts using round-half-up. Comparator values were read
106
  from the live
107
  [ViDoRe V3 MTEB leaderboard](https://mteb-leaderboard.hf.space/benchmark/ViDoRe%28v3%29)
108
+ on July 31, 2026.
109
+
110
+ | Model | **Final mean** | **Mean (Public)** | **Mean (Private)** | Computer Science | Energy | FinanceEn | FinanceFr | HR | Industrial | Nuclear | Pharmaceuticals | Physics | Telecom |
111
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
112
+ | **webAI-ColVec1.1-8b (this model)** | **64.95** | **65.32** | **63.47** | 80.08 | 70.12 | **71.90** | **54.87** | 68.55 | **57.65** | 53.66 | 67.88 | 51.50 | **73.29** |
113
+ | [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 64.26 | 64.72 | 62.43 | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | 57.41 | 53.59 | **68.19** | **51.73** | 71.27 |
114
+ | **[webAI-ColVec1.1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b)** | 63.90 | 64.24 | 62.53 | 80.34 | 69.50 | 69.18 | 53.13 | 66.90 | 56.36 | 53.30 | 67.25 | 51.24 | 71.76 |
115
+ | [VultronRetriever Core](https://huggingface.co/vultr/VultronRetrieverCore-Qwen3.5-4.5B) | 63.57 | 63.72 | 63.00 | 79.77 | 69.19 | 68.93 | 52.02 | 66.10 | 56.11 | **54.90** | 67.45 | 50.18 | 71.10 |
116
+ | [Nemotron ColEmbed VL 8B V2](https://huggingface.co/nvidia/nemotron-colembed-vl-8b-v2) | 63.42 | 63.54 | 62.92 | 79.29 | 69.82 | 67.29 | 51.54 | 66.32 | 56.03 | 53.84 | 67.19 | 50.84 | 72.00 |
117
+ | [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | 63.00 | 64.45 | 57.20 | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 47.66 | 67.32 | 48.38 | 66.74 |
118
+ | [webAI-ColVec1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1-4b) | 62.22 | 63.39 | 57.55 | 79.84 | 68.70 | 68.49 | 51.11 | 67.40 | 55.73 | 48.22 | 65.68 | 50.15 | 66.88 |
119
+ | [Tomoro ColQwen3 Embed 8B](https://huggingface.co/TomoroAI/tomoro-colqwen3-embed-8b) | 61.59 | 61.60 | 61.56 | 75.35 | 68.41 | 65.08 | 49.10 | 63.98 | 54.41 | 52.65 | 66.36 | 50.13 | 70.46 |
120
 
121
  The current MTEB leaderboard entries named `webAI-ColVec1-4b` and
122
  `webAI-ColVec1-9b` refer to the previous ColVec1 release, not these ColVec1.1
 
309
  its presence does not change the SDPA configuration used for the reported
310
  scores.
311
 
312
+ The ViDoRe V3 scores used the following core software versions:
313
 
314
  ```text
315
  Python 3.12