enricollen commited on
Commit
11cc2a0
·
verified ·
1 Parent(s): 6bd0ed7

Add new ColBERT model

Browse files
Files changed (1) hide show
  1. README.md +29 -2
README.md CHANGED
@@ -32,7 +32,8 @@ full development history: [github.com/enricollen/it-colbert](https://github.com/
32
  - **What:** ColBERT-style multi-vector retriever, specialized on Italian.
33
  - **Why it exists:** as far as I could find, there was no *Italian-only*
34
  late-interaction retriever — see "Why I built this" below.
35
- - **Size:** ModernBERT-base backbone (~150M parameters), 128-dim token vectors.
 
36
  - **Best at:** short-to-medium Italian passages (search, RAG chunks, FAQ
37
  retrieval). Weakest at long documents unless you use the chunking recipe
38
  below.
@@ -165,7 +166,7 @@ this matters and how much it recovers.
165
  | | |
166
  |---|---|
167
  | Base model | [`nickprock/Italian-ModernBERT-base-embed-mmarco-mnrl`](https://huggingface.co/nickprock/Italian-ModernBERT-base-embed-mmarco-mnrl) |
168
- | Architecture | ModernBERT backbone (~150M params) → dense projection → 128-dim token vectors |
169
  | Similarity | MaxSim (late interaction) |
170
  | Query length | 32 tokens |
171
  | Document length | 512 tokens (see the chunking note above for longer documents) |
@@ -295,6 +296,32 @@ it doesn't beat large multilingual dense embedders on most benchmarks —
295
  matching those was never the goal; they're a different, much larger model
296
  class.
297
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
298
  **Protocol notes that matter for these numbers:**
299
 
300
  - All late-interaction models above are indexed at the same document length
 
32
  - **What:** ColBERT-style multi-vector retriever, specialized on Italian.
33
  - **Why it exists:** as far as I could find, there was no *Italian-only*
34
  late-interaction retriever — see "Why I built this" below.
35
+ - **Size:** ModernBERT-base backbone (~135M parameters), 128-dim token vectors
36
+ the smallest model in the comparison table below, by a wide margin.
37
  - **Best at:** short-to-medium Italian passages (search, RAG chunks, FAQ
38
  retrieval). Weakest at long documents unless you use the chunking recipe
39
  below.
 
166
  | | |
167
  |---|---|
168
  | Base model | [`nickprock/Italian-ModernBERT-base-embed-mmarco-mnrl`](https://huggingface.co/nickprock/Italian-ModernBERT-base-embed-mmarco-mnrl) |
169
+ | Architecture | ModernBERT backbone (~135M params) → dense projection → 128-dim token vectors |
170
  | Similarity | MaxSim (late interaction) |
171
  | Query length | 32 tokens |
172
  | Document length | 512 tokens (see the chunking note above for longer documents) |
 
296
  matching those was never the goal; they're a different, much larger model
297
  class.
298
 
299
+ ### Quality per parameter
300
+
301
+ Worth stating plainly: this is also the smallest model in the whole
302
+ comparison, by a wide margin.
303
+
304
+ | Model | Parameters | MLDR-it (nDCG@10) |
305
+ |---|---|---|
306
+ | **ItColBERT** | **~135M** | **0.4008** (0.4610 chunked) |
307
+ | SauerkrautLM-Multi-ModernColBERT | 149M | 0.3122 |
308
+ | ColBERT-XM | 277M | 0.2734 |
309
+ | mLateOn | 307M | 0.4623 |
310
+ | multilingual-e5-large (dense) | 560M | 0.4310 † |
311
+ | bge-m3 (dense) | 568M | 0.4531 |
312
+ | jina-colbert-v2 | ~0.6B | 0.3858 † |
313
+
314
+ At roughly a quarter to a sixth the size of the ~560M-parameter multilingual
315
+ giants, ItColBERT beats `SauerkrautLM-Multi-ModernColBERT` (the same size
316
+ class) and `ColBERT-XM` (2× the parameters) outright, and statistically ties
317
+ `jina-colbert-v2` (~4.4× the parameters) on the primary out-of-domain
318
+ benchmark. `mLateOn` is the one model that beats it outright while also being
319
+ smaller than the dense giants — included here rather than left out, since
320
+ citing only the flattering comparisons would defeat the point of this
321
+ section. Fewer parameters also means a smaller index and cheaper inference,
322
+ which is part of why training and evaluating this entirely on one consumer
323
+ GPU was practical in the first place.
324
+
325
  **Protocol notes that matter for these numbers:**
326
 
327
  - All late-interaction models above are indexed at the same document length