manu commited on
Commit
0a0f21a
·
verified ·
1 Parent(s): 1425cb6

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -1
README.md CHANGED
@@ -29,7 +29,7 @@ Combined with a late interaction matching mechanism, *ColPali* largely outperfor
29
 
30
  ## Models
31
 
32
- - [*ColPali*](https://huggingface.co/vidore/colpali): *ColPali* is our main contribution, it is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs), to efficiently index documents from their visual features.
33
  It is a [PaliGemma-3B](https://huggingface.co/google/paligemma-3b-mix-448) extension that generates [ColBERT](https://arxiv.org/abs/2004.12832)- style multi-vector representations of text and images.
34
 
35
  - [*BiPali*](https://huggingface.co/vidore/bipali): It is an extension of original SigLIP architecture, the SigLIP-generated patch embeddings are fed to a text language model, PaliGemma-3B, to obtain LLM contextualized output patch embeddings.
@@ -37,6 +37,11 @@ Combined with a late interaction matching mechanism, *ColPali* largely outperfor
37
 
38
  - [*BiSigLIP*](https://huggingface.co/vidore/bisiglip): Finetuned version of original [SigLIP](https://huggingface.co/google/siglip-so400m-patch14-384), a strong vision-language bi-encoder model.
39
 
 
 
 
 
 
40
  ## Datasets
41
 
42
  We organized datasets into collections to constitute our benchmark ViDoRe and its derivates (OCR and Captioning). Below is a brief description of each of them.
@@ -52,6 +57,12 @@ We organized datasets into collections to constitute our benchmark ViDoRe and it
52
  - [*Captioning Baseline*](https://huggingface.co/collections/vidore/vidore-captioning-baseline-6658a2a62d857c7a345195fd): Datasets in this collection are the same as in ViDoRe but preprocessed for textual retrieving. The original ViDoRe benchmark was passed to Unstructured to partition each page into chunks. Visual chunks are captioned using Claude Sonnet.
53
 
54
 
 
 
 
 
 
 
55
  ## Intended use
56
 
57
  You can either load a specific dataset using the standard `load_dataset` function from huggingface.
 
29
 
30
  ## Models
31
 
32
+ - [*ColPali*](https://huggingface.co/vidore/colpali): *ColPali* is our main model contribution, it is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs), to efficiently index documents from their visual features.
33
  It is a [PaliGemma-3B](https://huggingface.co/google/paligemma-3b-mix-448) extension that generates [ColBERT](https://arxiv.org/abs/2004.12832)- style multi-vector representations of text and images.
34
 
35
  - [*BiPali*](https://huggingface.co/vidore/bipali): It is an extension of original SigLIP architecture, the SigLIP-generated patch embeddings are fed to a text language model, PaliGemma-3B, to obtain LLM contextualized output patch embeddings.
 
37
 
38
  - [*BiSigLIP*](https://huggingface.co/vidore/bisiglip): Finetuned version of original [SigLIP](https://huggingface.co/google/siglip-so400m-patch14-384), a strong vision-language bi-encoder model.
39
 
40
+
41
+ ## Benchmark
42
+
43
+ - [*Leaderboard*](https://huggingface.co/spaces/vidore/vidore-leaderboard): The ViDoRe leaderboard to track model performance on our new Visual Document Retrieval Benchmark, composed of various page-level retrieving tasks spanning multiple domains, languages, and settings.
44
+
45
  ## Datasets
46
 
47
  We organized datasets into collections to constitute our benchmark ViDoRe and its derivates (OCR and Captioning). Below is a brief description of each of them.
 
57
  - [*Captioning Baseline*](https://huggingface.co/collections/vidore/vidore-captioning-baseline-6658a2a62d857c7a345195fd): Datasets in this collection are the same as in ViDoRe but preprocessed for textual retrieving. The original ViDoRe benchmark was passed to Unstructured to partition each page into chunks. Visual chunks are captioned using Claude Sonnet.
58
 
59
 
60
+ ## Extra
61
+
62
+ - [*Demo*](https://huggingface.co/spaces/manu/ColPali-demo): A demo to try it out ! This will be improved in the coming days !
63
+
64
+ - *Blogpost*: To be announced
65
+
66
  ## Intended use
67
 
68
  You can either load a specific dataset using the standard `load_dataset` function from huggingface.