Spaces:
Running
Running
Update README.md
Browse files
README.md
CHANGED
|
@@ -29,7 +29,7 @@ Combined with a late interaction matching mechanism, *ColPali* largely outperfor
|
|
| 29 |
|
| 30 |
## Models
|
| 31 |
|
| 32 |
-
- [*ColPali*](https://huggingface.co/vidore/colpali): *ColPali* is our main contribution, it is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs), to efficiently index documents from their visual features.
|
| 33 |
It is a [PaliGemma-3B](https://huggingface.co/google/paligemma-3b-mix-448) extension that generates [ColBERT](https://arxiv.org/abs/2004.12832)- style multi-vector representations of text and images.
|
| 34 |
|
| 35 |
- [*BiPali*](https://huggingface.co/vidore/bipali): It is an extension of original SigLIP architecture, the SigLIP-generated patch embeddings are fed to a text language model, PaliGemma-3B, to obtain LLM contextualized output patch embeddings.
|
|
@@ -37,6 +37,11 @@ Combined with a late interaction matching mechanism, *ColPali* largely outperfor
|
|
| 37 |
|
| 38 |
- [*BiSigLIP*](https://huggingface.co/vidore/bisiglip): Finetuned version of original [SigLIP](https://huggingface.co/google/siglip-so400m-patch14-384), a strong vision-language bi-encoder model.
|
| 39 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
## Datasets
|
| 41 |
|
| 42 |
We organized datasets into collections to constitute our benchmark ViDoRe and its derivates (OCR and Captioning). Below is a brief description of each of them.
|
|
@@ -52,6 +57,12 @@ We organized datasets into collections to constitute our benchmark ViDoRe and it
|
|
| 52 |
- [*Captioning Baseline*](https://huggingface.co/collections/vidore/vidore-captioning-baseline-6658a2a62d857c7a345195fd): Datasets in this collection are the same as in ViDoRe but preprocessed for textual retrieving. The original ViDoRe benchmark was passed to Unstructured to partition each page into chunks. Visual chunks are captioned using Claude Sonnet.
|
| 53 |
|
| 54 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
## Intended use
|
| 56 |
|
| 57 |
You can either load a specific dataset using the standard `load_dataset` function from huggingface.
|
|
|
|
| 29 |
|
| 30 |
## Models
|
| 31 |
|
| 32 |
+
- [*ColPali*](https://huggingface.co/vidore/colpali): *ColPali* is our main model contribution, it is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs), to efficiently index documents from their visual features.
|
| 33 |
It is a [PaliGemma-3B](https://huggingface.co/google/paligemma-3b-mix-448) extension that generates [ColBERT](https://arxiv.org/abs/2004.12832)- style multi-vector representations of text and images.
|
| 34 |
|
| 35 |
- [*BiPali*](https://huggingface.co/vidore/bipali): It is an extension of original SigLIP architecture, the SigLIP-generated patch embeddings are fed to a text language model, PaliGemma-3B, to obtain LLM contextualized output patch embeddings.
|
|
|
|
| 37 |
|
| 38 |
- [*BiSigLIP*](https://huggingface.co/vidore/bisiglip): Finetuned version of original [SigLIP](https://huggingface.co/google/siglip-so400m-patch14-384), a strong vision-language bi-encoder model.
|
| 39 |
|
| 40 |
+
|
| 41 |
+
## Benchmark
|
| 42 |
+
|
| 43 |
+
- [*Leaderboard*](https://huggingface.co/spaces/vidore/vidore-leaderboard): The ViDoRe leaderboard to track model performance on our new Visual Document Retrieval Benchmark, composed of various page-level retrieving tasks spanning multiple domains, languages, and settings.
|
| 44 |
+
|
| 45 |
## Datasets
|
| 46 |
|
| 47 |
We organized datasets into collections to constitute our benchmark ViDoRe and its derivates (OCR and Captioning). Below is a brief description of each of them.
|
|
|
|
| 57 |
- [*Captioning Baseline*](https://huggingface.co/collections/vidore/vidore-captioning-baseline-6658a2a62d857c7a345195fd): Datasets in this collection are the same as in ViDoRe but preprocessed for textual retrieving. The original ViDoRe benchmark was passed to Unstructured to partition each page into chunks. Visual chunks are captioned using Claude Sonnet.
|
| 58 |
|
| 59 |
|
| 60 |
+
## Extra
|
| 61 |
+
|
| 62 |
+
- [*Demo*](https://huggingface.co/spaces/manu/ColPali-demo): A demo to try it out ! This will be improved in the coming days !
|
| 63 |
+
|
| 64 |
+
- *Blogpost*: To be announced
|
| 65 |
+
|
| 66 |
## Intended use
|
| 67 |
|
| 68 |
You can either load a specific dataset using the standard `load_dataset` function from huggingface.
|