|
|
--- |
|
|
license: mit |
|
|
license_link: https://huggingface.co/microsoft/Florence-2-base-ft/resolve/main/LICENSE |
|
|
pipeline_tag: image-text-to-text |
|
|
tags: |
|
|
- vision |
|
|
- ocr |
|
|
- segmentation |
|
|
--- |
|
|
# VisualHeist - figure, scheme and table segmentation from PDFs (with captions, headers & footnotes) |
|
|
|
|
|
## Model Summary |
|
|
|
|
|
VisualHeist is an object detection model finetuned to extract tables and figures from PDFs. VisualHeist has two versions: |
|
|
- visualheist-base[[HF]](https://huggingface.co/shixuanleong/visualheist-base) (0.23B) |
|
|
- visualheist-large[[HF]](https://huggingface.co/shixuanleong/visualheist-large) (0.77B) |
|
|
|
|
|
**The base model is recommended if you are running it on low-RAM systems** |
|
|
|
|
|
The models are finetuned from [microsoft/Florence-2](https://huggingface.co/microsoft/Florence-2-large-ft) checkpoints. VisualHeist is inspired by and adapted from [yifeihu/TF-ID](https://huggingface.co/yifeihu/TF-ID-large) |
|
|
|
|
|
- The models were finetuned with 3435 figures and 1716 tables from 110 PDF articles across various publishers. All bounding boxes are manually annotated using [CoCo Annotator](https://github.com/jsbroks/coco-annotator). |
|
|
- TF-ID models take an image of a single paper page as the input, and return image files for all figures, schemes and tables in the given page. |
|
|
|
|
|
|
|
|
## Training Code and Dataset |
|
|
- Dataset: [Zenodo repository](https://doi.org/10.5281/zenodo.14917752) |
|
|
- Code: [github.com/aspuru-guzik-group/MERMaid](https://github.com/aspuru-guzik-group/MERMaid) |
|
|
|
|
|
## Benchmarks |
|
|
|
|
|
We manually curated a diverse evaluation dataset consisting of 121 literature articles covering a range of topics, including |
|
|
organic and inorganic chemistry, atmospheric science, batteries, materials science, metal-organic frameworks (MOFs), biology, |
|
|
and science education. These PDFs, published between 1949 and 2025, include both main articles and supplementary materials. |
|
|
|
|
|
We also additionally curated another collection of 98 literature articles (MERMaid-100) reporting novel reaction methodologies that spans |
|
|
three distinct chemical domains: organic electrosynthesis, photocatalysis, and organic synthesis. |
|
|
|
|
|
Additional performance discussion can be found from our [preprint article](XXXXXXX) |
|
|
|
|
|
The full DOI lists can be downloaded from our[Zenodo repository](https://doi.org/10.5281/zenodo.14917752). |
|
|
|
|
|
The evaluation results for visualheist-large are: |
|
|
| | Total Images | F1 score | |
|
|
|---------------------------------------------------------------|--------------|----------------| |
|
|
| All | 1935 | 93% | |
|
|
| Main | 423 | 96% | |
|
|
| pre-2000 | 260 | 93% | |
|
|
| Supplementary Materials | 1252 | 92% | |
|
|
| MERMaid-100 | 100 | 99% | |
|
|
|
|
|
|
|
|
## Running the Model |
|
|
|
|
|
Refer to our [github repository](https://github.com/aspuru-guzik-group/MERMaid) for detailed instructions on how to run the model |
|
|
|
|
|
|
|
|
## BibTex and citation info |
|
|
|
|
|
``` |
|
|
<To be updated with our archive citation> |
|
|
``` |
