README.md

# Model Card for CLIBD

In this model repo we provide the official pretrained models used in the paper **CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale.**
The model usage and code can be found in the [github repo](https://github.com/bioscan-ml/clibd).

## Model Details

### Model Description

- **Finetuned from model:**

    -Image: timm model (["vit_base_patch16_224"](https://huggingface.co/timm/vit_base_patch16_224.mae))

    -DNA barcode: BarcodeBERT ["bioscanr/barcodeBERT pre-trained on CANADA-1.5M"](https://huggingface.co/bioscan-ml/bioscan-clibd/tree/main/ckpt/BarcodeBERT/5_mer)

    -Text: Pre-trained BERT model  (["prajjwal1/bert-small"](https://huggingface.co/prajjwal1/bert-small))
### Model Sources

- **Repository:** https://github.com/bioscan-ml/clibd
- **Paper:** https://arxiv.org/abs/2405.17537

### Model Checkpoints

- **ckpt/bioscan_clip/final_experiments/image_dna_4gpu_50epoch/best.pth:** The model trained on the BIOSCAN-1M dataset by aligning images and DNA.
- **ckpt/bioscan_clip/final_experiments/image_dna_text_4gpu_50epoch/best.pth:** The model trained on the BIOSCAN-1M dataset by aligning images, DNA, and taxonomy labels.
- **ckpt/bioscan_clip/new_5M_training/image_dna_4gpu_50epoch/best.pth:** The model trained on the BIOSCAN-5M dataset by aligning images and DNA.
- **ckpt/bioscan_clip/new_5M_training/image_dna_text_4gpu_50epoch/best.pth:** The model trained on the BIOSCAN-5M dataset by aligning images, DNA, and taxonomy labels.

## Training Data

-[BIOSCAN-1M](https://huggingface.co/datasets/bioscan-ml/BIOSCAN-1M).

-[BIOSCAN-5M](https://huggingface.co/datasets/bioscan-ml/BIOSCAN-5M).

You can also find the processed data from [here](https://huggingface.co/datasets/bioscan-ml/bioscan-clibd).

**BibTeX:**
```bibtex
@article{gong2024clibd,
  title={{CLIBD}: Bridging Vision and Genomics for Biodiversity Monitoring at Scale},
  author={Gong, ZeMing and Wang, Austin T. and Huo, Xiaoliang and Haurum, Joakim Bruslund and Lowe, Scott C. and Taylor, Graham W. and Chang, Angel X.},
  journal={arXiv preprint arXiv:2405.17537},
  year={2024},
  eprint={2405.17537},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  doi={10.48550/arxiv.2405.17537},
}
```

## Acknowledgement

We would like to express our gratitude for the use of the INSECT dataset, which played a pivotal role in the completion of our experiments. Additionally, we acknowledge the use and modification of code from the [Fine-Grained-ZSL-with-DNA](https://github.com/sbadirli/Fine-Grained-ZSL-with-DNA) repository, which facilitated part of our experimental work. The contributions of these resources have been invaluable to our project, and we appreciate the efforts of all developers and researchers involved.

This reseach was supported by the Government of Canada’s New Frontiers in Research Fund (NFRF) [NFRFT-2020-00073],
Canada CIFAR AI Chair grants, and the Pioneer Centre for AI (DNRF grant number P1).
This research was also enabled in part by support provided by the Digital Research Alliance of Canada (alliancecan.ca).