README.md

metadata

description: >-
  Pretrained weights for CLIBD, a multimodal model bridging vision and genomics
  for biodiversity monitoring.

Model Card for CLIBD

Official pretrained models for CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale. Model usage and code: github.com/bioscan-ml/clibd.

Model Details

Model Description

• Finetuned from model:

  -Image: timm model ("vit_base_patch16_224")

  -DNA barcode: BarcodeBERT "bioscanr/barcodeBERT pre-trained on CANADA-1.5M"

  -Text: Pre-trained BERT model ("prajjwal1/bert-small")

Model Sources

• Repository: https://github.com/bioscan-ml/clibd
• Paper: https://arxiv.org/abs/2405.17537

Model Checkpoints

• ckpt/bioscan_clip/final_experiments/image_dna_4gpu_50epoch/best.pth: The model trained on the BIOSCAN-1M dataset by aligning images and DNA.
• ckpt/bioscan_clip/final_experiments/image_dna_text_4gpu_50epoch/best.pth: The model trained on the BIOSCAN-1M dataset by aligning images, DNA, and taxonomy labels.
• ckpt/bioscan_clip/new_5M_training/image_dna_4gpu_50epoch/best.pth: The model trained on the BIOSCAN-5M dataset by aligning images and DNA.
• ckpt/bioscan_clip/new_5M_training/image_dna_text_4gpu_50epoch/best.pth: The model trained on the BIOSCAN-5M dataset by aligning images, DNA, and taxonomy labels.

Training Data

-BIOSCAN-1M.

-BIOSCAN-5M.

You can also find the processed data from here.

BibTeX:

@article{gong2024clibd,
  title={{CLIBD}: Bridging Vision and Genomics for Biodiversity Monitoring at Scale},
  author={Gong, ZeMing and Wang, Austin T. and Huo, Xiaoliang and Haurum, Joakim Bruslund and Lowe, Scott C. and Taylor, Graham W. and Chang, Angel X.},
  journal={arXiv preprint arXiv:2405.17537},
  year={2024},
  eprint={2405.17537},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  doi={10.48550/arxiv.2405.17537},
}

Acknowledgement

We would like to express our gratitude for the use of the INSECT dataset, which played a pivotal role in the completion of our experiments. Additionally, we acknowledge the use and modification of code from the Fine-Grained-ZSL-with-DNA repository, which facilitated part of our experimental work. The contributions of these resources have been invaluable to our project, and we appreciate the efforts of all developers and researchers involved.

This research was supported by the Government of Canada’s New Frontiers in Research Fund (NFRF) [NFRFT-2020-00073], Canada CIFAR AI Chair grants, and the Pioneer Centre for AI (DNRF grant number P1). This research was also enabled in part by support provided by the Digital Research Alliance of Canada (alliancecan.ca).