Images of the natural world are an abundant source of biological information. There are many computational methods and tools, particularly computer vision, for extracting information from images. However, existing methods consist of bespoke models, not adaptable or extendable from their targeted task to new questions, contexts, and datasets. We thus develop the first large-scale multimodal model, BioCLIP, for general biology questions on images. We leverage the unique properties of biology (abundance and variety of images and availability of rich structured biological knowledge) as the the application domain for computer vision.

If you use this software, please cite both the article from preferred-citation and the software itself. Article Citation: Stevens, S., Wu, J., Thompson, M. J., Campolongo, E. G., Song, C. H., Carlyn, D. E., Dong, L., Dahdul, W. M., Stewart, C., Berger-Wolf, T., Chao, W., & Su, Y. (2024). BioCLIP: A Vision Foundation Model for the Tree of Life [Conference paper]. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19412-19424.