BIRCO is a collection of existing Information Retrieval datasets after carefull curation to make it suitable for Large Language Model (LLM) based systems evaluation. Here are the references for each of the 5 datasets used in BIRCO:1. DORIS-MAE: Wang, Jianyou Andre, et al. "Scientific document retrieval using multi-level aspect-based queries." Advances in Neural Information Processing Systems 36 (2024). (https://proceedings.neurips.cc/paper_files/paper/2023/hash/78f9c04bdcb06f1ada3902912d8b64ba-Abstract-Datasets_and_Benchmarks.html)2. ArguAna: Wachsmuth, Henning, Shahbaz Syed, and Benno Stein. "Retrieval of the best counterargument without prior topic knowledge." Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018. (https://aclanthology.org/P18-1023/)3. WhatThatBook: Lin, Kevin, et al. "Decomposing Complex Queries for Tip-of-the-tongue Retrieval." arXiv preprint arXiv:2305.15053 (2023). (https://arxiv.org/abs/2305.15053)4. Clinical-Trial: Koopman, Bevan, and Guido Zuccon. "A test collection for matching patients to clinical trials." Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 2016. (https://dl.acm.org/doi/abs/10.1145/2911451.2914672)5. RELIC: Thai, Katherine, et al. "RELiC: Retrieving Evidence for Literary Claims." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022. (https://aclanthology.org/2022.acl-long.517/)The dataset is stored as a json format. The structure of the file is as follows in python dict:├── ada_embedding_for_datasets_v1.pickle│├── "doris-mae"│ ├── "query" (60 queries)│ │ ├── query_id_1: "query text 1"│ │ ├── query_id_2: "query text 2"│ │ └── query_id_3: "query text 3"│ │ ...│ ├── "corpus" (5543 paper abstracts)│ │ ├── corpus_id_1: "corpus text 1"│ │ ├── corpus_id_2: "corpus text 2"│ │ └── corpus_id_3: "corpus text 3"│ │ ...│ └── "qrel" (avg. candidate pool size: 110.55)│ ├── query_id_1│ │ ├── corpus_id_1: relevance_score (rational number between 0-2)│ │ ├── corpus_id_2: relevance_score│ │ └── corpus_id_3: relevance_score│ │ ...│ ├── query_id_2│ │ ├── corpus_id_1: relevance_score│ │ ├── corpus_id_2: relevance_score│ │ └── corpus_id_3: relevance_score│ │ ...│ └── query_id_3│ ├── corpus_id_1: relevance_score│ ├── corpus_id_2: relevance_score│ └── corpus_id_3: relevance_score│ ...│├── "arguana" │ ├── "query" (100 queries)│ │ ├── query_id_1: "query text 1" │ │ ├── query_id_2: "query text 2"│ │ └── query_id_3: "query text 3"│ │ ...│ ├── "corpus" (3148 arguments)│ │ ├── corpus_id_1: "corpus text 1"│ │ ├── corpus_id_2: "corpus text 2"│ │ └── corpus_id_3: "corpus text 3"│ │ ...│ └── "qrel" (avg. candidate pool size: 50.01)│ ├── query_id_1│ │ ├── corpus_id_1: relevance_score (either 0 or 1)│ │ ├── corpus_id_2: relevance_score│ │ └── corpus_id_3: relevance_score│ │ ...│ ├── query_id_2│ │ ├── corpus_id_1: relevance_score│ │ ├── corpus_id_2: relevance_score│ │ └── corpus_id_3: relevance_score│ │ ...│ └── query_id_3│ ├── corpus_id_1: relevance_score│ ├── corpus_id_2: relevance_score│ └── corpus_id_3: relevance_score│ ...│├── "wtb" │ ├── "query" (100 queries)│ │ ├── query_id_1: "query text 1"│ │ ├── query_id_2: "query text 2"│ │ └── query_id_3: "query text 3"│ │ ...│ ├── "corpus" (1767 book descriptions)│ │ ├── corpus_id_1: "corpus text 1"│ │ ├── corpus_id_2: "corpus text 2"│ │ └── corpus_id_3: "corpus text 3"│ │ ...│ └── "qrel" (avg. candidate pool size: 50.43)│ ├── query_id_1│ │ ├── corpus_id_1: relevance_score (either 0 or 1)│ │ ├── corpus_id_2: relevance_score│ │ └── corpus_id_3: relevance_score│ │ ...│ ├── query_id_2│ │ ├── corpus_id_1: relevance_score│ │ ├── corpus_id_2: relevance_score│ │ └── corpus_id_3: relevance_score│ │ ...│ └── query_id_3│ ├── corpus_id_1: relevance_score│ ├── corpus_id_2: relevance_score│ └── corpus_id_3: relevance_score│ ...│├── "clinical-trial" (avg. candidate pool size )│ ├── "query" (50 queries)│ │ ├── query_id_1: "query text 1"│ │ ├── query_id_2: "query text 2"│ │ └── query_id_3: "query text 3"│ │ ...│ ├── "corpus" (3256 clinical trial descriptions)│ │ ├── corpus_id_1: "corpus text 1"│ │ ├── corpus_id_2: "corpus text 2"│ │ └── corpus_id_3: "corpus text 3"│ │ ...│ └── "qrel" (avg. candidate pool size: 68.40)│ ├── query_id_1│ │ ├── corpus_id_1: relevance_score (0, 1, or 2)│ │ ├── corpus_id_2: relevance_score│ │ └── corpus_id_3: relevance_score│ │ ...│ ├── query_id_2│ │ ├── corpus_id_1: relevance_score│ │ ├── corpus_id_2: relevance_score│ │ └── corpus_id_3: relevance_score│ │ ...│ └── query_id_3│ ├── corpus_id_1: relevance_score│ ├── corpus_id_2: relevance_score│ └── corpus_id_3: relevance_score│ ...│└── "relic" ├── "query" (100 queries) │ ├── query_id_1: "query text 1" │ ├── query_id_2: "query text 2" │ └── query_id_3: "query text 3" │ ... ├── "corpus" (5017 quotations from books) │ ├── corpus_id_1: "corpus text 1" │ ├── corpus_id_2: "corpus text 2" │ └── corpus_id_3: "corpus text 3" │ ... └── "qrel" (avg. candidate pool size: 50.59) ├── query_id_1 │ ├── corpus_id_1: relevance_score (either 0 or 1) │ ├── corpus_id_2: relevance_score │ └── corpus_id_3: relevance_score │ ... ├── query_id_2 │ ├── corpus_id_1: relevance_score │ ├── corpus_id_2: relevance_score │ └── corpus_id_3: relevance_score │ ... └── query_id_3 ├── corpus_id_1: relevance_score ├── corpus_id_2: relevance_score └── corpus_id_3: relevance_score ...