Identifying active compounds for target proteins is fundamental in early drug discovery. Recently, data-driven computational methods have demonstrated promising potential in predicting compound activities. However, there lacks a well-designed benchmark to comprehensively evaluate these methods from a practical perspective. To fill this gap, we propose a benchmark, named CARA.Through carefully distinguishing assay types, designing train-test splitting schemes and selecting evaluation metrics, CARA can consider the biased distribution of current real-world compound activity data and avoid overestimation of model performances. We observed that current models can make successful predictions for certain proportions of assays, while the performances varied across different assays. In addition, evaluation of several few-shot training strategies demonstrated different performances related to task types. Overall, we provide a high-quality dataset for developing and evaluating compound activity prediction models, and the analyses in this work may inspire better applications of data-driven models in drug discovery.

Update [v1.0.1] Removed four assays (i.e., CHEMBL1613842_Potency, CHEMBL1614515_Potency, CHEMBL1614171_Potency, CHEMBL1614217_Potency) in VS_All_support.json and VS_All_query.json to keep consistent with our original paper. These four assays may contain small-molecule compounds with different pharmacological profiles such as activators and inhibitors. After correction, there were 96 assays in support, query, and test splits of VS_All task.