Objectives: A rapidly developing scenario like a pandemic requires the prompt production of high-quality systematic reviews, which can be automated using artificial intelligence (AI) techniques. We evaluated the application of AI tools in COVID-19 evidence syntheses. Study design: After prospective registration of the review protocol, we automated the download of all open-access COVID-19 systematic reviews in the COVID-19 Living Overview of Evidence database, indexed them for AI-related keywords, and located those that used AI tools. We compared their journals’ JCR Impact Factor, citations per month, screening workloads, completion times (from pre-registration to preprint or submission to a journal) and AMSTAR-2 methodology assessments (maximum score 13 points) with a set of publication date matched control reviews without AI. Results: Of the 3999 COVID-19 reviews, 28 (0.7%, 95% CI 0.47-1.03%) made use of AI. On average, compared to controls (n=64), AI reviews were published in journals with higher Impact Factors (median 8.9 vs 3.5, P<0.001), and screened more abstracts per author (302.2 vs 140.3, P=0.009) and per included study (189.0 vs 365.8, P<0.001) while inspecting less full texts per author (5.3 vs 14.0, P=0.005). No differences were found in citation counts (0.5 vs 0.6, P=0.600), inspected full texts per included study (3.8 vs 3.4, P=0.481), completion times (74.0 vs 123.0, P=0.205) or AMSTAR-2 (7.5 vs 6.3, P=0.119). Conclusion: AI was an underutilized tool in COVID-19 systematic reviews. Its usage, compared to reviews without AI, was associated with more efficient screening of literature and higher publication impact. There is scope for the application of AI in automating systematic reviews.

Please read README.txt for usage notes. Shasums for all included files (and downloaded reviews) are provided in SHASUMS.asc (signed by PGP key 0x39DB6ED67D52F190). Re-running the commands with a different (more recent) "References.ris" file might result in different shasums due to changes in the hosting servers; for this reason, all intermediary files are provided.

Dataset produced from bibliographic references to COVID-19 systematic reviews obtained from the COVID-19 Living Overview of Evidence database. We obtained accessibility information and download links from the Unpaywall database, and indexed the resulting downloaded files with the OpenSemanticSearch search engine.