It has recently been argued that ChatGPT might become a serious competitor to quantitative approaches to identify similar institutions for the purposes of comparing research performance in benchmarking studies. However, it is largely unknown how ChatGPT finds out ‘similar’ universities and whether results depend on the information provided by the user, and are stable over different queries. To address these questions, in this study, we resort to a sample of more than 1,000 universities included in the European Tertiary Education Register (ETER). We test different ChatGPT queries providing different levels of information on the focal university and we compare results with similarities computed from quantitative data in ETER. Preliminary results suggest that ChatGPT is able to identify some good peers, but the results strongly depend on the querying strategy and on the universities’ characteristics.