Freesound.org is an online collaborative sound database where people from different disciplines share recorded sound clips under Creative Commons licenses. Freesound’s search functionality is one the feature that is used by thousands of users everyday. Due to the rapid expansion of the sound collection and the variety within sounds, ranking quality of search results has become an important part of the platform. Automatically judging the quality of the ranking algorithms based on user clickthrough data holds promise for analyzing retrieval quality faster and cheaper. We investigate whether certain observable statistics relate to retrieval quality of Freesound. We propose a model that leverages the thousands of clicks received by Freesound to predict the retrieval quality. Six absolute metrics as usage statistics are hypothesized to motonically change with retrieval quality. We design three ranking strategies and expose users to each of these rankings as the default sorting of search results. We show that the change in metrics is correlated with the rank correlation of different rankings.