Music similarity is used in many applications ranging from music recommendations to media retrieval systems. Most of music similarity metrics are computed as distances between songs on a certain multidimensional feature space. This feature space typically consists of audio descriptors that are extracted from raw audio (rhythm, tonality, spectral features). By using similarity distances we can perform a query in database, and it will return number of tracks with shortest similarity distances, thus the most similar ones. Intrinsically similarity is a subjective concept, because different listeners perceive different things as similar. In the research of music similarity it had been defined in different ways and various similarity measures had been proposed. Evaluation of those metrics is a difficult task, and is usually done via objective metrics because it requires less resources. Similarity can be measured either subjectively by using relative comparison by human subjects or objectively by using music metadata such as artist, album, genre and cultural context. Example of context usage is co-occurrence of the tracks on radio broadcasts or compilation CDs. However, subjective evaluation of most similarity metrics that are introduced in the research is usually performed with limited number of participants due to nature of experiments. In this thesis we address this problem and propose a system that can be used to perform subjective evaluation of different similarity metrics at large scale. AcousticBrainz is an online platform that aggregates music descriptors extracted from audio with globally identifiable ID for each track. We use this platform for implementation of state-of-the art similarity metrics as well as crowd-sourcing evaluation capabilities. Using user feedback as the evaluation source we are able to compare performance of various algorithms at large scale. That is achieved by providing means of similarity computation between any of millions of songs in AcousticBrainz within acceptable time. We also explore the possibility of allowing users to create their own hybrid similarity metrics and evaluate their performance in the system. The system allows to compare different metrics according to different similarity dimensions (timbre, rhythm) and calculate the performance based on subjective evaluation by users. Our results confirm the relative performance of algorithms from the previous research based on subjective evaluation and the platform shows support for much larger scale experiments and potential of being used for various practical purposes.