<h2>MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models</h2> <p>MuChoMusic is a benchmark designed to evaluate music understanding in multimodal language models focused on audio. It includes 1,187 multiple-choice questions validated by human annotators, based on 644 music tracks from two publicly available music datasets. These questions cover a wide variety of genres and assess knowledge and reasoning across several musical concepts and their cultural and functional contexts. The benchmark provides a holistic evaluation of five open-source models, revealing challenges such as over-reliance on the language modality and highlighting the need for better multimodal integration.</p> <h3>Note on Audio Files</h3> <p>This dataset comes without audio files. The audio files can be downloaded from two datasets: <a href="https://doi.org/10.5281/zenodo.10072001" target="_new" rel="noreferrer">SongDescriberDataset (SDD)</a> and <a href="https://www.kaggle.com/datasets/googleai/musiccaps" target="_new" rel="noreferrer">MusicCaps</a>. Please see the <a href="https://github.com/mulab-mir/muchomusic" target="_new" rel="noreferrer">code repository</a> for more information on how to download the audio.</p> <h3>Citation</h3> <p>If you use this dataset, please cite our <a href="https://arxiv.org/abs/2408.01337" target="_blank" rel="noopener">paper</a>:</p> <pre><code>@inproceedings{weck2024muchomusic, title={MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models}, author={Weck, Benno and Manco, Ilaria and Benetos, Emmanouil and Quinton, Elio and Fazekas, György and Bogdanov, Dmitry}, booktitle = {Proceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR)}, year={2024} }</code></pre> Weck B, Manco I, Benetos E, Quinton E, Fazekas G, Bogdanov D. MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models. In: Kaneshiro B, Mysore G, Nieto O, Donahue C, Huang CZA, Lee JH, McFee B, McCallum M, editors. Proceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR2024); 2024 November 10-14; San Francisco, USA.