What started as a field with an emphasis on optimally serving users' interactive information needs has now become dominated by methods that focus on improving the mean average precision (MAP) of a clearly defined task disconnected from its application. With the pervasiveness of the Internet and all the sensors available to derive contextual user information, it is time to bring the data and the user back together. As a field, we must consider understanding the subjective and descriptive nature of users and understanding data as equally interesting research topics that are both worthy of publication. At the 2012 ACM Second Annual International Conference on Multimedia Retrieval (ICMR) in Hong Kong, a panel took place with Marcel Worring as the moderator and the other authors of this article as the panelists. This panel discussion explored this intriguing question: Where is the user in multimedia retrieval?