We present the Cardiff Conversation Database (CCDb), a unique 2D audiovisual database containing natural conversations between pairs of people. The database currently contains 30 conversations. To date, eight conversations are fully annotated for speaker activity, facial expressions, head motion, and non-verbal utterances. In this paper we describe the data collection and annotation process. We also provide results of baseline experiments in which an SVM classifier was used to identify which parts of the recordings are from the frontchannel speaker and which are backchannel signals. We believe this database will make a useful contribution to computer vision, affective computing, and cognitive science communities by providing raw data, features, annotations and baseline comparisons.