We propose the end-to-end multimodal fact-checking and explanation generation, where the input is a claim and a large collection of web sources, including articles, images, videos, and tweets, and the goal is to assess the truthfulness of the claim by retrieving relevant evidences and predicting a truthfulness label (i.e., support, refute and not enough information), and to generate a rationalization statement to explain the reasoning and ruling process. To support this research, we construct MOCHEG, a large-scale dataset consisting of 21,184 claims where each claim is annotated with a truthfulness label and ruling statement, with 43,148 text evidences and 15,373 image evidences.

The complete dataset can be accessed from http://nlplab1.cs.vt.edu/~menglong/project/multimodal/fact_checking/MOCHEG/dataset/. The paper is in https://arxiv.org/abs/2205.12487. The version on the zenodo website does not contain the images.