Recent advances in paleoclimatology and the growing digital availability of large historical datasets on human activity have created new opportunities to investigate long‐term interactions between climate and society. However, noncritical use of historical datasets can create pitfalls, resulting in misleading findings that may become entrenched as accepted knowledge. We demonstrate pitfalls in the content, use and interpretation of historical datasets in research into climate and society interaction through a systematic review of recent studies on the link between climate and (a) conflict incidence, (b) plague outbreaks and (c) agricultural productivity changes. We propose three sets of interventions to overcome these pitfalls, which involve a more critical and multidisciplinary collection and construction of historical datasets, increased specificity and transparency about uncertainty or biases, and replacing inductive with deductive approaches to causality. This will improve the validity and robustness of interpretations on the long‐term relationship between climate and society. This article is categorized under: Climate, History, Society, Culture > Disciplinary Perspectives Recent literature investigating long‐term interactions between climate and society increasingly utilizes historical big data. Too often this is done without applying historical criticism, which may lead to misguided narratives. We propose a set of interventions to avoid this and optimize the use of historical datasets.