Our study aims to describe a computer-aided method of searching for conspiracy thinking in unstructured textual data. Collecting such data from the Internet usually involves using keywords to find relevant documents for further analysis. Although this step determines the results, many researchers select keywords arbitrarily without evaluating their tools. We introduced a method of keyword expansion that combines word embeddings and human cognitive abilities to identify potential keywords. In our study, we found that the relatively informed participants (N = 154) could not recall even a short list of relevant keywords, and the ones they selected were mostly useless in detecting conspiracy thinking. The designed Conspiracy Thinking Index performed better in detecting conspiracy-related text in a large text corpus (≈ 1.1M tweets) than supervised machine learning algorithms while remaining simple and transparent.