The introduction of new bibliometric software packages in recent years has led to an exponential increase in the publication rate of bibliometric analyses. A review of 46 recently published articles that used one of these software packages—VOSviewer—to visualize the structure and evolution of a set of documents, suggests that there are three areas where the quality of these studies can be easily improved. First, replication – only two of the 13 articles that used experts to screen out irrelevant papers provided specific information on the surviving sample. Second, robustness – only 10 of the 46 articles provided extensive, actionable discussion of structure. None of these provided sufficient explanation of structure-based findings. Third, relevance – journal-based analyses used bibliometrics to support preconceived positions—namely that the journal being analyzed was exceptional in some positive way, while keyword-based analyses were susceptible to author bias, contributing to the risk of misrepresentation of the area under study. We provide suggestions as to how the bibliometric community might improve the state of the art in a way that can be used by non-experts.