AbstractBackgroundThe Netherlands Pharmacovigilance Centre Lareb uses an operating Signal Impact Assessment Tool (SIAT) as aid in signal selection. SIAT prioritized signals into one of four categories: strong/moderate signal strength and similarly health impact. Although the SIAT has been used for many years, validity and reliability was never explored.PurposeThe aim of this study is to test the validity and reliability of the operating and weight‐adjusted SIAT.MethodFor validity testing, judgments of a Delphi panel of three pharmacovigilance experts were used as a ‘gold standard’. First, the panel judged the weighting of the items included in the SIAT. Then, during two phases, the panel rated 40 signals in one of the four categories. Two researchers scored the signals using the SIAT. Panel judgments were compared with scores for the operating and weight‐adjusted SIAT. Inter‐ and intra‐observer variability was also tested. The Cohen's Kappa coefficient (k) was calculated. At least substantial agreement (k > 0.6) was considered to be necessary for an acceptable reliability.ResultsValidity did not meet predefined criteria: operating and weight‐adjusted tool, respectively, k‐phase1 = 0.83 and 0.83, k‐phase2 = 0.18 and 0.36. Differences were found for signal strength and health impact. Inter‐ and intra‐observer variabilities were good, k of 0.78 and 0.72, respectively.ConclusionsThe SIAT was found to have an insufficient validity and proper reliability. Although SIAT scores should not be decisive in the decision making process, items included can be used as an aid to decide which signals deserve further investigation. Copyright © 2016 John Wiley & Sons, Ltd.