The petrochemical industry considers offline partial discharge (PD) measurement an important quality assessment for new medium voltage motor and generator stator windings. A recent edition of API 541 requires an instrument compliant with IEC 60270 for the measurement of PD on sacrificial stator coils, and proposes 100 pC (pico-Coulombs) for guidance on acceptance criteria until more data becomes available. Given the industry's faith in PD as an acceptance test based on a representative sample, it warrants closer scrutiny. The characteristic PD distribution pattern produced by each test instrument provides information on the origin of the discharges and therefore can be used to determine what type of defect has produced them. This paper shows how external factors such as instrument selection, testing environment, and time interval between tests can influence PD test results. Three different IEC-compliant instruments were used to measure PD from sample coils of the same design, under identical conditions. Each sample coil was modified to create the same imposed defect, with the same technique used to interpret the PD magnitude and distribution pattern produced in each case. Each instrument recorded a different charge magnitude and characteristic PD pattern. The variation in charge magnitude measured by each instrument exceeds the proposed API 541 acceptance limit, and the observed PD patterns did not accurately reflect the imposed coil defect. Offline PD measurement therefore lacks sufficient repeatability, reproducibility and accuracy to support a standard acceptance limit.