The performance of a supercapacitor can be characterized by a series of key parameters, including the cell capacitance, operating voltage, equivalent series resistance, power density, energy density, and time constant. To accurately measure these parameters, a variety of methods have been proposed and are used in academia and industry. As a result, some confusion has been caused due to the inconsistencies between different evaluation methods and practices. Such confusion hinders effective communication of new research findings, and creates a hurdle in transferring novel supercapacitor technologies from research labs to commercial applications. Based on public sources, this article is an attempt to inventory, critique and hopefully streamline the commonly used instruments, key performance metrics, calculation methods, and major affecting factors for supercapacitor performance evaluation. Thereafter the primary sources of inconsistencies are identified and possible solutions are suggested, with emphasis on device performance vs. material properties and the rate dependency of supercapacitors. We hope, by using reliable, intrinsic, and comparable parameters produced, the existing inconsistencies and confusion can be largely eliminated so as to facilitate further progress in the field.