Computer-aided diagnosis (CAD) has great potential in providing real benefits to doctors and patients. Recent studies have, however, found lack of trust in CAD by radiologists in clinical diagnostic decision making. One of the main reasons is the lack of an appropriate confidence measure. This paper presents the first-ever study of classification confidence in the context of breast mass classification. We evaluated 11 state-of-the-art classification algorithms on breast mass image data using their confidence of classification metric, in addition to other standard evaluation metrics including accuracy and area under the curve (ROC). Experimental results show that although most classifiers produced very similar results with less than 2% difference in terms of accuracy and ROC, their performances are significantly different in terms of confidence levels. We suggest that the confidence measure should be used in conjunction with the existing performance metrics such as accuracy and ROC.