Various intrusion‐tolerant certification authority (CA) systems have been proposed to provide attack resilient certificate signing (or update) services. However, it is difficult to compare them against each other directly, due to the diversity in system organizations, threshold signature schemes, protocols and usage scenarios. We present a framework for intrusion‐tolerant CA system evaluation, which consists of three components, namely, an intrusion‐tolerant CA model, a threat model and a metric for comparative evaluation. The evaluation framework covers system organizations, protocols, usage scenarios, the period of certificate validity, the revocation rate and the mean time to recovery. Based on the framework, four representative systems are evaluated and compared in three typical usage scenarios, producing reasonable and insightful results. The interdependence between usage scenarios and system characteristics is investigated, providing a guideline to design better systems for different usage scenarios. The proposed framework provides an effective and practicable method to evaluate intrusion‐tolerant CA systems quantitatively, and helps customers to choose and configure an intrusion‐tolerant CA system. Moreover, the comparison results offer valuable insights to further improve the attack resilience of intrusion‐tolerant CA systems. Copyright © 2011 John Wiley & Sons, Ltd.