AbstractThe kappa statistic is widely used to assess the agreement between two raters. Motivated by a simulation‐based cluster bootstrap method to calculate the variance of the kappa statistic for clustered physician–patients dichotomous data, we investigate its special correlation structure and develop a new simple and efficient data generation algorithm. For the clustered physician–patients dichotomous data, based on the delta method and its special covariance structure, we propose a semi‐parametric variance estimator for the kappa statistic. An extensive Monte Carlo simulation study is performed to evaluate the performance of the new proposal and five existing methods with respect to the empirical coverage probability, root‐mean‐square error, and average width of the 95% confidence interval for the kappa statistic. The variance estimator ignoring the dependence within a cluster is generally inappropriate, and the variance estimators from the new proposal, bootstrap‐based methods, and the sampling‐based delta method perform reasonably well for at least a moderately large number of clusters (e.g., the number of clusters ). The new proposal and sampling‐based delta method provide convenient tools for efficient computations and non‐simulation‐based alternatives to the existing bootstrap‐based methods. Moreover, the new proposal has acceptable performance even when the number of clusters is as small as K = 25. To illustrate the practical application of all the methods, one psychiatric research data and two simulated clustered physician–patients dichotomous data are analyzed. Copyright © 2014 John Wiley & Sons, Ltd.