AbstractIn ranked‐set sampling, the rankings may be either perfect or imperfect. Statistical procedures that assume perfect rankings tend to be more efficient than procedures that do not assume perfect rankings when perfect rankings actually hold, but may perform poorly if the rankings are imperfect. Several procedures have been developed for testing the null hypothesis of perfect rankings, but these procedures break down if the data are not continuous. In this article, we develop tests of perfect rankings that can be applied with binary data. Motivated by new theoretical results about how the success probabilities in the judgment strata differ under perfect and imperfect rankings, we develop a consistent test with a test statistic that is asymptotically normal. We find, however, that the test does not properly control the type I error rate with small samples. This motivates us to instead implement a bootstrap version of the test. This bootstrap test controls the type I error rate even with small sample sizes. Functions for implementing both tests using R are available in the Supplementary Material. The Canadian Journal of Statistics 45: 326–339; 2017 © 2017 Statistical Society of Canada