ABSTRACT Although speaker diarization has evolved to be more robust and more refined, including incorporating modern automatic speech recognition (ASR), current systems still suffer from several disruptive factors, like noise. We comprehensively evaluate the limitations of current diarization systems to uncover the underlying causes that hinder accuracy. Five open‐source diarization pipelines—both diarization‐only and joint ASR and diarization systems—are assessed on a set of heterogeneous benchmark data sets. We compare the performance of joint pipelines against those of diarization‐only systems, and analyse which audio characteristics hinder speaker discrimination, as well as the impact of using the speaker count as an input parameter. Our results indicate that diarization‐only and joint approaches are competitive with each other in unsupervised scenarios, and that providing the speaker count does not improve performance consistently. We also identify short audio duration and low speech‐to‐noise ratio (SNR) as the most impairing properties. We recommend using speech representation learning to further uncover underlying factors that affect diarization, pre‐processing techniques to remove noise, and performing hyper‐parameter tuning on, for example, the speech window length, and speech detection thresholds.