Fitness equalizing mechanisms, such as trade-offs, are recognized as one of the main factors promoting species coexistence in community ecology. However, they have rarely been explored in microbial communities. Although microbial communities are highly diverse, the coexistence of their multiple taxa is largely attributed to niche differences and high dispersal rates, following the principle “everything is everywhere, but the environment selects”. We use a dynamical stochastic model based on the Theory of Island Biogeography to study highly diverse microbial communities over time across three different systems (soils, alpine lakes, and shallow saline lakes). Assuming fitness equalization mechanisms, here we newly analytically derive colonization-persistence trade-offs, and report a signal of such trade-offs in natural microbial communities. Moreover, we show that different subsets of species in the community drive this trade-off. Rare taxa, which are occasional and more likely to follow independent colonization/extinction dynamics, drive this trade-off in the aquatic communities, while the core sub-community did it in the soils. We conclude that equalizing mechanisms may be more important than previously recognized in microbial communities. Our work also emphasizes the fundamental value of dynamical models for understanding temporal patterns and processes in highly diverse communities.