Seafloor sediments cover the majority of planet Earth and microorganisms inhabiting these environments play a central role in marine biogeochemical cycles. Yet, description of the biogeography and distribution of sedimentary microbial life is still too sparse to evaluate the relative contribution of processes driving this distribution, such as the levels of drift, connectivity, and specialization. To address this question, we analyzed 210 archaeal and bacterial metabarcoding libraries from a standardized and horizon-resolved collection of sediment samples from 18 stations along a longitudinal gradient from the eastern Mediterranean to the western Atlantic. Overall, we found that biogeographic patterns depended on the scale considered: while at local scale the selective influence of contemporary environmental conditions appeared strongest, the heritage of historic processes through dispersal limitation and drift became more apparent at regional scale, and ended up superseding contemporary influences at inter-regional scale. When looking at environmental factors, the structure of microbial communities was correlated primarily with water depth, with a clear transition between 800 and 1,200 meters below sea level. Oceanic basin, water temperature, and sediment depth were other important explanatory parameters of community structure. Finally, we propose increasing dispersal limitation and ecological drift with sediment depth as a probable factor for the enhanced divergence of deeper horizons communities.