Non-lichenized rock-inhabiting fungi (RIF) are slow-growing melanized ascomycetes colonizing rock surfaces in arid environments. They possess adaptations, which allow them to tolerate extreme abiotic conditions, such as high UV radiations and extreme temperatures. They belong to two separate lineages, one consisting in the sister classes Dothideomycetes and Arthoniomycetes (Dothideomyceta), and the other consisting in the order Chaetothyriales (Eurotiomycetes). Because RIF often form early diverging groups in Chaetothyriales and Dothideomyceta, the ancestors of these two lineages were suggested to most likely be rock-inhabitants. The lineage of RIF related to the Chaetothyriales shows a much narrower phylogenetic spectrum than the lineage of RIF related to Dothideomyceta, suggesting a much more ancient origin for the latter. Our study aims at investigating the times of origin of RIF using a relaxed clock model and several fossil and secondary calibrations. Our results show that the RIF in Dothideomyceta evolved in the late Devonian, much earlier than the RIF in Chaetothyriales, which originated in the middle Triassic. The origin of the chaetothyrialean RIF correlates well with a period of recovery after the Permian-Triassic mass extinction and an expansion of arid landmasses. The period preceding the diversification of the RIF related to Dothideomyceta (Silurian--Devonian) is also characterized by large arid landmasses, but temperatures were much cooler than during the Triassic. The paleoclimate record provides a good explanation for the diversification of fungi subjected to abiotic stresses and adapted to life on rock surfaces in nutrient-poor habitats.