Symbiotic microbial communities are important for host health, but the processes shaping these communities are poorly understood. Understanding how community assembly processes jointly affect microbial community composition is limited because inflexible community models rely on rejecting dispersal and drift before considering selection. We developed a flexible community assembly model based on neutral theory to ask: How do dispersal, drift, and selection concurrently affect the microbiome across environmental gradients? We applied this approach to examine how a fungal pathogen affected the assembly processes structuring the amphibian skin microbiome. We found that the rejection of neutrality for the amphibian microbiome across a fungal gradient was not strictly due to selection processes, but was also a result of species-specific changes in dispersal and drift. Our modeling framework brings the qualitative recognition that niche and neutral processes jointly structure microbiomes into quantitative focus, allowing for improved predictions of microbial community turnover across environmental gradients.