Milk from cows grazing on alpine pastures has higher quality than milk from indoor-fed cows, likely due to diet-driven differences in rumen microbiota. We assessed the effects of supplementing alpine herbs—each varying in its content of fiber, protein, and polyphenol—on rumen microbiota via in vitro fermentation, comparing these to a grass hay control using metagenomic sequencing. Fermentations with alpine herbs compared to grass hay control had higher content of fibrolytic Prevotella and lower abundances of Butyrivibrio, Ruminococcaceae, Anaerovibrio, Succiniclasticum, and Desulfovibrio. Fermentations with high starch content (Alchemilla vulgaris, Gallium odoratum and Sanguisorba officinalis) had low, microbial diversity, while fermentations with high content of structural fibre (Sisymbrium officinale, Tanacetum vulgare, and Cicerbita alpina) had high microbial diversity. C. alpina, Sa. officinalis, and T. vulgare fermentations that had high lignin content showed a higher abundance of Bacteroidetes and a lower abundance of Firmicutes. Fermentations with high protein content (G. odoratum and T. vulgare) induced higher abundance of fibrolytic Lachnospiraceae. Sa. officinalis and A. vulgaris fermentations with high content of polyphenols were associated with increased abundances of Streptococcus and family RF-16 and lower abundances of family BS11 and Desulfovibrio. Fermentations with C. alpina and Si. Officinale induced higher abundance of fibrolytic Fibrobacter succinogenes. The beta diversity between fermentations corresponded to differences in the contents of protein, lignin, and polyphenols in the plant material. In conclusion, different herbs can promote the abundance of various fibrinolytic bacteria and change the microbial diversity, which has potential to increase the feed efficiency and the robustness of microbiota and reduce methane production.