Debate rages as to whether habitat fragmentation leads to the decline of biodiversity once habitat loss is accounted for. Previous studies have defined fragmentation variously, but research needs to address “fragmentation per se,” which excludes confounding effects of habitat loss. Our study controls for habitat area and employs a mechanistic multi-species simulation to explore processes that may lead some species groups to be more or less sensitive to fragmentation per se. Our multi-land-cover, landscape-scale, individual-based model incorporates the movement of generic species, each with different land cover preferences. We investigate how fragmentation per se changes diversity patterns; within (alpha), between (beta) and across (gamma) patches of a focal-land-cover, and if this differs among species groups according to their specialism and dependency on this focal-land-cover. We defined specialism as the increased competitive ability of specialists in suitable habitat and decreased ability in less suitable land covers compared to generalist species. We found fragmentation per se caused an increase in gamma diversity in the focal-land-cover if we considered all species regardless of focal-land-cover preference. However, critically for conservation, the gamma diversity of species for whom the focal land cover is suitable habitat declined under fragmentation per se. An exception to this finding occurred when these species were specialists, who were unaffected by fragmentation per se. In general, focal-land-cover species were under pressure from the influx of other species, with fragmentation per se leading to a loss of alpha diversity not compensated for by increases in beta diversity and, therefore, gamma diversity fell. The specialist species, which were more competitive, were less affected by the influx of species and therefore alpha diversity decreased less with fragmentation per se and beta diversity compensated for this loss, meaning gamma diversity did not decrease. Our findings help to inform the fragmentation per se debate, showing that effects on biodiversity can be negative or positive, depending on species’ competitive abilities and dependency on the fragmented land cover. Such differences in the effect of fragmentation per se would have important consequences for conservation. Focusing conservation efforts on reducing or preventing fragmentation in areas with species vulnerable to fragmentation.