Division of labour is of fundamental importance for the success of societies, yet little is known about how individual specialization affects the fitness of the group as a whole. While specialized workers may be more efficient in the tasks they perform than generalists, they may also lack the flexibility to respond to rapid shifts in task needs. Such rigidity could impose fitness costs when societies face dynamic and unpredictable events, such as an attack by socially parasitic slavemakers. Here, we experimentally assess the colony-level fitness consequences of behavioural specialization in Temnothorax longispinosus ants that are attacked by the slavemaker ant T. americanus . We manipulated the social organization of 102 T. longispinosus colonies, based on the behavioural responses of all 3842 workers. We find that strict specialization is disadvantageous for a colony's annual reproduction and growth during slave raids. These fitness costs may favour generalist strategies in dynamic environments, as we also demonstrate that societies exposed to slavemakers in the field show a lower degree of specialization than those originating from slavemaker-free populations. Our findings provide an explanation for the ubiquity of generalists and highlight their importance for the flexibility and functional robustness of entire societies.