Abstract†Pycnodontiformes was a successful lineage of primarily marine fishes that broadly diversified during the Mesozoic. They possessed a wide variety of body shapes and were adapted to a broad range of food sources. Two other neopterygian clades possessing similar ecological adaptations in both body morphology (†Dapediiformes) and dentition (Ginglymodi) also occurred in Mesozoic seas. Although these groups occupied the same marine ecosystems, the role that competitive exclusion and niche partitioning played in their ability to survive alongside each other remains unknown. Using geometric morphometrics on both the lower jaw (as constraint for feeding adaptation) and body shape (as constraint for habitat adaptation), we show that while dapediiforms and ginglymodians occupy similar lower jaw morphospace, pycnodontiforms are completely separate. Separation also occurs between the clades in body shape so that competition reduction between pycnodontiforms and the other two clades would have resulted in niche partitioning. Competition within pycnodontiforms seemingly was reduced further by evolving different feeding strategies as shown by disparate jaw shapes that also indicate high levels of plasticity. Acanthomorpha was a teleostean clade that evolved later in the Mesozoic and which has been regarded as implicated in driving the pycnodontiforms to extinction. Although they share similar body shapes, no coeval acanthomorphs had similar jaw shapes or dentitions for dealing with hard prey like pycnodontiforms do and so their success being a factor in pycnodontiform extinction is unlikely. Sea surface temperature and eustatic variations also had no impact on pycnodontiform diversity patterns according to our results. Conversely, the occurrence and number of available reefs and hardgrounds as habitats through time seems to be the main factor in pycnodontiform success. Decline in such habitats during the Late Cretaceous and Palaeogene might have had deleterious consequences for pycnodontiform diversity. Acanthomorphs occupied the niches of pycnodontiforms during the terminal phase of their existence.