Galaxy clusters are the most recent of cosmological structures to have formed by the present time in the currently favoured hierarchical scenario of structure formation and are widely regarded as powerful probes of cosmology and galaxy formation physics alike. Over the past few years, it became increasingly clear that precision cluster cosmology requires the development of detailed, realistic theoretical models of galaxy clusters and the confrontation of synthetic surveys generated using these models with observations. This motivates a campaign of large cosmological hydrodynamical simulations, with plausible 'sub-grid' prescriptions for the relevant galaxy formation physics.This thesis presents a new suite of large-volume cosmological hydrodynamical simulations called cosmo-OWLS. They form an extension to the Overwhelmingly Large Simulations (OWLS) project, and have been designed to help improve our understanding of cluster astrophysics and the non-linear structure formation, which are now the limiting systematic errors when using clusters as cosmological probes. Starting from identical initial conditions in either the Planck or WMAP7 cosmologies, the most important 'sub-grid' physics, including feedback from supernovae and active galactic nuclei (AGN) has been systematically varied. Via the production of synthetic surveys of the simulations and comparisons with observations, the realism of these state-of-the-art models was explored. At the same time, the simulations were shown to providea valuable tool for interpreting the observational data, as well as powerful means for testing commonly-employed methods for estimating, for example, cluster masses and determining survey selection functions, which are crucial for cluster cosmology. The properties of the simulated galaxy groups and clusters were first compared to a wide range of observational data,such as x-ray luminosity and temperature, gas mass fractions, entropy and density profiles, Sunyaev-Zel'dovich flux, I-band mass-to-light ratio, dominance of the brightest cluster galaxy, and central massive black hole (BH) masses, by producing synthetic observations and mimicking observational analysis techniques. These comparisons demonstrated that some AGN feedback models can produce a realistic population of galaxy groups and clusters, broadly reproducing both the median trend and, for the first time, the scatter in physical properties over approximately two decades in mass