Accurately reproducing the morphology of galaxies in our Universe is a crucial test for hydrodynamical cosmological simulations of galaxy evolution. We assess the extent to which the fine details of the stellar galaxy morphology are reproduced in simulations using a novel deep learning approach. Our unsupervised methodology provides a metric that is a robust measure of the relationship between the galaxy morphological details and its overall structure, which goes well beyond current parametric and non-parametric approaches. We test fully realistic r-band mock observations of galaxies from the Illustris simulation and the improved IllustrisTNG model against SDSS observations. We can quantitatively identify the improvements of IllustrisTNG, particularly in the high-resolution TNG50 run, over the original Illustris. However, we find that the fine details of galaxy structure are still different between observed and simulated galaxies. This difference is driven by small, spheroidal, and quenched galaxies which are globally less accurate regardless of resolution and which have experienced little improvement between the simulations explored. We speculate that this disagreement may stem from a still too coarse numerical resolution, which struggles to properly capture the inner, dense regions of quenched spheroidal galaxies. I will also present efforts being made to extend this strategy to HST data, and will put these in the perspective of the coverage area of the Roman telescope.