How does nuclear binding emerge from first principles? Our current best understanding of nuclear forces is based on a systematic low-energy expansion called chiral effective field theory. However, recent {\it ab initio} calculations of nuclear structure have found that not all chiral effective field theory interactions give accurate predictions with increasing nuclear density. In this letter we address the reason for this problem and the first steps toward a solution. Using nuclear lattice simulations, we deduce the minimal nuclear interaction that can reproduce the ground state properties of light nuclei, medium-mass nuclei, and neutron matter simultaneously with no more than a few percent error in the energies and charge radii. We find that only four parameters are needed. With these four parameters one can accurately describe neutron matter up to saturation density and the ground state properties of nuclei up to calcium. Given the absence of sign oscillations in these lattice Monte Carlo simulations and the mild scaling of computational effort scaling with nucleon number, this work provides a pathway to high-quality simulations in the future with as many as one or two hundred nucleons.

Version accepted for publication in Physics Letters B. For better readability, the main section and the supplementary material are merged