We study neutron matter and symmetric nuclear matter with the quark-meson model for the two-nucleon interaction. The Bethe-Bruckner-Goldstone many-body theory is used to describe the correlations up to the three hole-line approximation with no extra parameters. At variance with other non-relativistic realistic interactions, the three hole-line contribution turns out to be non-negligible and to have a substantial saturation effect. The saturation point of nuclear matter, the compressibility, the symmetry energy and its slope are within the phenomenological constraints. Since the interaction also reproduces fairly well the properties of the three nucleon system, these results indicate that the explicit introduction of the quark degrees of freedom within the considered constituent quark model is expected to reduce the role of three-body forces.

4 pages, 4 figures, 2 tables, matches the journal version