A distinctive feature of the irregular moons of the giant planets is their orbital grouping. Previously, the prograde and retrograde groups of irregular moons at Jupiter were believed to be groups of fragments produced by the disruption of two large moons. More recently, we have shown that the retrograde group has not one but probably four or more parent bodies. We also found that fragments were launched from two of the four identified parent moons, producing two clusters of irregular moons with members of each group having similar orbits. Named the Ananke and Carme families, these two groups consist of seven and nine known member moons, respectively. The origin of this orbital clustering is unknown. Current rates of collisions among satellites in the retrograde group are too low to explain them. Collisions with cometary impactors are even less likely. Groups of irregular satellites with similar inclinations at Saturn are also yet to be explained. It is conceivable that the satellite families are remnants from early epochs of solar system formation when impactors were more numerous. In this paper we investigate the possibility that satellite families formed via collisions between large parent moons and stray planetesimals. We find that the Ananke and Carme families at Jupiter could indeed have been produced by this mechanism, unless the residual disk of planetesimals in heliocentric orbit was already severely depleted when the irregular satellites formed. Conversely, we find that formation of the Himalia group of prograde Jovian satellites by the same mechanism is unlikely unless a massive residual planetesimal disk was still present when the progenitor moon of the Himalia group was captured. We place constraints on the mass of the residual disk (1) when satellites were captured, and (2) when the Ananke and Carme families formed. These values depend sensitively on the assumed size-frequency distribution of planetesimals.