There are two sets of four-by-four matrices introduced by Dirac. The first set consists of fifteen Majorana matrices derivable from his four γ matrices. These fifteen matrices can also serve as the generators of the group SL(4, r). The second set consists of ten generators of the Sp(4) group which Dirac derived from two coupled harmonic oscillators. It is shown possible to extend the symmetry of Sp(4) to that of SL(4, r) if the area of the phase space of one of the oscillators is allowed to become smaller without a lower limit. While there are no restrictions on the size of phase space in classical mechanics, Feynman’s rest of the universe makes this Sp(4)-to-SL(4, r) transition possible. The ten generators are for the world where quantum mechanics is valid. The remaining five generators belong to the rest of the universe. It is noted that the groups SL(4, r) and Sp(4) are locally isomorphic to the Lorentz groups O(3, 3) and O(3, 2) respectively. This allows us to interpret Feynman’s rest of the universe in terms of space-time symmetry.