We continue the work of [Phys. Rev. D 72, 024001 (2005)] in which gravity is considered as the Goldstone realization of a spontaneously broken diffeomorphism group. We complete the discussion of the coset space $\mathrm{Diff}\text{ }(d,\mathbb{R})/SO(1,d\ensuremath{-}1)$ formed by the $d$-dimensional group of analytic diffeomorphisms and the Lorentz group. We find that this coset space is parametrized by coordinates, a metric, and an infinite tower of higher-spin or generalized connections. We then study effective actions for the corresponding symmetry breaking which gives mass to the higher spin connections. Our model predicts that gravity is modified at high energies by the exchange of massive higher spin particles.