Using relationships between open and closed strings, we present a construction of tree-level scattering amplitudes for gravitons minimally coupled to matter in terms of gauge theory partial amplitudes. In particular, we present examples of amplitudes with gravitons coupled to vectors or to a single fermion pair. We also present two examples with massive graviton exchange, as would arise in the presence of large compact dimensions. The gauge charges are represented by flavors of dynamical scalars or fermions. This also leads to an unconventional decomposition of color and kinematics in gauge theories.