Light is used to steer the motion of atoms in free space, enabling cooling and trapping of matter waves through ponderomotive forces and Doppler-mediated photon scattering. Likewise, light interaction with free electrons has recently emerged as a versatile approach to modulate the electron wave function for applications in ultrafast electron microscopy. Here, we combine these two worlds, theoretically demonstrating that matter waves can be optically manipulated via inelastic interactions with optical fields. This allows us to modulate the translational part of the wave function and produce temporally and spatially compressed atomic beam pulses. We realize such modulation through stimulated photon absorption and emission by atoms traversing phase-matching evanescent optical fields generated upon light scattering by a nanostructure and via stimulated Compton scattering in free space without any assistance from material media. Our results support optical manipulation of matter waves as a powerful tool for microscopy, spectroscopy, and exploration of fundamental phenomena associated with light-atom interactions.