We propose and demonstrate a single-photon sensitive technique for optical vibrometry. It uses high speed photon counting to sample the modulated backscattering from a vibrating target. Designed for remote vibration sensing with ultralow photon flux, we show that this technique can detect small displacements down to 110 nm and resolve vibration frequencies from DC up to several kilohertz, with ≤ 0.01 detected photons per pulse. This single-photon sensitive optical vibrometry may find important applications in acousto-optic sensing and imaging, especially in photon-starved environments.