We propose a new speckle-based plethysmography technique, termed rolling shutter speckle plethysmography (RSSPG), which can quantitatively measure the velocity and volume fluctuations of blood flow during the cardiac cycle. Our technique is based on the rolling shutter speckle imaging, where the short row-by-row time differences in the rolling shutter image sensors are used to measure the temporal decorrelation behavior of vertically elongated speckles from a single image capture. Temporal analysis of the speckle field provides rich information regarding the dynamics of the scattering media, such as both the dynamic scattering fraction and the speckle decorrelation time. Using a sequence of images, RSSPG can monitor fluctuations in the blood flow dynamics while separating velocity and volume changes in blood vessels and obtaining high-quality plethysmography waveforms compared to regular photoplethysmography. We demonstrate the quantitative RSSPG based on accurate fitting of the speckle dynamics model, as well as the qualitative RSSPG based on simple row-by-row correlation (RIC) calculation for fast and robust analysis. Based on exploratory in vivo experiment, we show that RSSPG can reliably measure pulsatile waveforms and heart rate variations in various conditions, potentially providing physiologically relevant information for cardiovascular monitoring.