Phase shifted sinusoidal projection patterns are extensively used in structured light systems due to their low decoding complexity and projection of only three frames per reconstruction. However, they require correct unwrapping of phase images to reconstruct depth accurately. For time varying scenes, it is important for the phase unwrapped results to be temporally coherent. In this paper, we propose a spatio-temporal phase unwrapping algorithm for a structured light system made of a projector and a pair of stereo cameras. Unlike existing frame by frame spatial domain approaches, our proposed algorithm results in a temporally consistent three-dimensional unwrapped phase volume for time varying scenes. We experimentally show the effectiveness of our approach in recovering absolute phase for scenes with multiple disjoint objects, significant motion, and large depth discontinuities.