This paper presents a consistent framework for continuous stereo self-calibration. Based on a practical analysis of the sensitivity of stereo reconstruction to camera calibration uncertainties, we identify important parameters for self-calibration. We evaluate different geometric constraints for estimation and tracking of these parameters: bundle adjustment with reduced structure representation relating corresponding points in image sequences, the epipolar constraint between stereo image pairs, and trilinear constraints between image triplets. Continuous, recursive calibration refinement is obtained with a robust, adapted iterated extended Kalman filter. To achieve high accuracy, physically relevant geometric optimization criteria are formulated in a Gauss-Helmert type model. The self-calibration framework is tested on an active stereo system. Experiments with synthetic data as well as on natural indoor and outdoor imagery indicate that the different constraints are complementing each other and thus a method combining two of the above constraints is proposed: While reduced order bundle adjustment gives by far the most accurate results (and might suffice on its own in some environments), the epipolar constraint yields instantaneous calibration that is not affected by independently moving objects in the scene. Hence, it expedites and stabilizes the calibration process.