AbstractThis article addresses the problem of determining the 3‐dimensional locations of salient points in the environment of a moving camera based on a monocular image sequence obtained by the camera. The camera's translational and rotational velocities are assumed to be known approximately via inertial sensors. The motion of the camera is represented by a constant velocity model. Salient points in the image sequence are extracted using Gabor wavelets and tracked using labeled graph matching. The 3‐D positions of the selected environmental points relative to the camera are then estimated recursively using an extended Kalman filter (EKF), after initialization by two‐frame motion stereo. The motion parameters of the camera are also refined simultaneously. Experimental results on real data are given. © 1992 John Wiley & Sons, Inc.