In this paper we propose a parallel network which gradually computes the three-dimensional (3-D) structure of a moving scene from its image sequence, using an incremental scheme based upon a constraint called the maximal rigidity principle. At each instant an internal model (i.e., current estimate) of the 3-D structure is updated, based upon the observations accumulated until then. The updating process favors rigid transformations but tolerates a limited amount of deviation from rigidity. This deviation eventually leads the internal model to converge towards the actual 3-D structure of the scene, An application of this network to the problem of structure from two views is also presented. The main advantage of this architecture is its ability to accurately estimate the 3-D structure of a scene, at a low computational cost. Testing has been successfully performed on synthetic data as well as real image sequences.