Object based video coding is gaining considerable attention as it should allow possible object-based interactivity and the elimination of mosquito and block artifacts typical of block coding. Algorithms ensuring a good scene segmentation based only on motion information are still far from reliable. In this work we present an algorithm for jointly estimating motion fields and obtaining a segmentation of the scene in coherent motion areas, using only information obtained by block matching. A quadratic model is used to represent inter block dependencies of motion estimates. Region growing is used to gather blocks endowed with compatible matching functions and produce an initial scene segmentation. Refinements are obtained with a subsequent iterative procedure. Experimental results show a good correspondence of the motion field segmentation with the objects in the scene and prediction errors close to those of block matching, but with a much more regular motion field.