We present a system for modeling buildings from a single correlation-based Digital Elevation Model (DEM). The model is constructed in two stages. The first stage segments the DEM into planar surface patches that describe the building. The second stage generates the final polygonal model of the building using weak geometric constraints. We use robust estimation methods at different stages of our modeling process to develop an efficient and noise-insensitive modeling system. The proposed system is fully automatic and does not use any a priori information about the shape of the buildings. We present results on isolated buildings and on a large area of the city of Berlin.