Past research into multi-modality sensor data fusion has given rise to approaches that are generally heuristic and ad hoc. In this paper we utilize the calculus of variations as the underlying framework for fusing registered images of different modalities when models relating these modalities are available. The result is a mathematically rigorous method for improving the accuracy with which parameters can be estimated. Using both dense and sparse simulated range and intensity data, the proposed approach is demonstrated on the problem of estimating the surface representing the three dimensional structure of a scene. The results indicate that a four to five-fold increase in surface estimation accuracy with respect to the original input data can be realized. Furthermore, an 8%–250% increase in accuracy over surface estimation from each sensing modality alone (i.e., via shape from shading or surface reconstruction) can be realized.