In this paper, we assume that we are given the images of two segmented objects, one of which may be an affinely distorted version of the other, and wish to recover the values of the parameters of the affine transformation between the two images. The images may also differ by the overall level of illumination. The multiplicative constant of such difference may also be recovered. We present a generic theoretical framework to solve this problem. In terms of this framework, other proposed methods may be interpreted. We show how, in this framework, one can recover the affine parameters in a way that is robust to various effects, such as occlusion and illumination variation. The proposed method is generic enough to be applicable also to matching two images that do not depict the same scene or object.