Similarity matching of SIFT descriptors has been popularly used for object recognition. Conventional keypoint-to-keypoint matching is highly prone to error when the repetitive structure of a building generates a number of keypoints with similar descriptors. We propose a two-step clustering of SIFT keypoints, which can be used in cluster-to-cluster matching for building recognition. The first stage of clustering is on the 128-D local gradient vector and the second stage is on the 2-D coordinates and 1-D dominant orientation. Two-step clustering generates sub-clusters which are well separated both in location and in rotational symmetry. We achieved successful result in grouping of semantically homogeneous keypoints into clusters. Keywords- SIFT, mean-shift clustering, relaxation, matching, building recognition