Similarity based clustering, which is to find the extrinsic clusters in data by taking as input a collection of real-valued similarities between data points, has been playing an important role in data analysis and engineering. Lots of work had been done in this field. However, data clustering is an rather challenge problem as there is no labeled data available. We observe that an ideal similarity matrix should be close to an adjacency matrix up to a scale. Based on this idea, we develop a scaled adjacency matrix (SAM) clustering algorithm that could find an optimal adjacency matrix in some sense for a given similarity matrix. Based on the learnt adjacency matrix, clustering could be performed straightforwardly. Upon three assumptions on the similarity matrix, we prove that the performance of SAM is robust. Experimental results also show that SAM is effective.