Cytogenetics plays a central role in the detection of chromosomal abnormalities and in the diagnosis of genetic diseases. A karyogram is an image representation of human chromosomes arranged in order of decreasing size and paired in 23 classes. In this paper we propose an approach to automatically pair the chromosomes into a karyogram, using the information obtained in a rough SVM-based classification step, to help the pairing process mainly based on similarity metrics between the chromosomes. Using a set of geometric and band pattern features extracted from the chromosome images, the algorithm is formulated on a Bayesian framework, combining the similarity metric with the results from the classifier. The solution is obtained solving a mixed integer program. Two datasets with contrasting quality levels and 836 chromosomes each were used to test and validate the algorithm. Relevant improvements with respect to the algorithm described by the authors in [1] were obtained with average paring rates above 92%, close to the rates obtained by human operators.