We propose a novel approach for American Sign Langauge (ASL) phrase verification that combines confidence measures (CM) obtained from aligning forward sign models (the conventional approach) to the input data with the CM's obtained from aligning reversed sign models to the same input. To demonstrate our approach we have used two CM's, the Normalized likelihood score and the Log-Likelihood Ratio (LLR).We perform leave-one-signer-out cross validation on a dataset of 420 ASL phrases obtained from five deaf children playing an educational game called CopyCat. The results show that for the new method the alignment selected for signs in a test phrase has a significantly better match to the ground truth when compared to the traditional approach. Additionally, when a low false reject rate is desired the new technique can provide a better verification accuracy as compared to the conventional approach.