We study automatic target recognition (ATR) in infrared (IR) imagery by applying two recent computer vision techniques, Histogram of Oriented Gradients (HOG) and Bag-of-Words (BoW). We propose the idea of dense HOG features which are extracted from a set of high-overlapped local patches in a small IR chip and we apply a vocabulary tree that is learned from a set of training images to support efficient and scalable BoW-based ATR. We develop a relevance grouping of vocabulary (RGV) technique to improve the ATR performance by additional voting from grouped visual words. Different from traditional word grouping techniques, RGV groups visual words of the same dominant class to enhance the voting confidence in BoW-based classification. The proposed ATR algorithm is evaluated against recent sparse representation-based classification (SRC) approaches that reportedly outperform traditional methods. Experimental results on the COMANCHE IR dataset demonstrate the advantages of the newly proposed algorithm (BoW-RGV) over the recent SRC approaches.