Hand sign recognition is an essential part in robot control, human computer interaction, communication with deaf or speech impaired people etc. where performance and time complexity are very important factors. Numerous researches are conducted to offer solutions for sign language classification. Among them, orientation based hashcode (OBH) model recognizes sign images at a lower time but with A lower accuracy. In this paper, we propose a system which consists of OBH, additional feature extraction and machine learning method. It is able to classify sign language finger spelling alphabets efficiently within a short time. Feature vector using Gabor filter and number of fingertips are used as attributes alongside orientation based hashcode for classification through Artificial Neural Network (ANN). Before feeding features into ANN model, Principle Component Analysis (PCA) is used to omit the redundant features. The dataset contains 576 American Sign Language (ASL) alphabet sign images (both RGB and depth images) of 24 different categories which are captured by Microsoft Kinect sensor. The proposed methodology is proved to be 95.8% accurate against randomly selected test dataset and 93.85% accurate using 9-fold validation.