Abstract—Recognizing sign language gestures for different languages has been found as a promising field of research that explores the possibility of communication by interpreting various signs and translating them into text or speech. Establishing a better communication way between deaf-mute people and ordinary people is the prime objective of this research arena. There are many existing Sign Language Recognition (SLR) systems throughout the world and these SLR systems are implemented using various methods, tools and techniques with a view to achieving better recognition accuracy. This research work aims at applying the concept of Convolutional Neural Network (CNN) for recognizing Bengali Sign Language gesture images for digits only in real time. Bengali sign language images for digits are collected from different individuals and the CNN model is trained with these images after performing several pre-processing tasks i.e. resizing to a specific dimension, converting these RGB images to the gray scale images, finding the equivalent binary images and rotating the images into different degrees both in left and right direction. The experiment is conducted using two major techniques. Firstly, the model has been trained with the dataset containing the equivalent binary images of the row images collected directly from different individuals. Secondly, the dataset is enriched by rotating all the images into 3°, 6°, 9°, 12°and 15°in both left and right directions. After applying the rotation technique, the recognition accuracy is found to be increased significantly. The maximum recognition accuracy of the proposed CNN model with the dataset without image rotation technique is 94.17% whereas the recognition accuracy is 99.75% while including the rotated images in the dataset. Index Terms—Bengali Sign Language Recognition, Convolutional Neural Network, Image Recognition, Real Time Recognition