Thousands of people are dying every year due to road accidents; in fact 23% of world fatal accidents are pedestrians related, where 40% of them occur in Africa as reported by the World Health Organisation (WHO). Predicting the walking direction of a pedestrian could help to avoid an eventual accident. Existing studies can not handle pose and orientation transformations of the input object contrary to our proposed method. This paper describes a novel approach to determine the pedestrian orientation using Capsule Networks (CapsNet) based scheme. CapsNet are a new deep learning architecture that overcome some limitations of the existing studies, they are group of neurons invariant to rotation and affine transformations, which represent a specific interest to this work. Capsule Networks predicts the walking directions of pedestrians to prevent such mortal accidents, using four main walking directions (front, back, left and right).For this purpose, a new pedestrians dataset gathered from the most popular cities in Morocco is collected to be studied and used as a proof of the proposed approach. To enhance this proposed approach, we evaluated it using Daimler dataset and compared it to Convolutional Neural Networks (CNN) architectures. Experimental results reveal that the performance of the proposed approach reaches an accuracy of 97.60% on daimler dataset and 73.64% on our Moroccan collected dataset.