This work presents our research aimed to develop a driver safety assistant system. The idea is to use in-vehicle camera with vision sensor to detect emotional distress level of the driver while driving. An algorithm to identify facial expression of the driver is developed using Python programming language. In addition, a prototype of facial expression detection along with a car parking assist system is developed by using an Arduino Uno ATMEL ATMEGA328 microcontroller interfaced with a webcam and a motor to demonstrate the concept. The camera mounted on the dashboard continuously monitors the driver’s face and captures the facial expressions. The facial expressions so captured help assess the driver’s (particularly, the truck driver) situation and identify it in terms of severe pain, headache, cardiac arrest, etc. Once the system identifies the situation, controller then assists in driving the car to the curb and bringing it to a complete stop. The facial expression identification algorithm uses the sensors (like speed, steering etc.) to detect the abnormality from the facial expression and subsequently alert the driver for 30 s. The system continuously checks the driver’s profile. If the driver is driving while continuously in pain for another 30 s, further assistance in terms of embedded vehicle controlling system will take charge of maneuvering the vehicle and slowly parking on the curb. While parking to the right side of the road, the vehicle control system will continuously check the traffic on the adjacent lane before parking slowly on the curb. Additionally, turn indicators will help maneuver the vehicle by keeping the turn signal on. To model the system, a network is trained using deep learning with 5000 data instances. The trained model is then validated by using real time images from camera to check whether the image of face confirms to the normal pattern or in pain.