One of the most challenging problems faced by developers and engineers is the ability to hypothesize human behaviour. The study of user behaviour has always been an integral part of security analysis and threat detection. However, it takes on more incentive in the context of autonomous vehicles. Given such a dynamic context, quick intuitions may prove to be very misleading; resulting in misconceptions about the technology, its impact, and the nature of innovation. Considering the potential magnitude of the ramification from this technology, it is advisable to maintain caution and design a solution which accounts for all possible vulnerabilities. This works presents a novel architecture towards securing intelligent vehicles from physical roadside compromise. It has been designed with the purpose of questioning everything the vehicle is seeing, and verifying whether there is any legitimacy involved in what it's registering as being observed. With this work, an evaluation of a classification system is presented for scenarios where a vehicle maybe susceptible to physical damage. In the present study, we experimentally investigate the possibility of masquerading fake road side units (such as road signs) to override typical driving behaviour. Driving data was logged for participants who drove a vehicle in a fixed loop measuring approximately ∼1.4 miles in the city of Cincinnati. The collected data was then split into testing and training samples; wherein classifiers were trained and the model evaluated against the same. Our results indicate that by using a 80–20 split, 96% of masquerading attacks could be identified accurately.