Due to the ever-growing demands for electronic chips in different sectors, semiconductor companies have been mandated to offshore their manufacturing processes. This unwanted matter has made security and trustworthiness of their fabricated chips concerning and has caused the creation of hardware attacks. In this condition, different entities in the semiconductor supply chain can act maliciously and execute an attack on the design computing layers, from devices to systems. Our attack is a hardware Trojan that is inserted during mask generation/fabrication in an untrusted foundry. The Trojan leaves a footprint in the fabrication through addition, deletion, or change of design cells. To tackle this problem, we propose EVHA (Explainable Vision System for Hardware Testing and Assurance) in this work, which can detect the smallest possible change to a design in a low-cost, accurate, and fast manner. The inputs to this system are scanning electron microscopy images acquired from the integrated circuits under examination. The system output is the determination of integrated circuit status in terms of having any defect and/or hardware Trojan through addition, deletion, or change in the design cells at the cell level. This article provides an overview on the design, development, implementation, and analysis of our defense system.