Vehicle-to-everything (V2X) communication in the mmWave band is one way to achieve high data-rates for applications like infotainment, cooperative perception, and augmented reality assisted driving etc. MmWave communication relies on large antennas arrays, and configuring these arrays poses high training overhead. In this article, we motivate the use of infrastructure mounted sensors (which will be part of future smart cities) for mmWave communication. We provide numerical and measurement results to demonstrate that information from these infrastructure sensors reduces the mmWave array configuration overhead. Finally, we outline future research directions to help materialize the use of infrastructure sensors for mmWave communication.