In order to investigate the damage to microstructure and some other micromechanical responses during a fatigue test on asphalt mixture, Particle Flow Code (PFC) was used to reconstruct a two-dimensional discrete element model of asphalt mixture, based on computed tomography (CT) images and image-processing techniques. The indirect tensile fatigue test of asphalt mixture was simulated with this image-based microstructural model, and verified in the laboratory. It was found that there were four stages during the fatigue failure: no crack, crack initiation, crack developing, and interconnected crack. Cracks mainly developed between the aggregate and asphalt mortar, near the loading axis. The corresponding stages of failure, the developing trend and the distribution characteristics of the cracks matched well with those in the laboratory test. Furthermore, the trends of both the time-load curve and time-displacement curve from the simulation test were also consistent with those from the experimental test. In short, the distribution characteristics of cracks and internal forces of asphalt mixture show that it is feasible to simulate the fatigue performance of the asphalt mixture by a discrete element method (DEM).