The fracture characteristics of semirigid asphalt concrete (SRAC) prepared from a highly porous asphalt mixture (matrix) with 18% to 22% air voids and cement paste were evaluated. Two asphalt binders and several polymer-modified cement pastes that were grouted into the porous matrix were used. The SRAC beam specimen was prepared, and an initial notch was made by cutting with a saw at the bottom center of each beam for the static three-point bending (3PB) test at low temperatures (10°C, 0°C, and −10°C). Fracture toughness (KIC) was calculated with 3PB test data to examine the fracture resistance of the SRAC. The two SRAC binders, a normal asphalt (PG 64-22) and a polymer-modified asphalt (PG 76-22), exhibited significant differences in KIC values within the temperature and the paste type. The SRAC using PG 76-22 showed KIC values 17% higher than the SRAC using PG 64-22 at −10°C. Therefore, the SRAC using PG 76-22 binder is expected to have better low-temperature fracture resistance than the SRAC using PG 64-22 binder in the region where the low temperature commonly reaches −10°C.