Asphalt fine aggregate matrix (FAM) is a predominant component directly related to field performances of hot asphalt mix (HMA), it is necessary to investigate material properties of FAM. Prior to preparing FAM specimens, the asphalt content was calculated by keeping the filler–bitumen (FB) ratio the same as in the corresponding HMA. A non-destructive fabrication method instead of coring and cutting methods was developed to compact FAM cylinders, and the joint base was designed to be concentric with the loading axis of testing system. Rheological responses of FAM were studied using the dynamic shear rheometer (DSR). Two repeated tests prove that the FAM compactor and the jointed base meet the requirement of data validation. Results show that rheological performances of FAM are significantly affected by asphalt content, gradation, air void content, and testing frequency. Air void is concluded to be the decisive factor which influences the stability of FAM, and the fiber is demonstrated to play a role on enhancing the flow resistance of FAM-F even though with the richest asphalt content.