The use of waste tires to prepare rubberized asphalt has been a hot trend in recent years, and the characteristics of adhesion between rubberized asphalt and aggregates are important factors affecting the performance of asphalt pavement. However, there is a lack of uniform results on the adhesion characteristics of rubberized asphalt. Therefore, crumb-rubber-modified asphalt (CRMA) with 15%, 20%, and 25% rubber contents was prepared in this work, and the basic rheological parameters and cohesive energy of the rubberized asphalt were characterized by DSR. The adhesion properties between rubberized asphalt and aggregates were characterized based on macroscopic binder bond strength (BBS), surface free energy (SFE) theory, and nanoscale atomic force microscopy (AFM) tests. The results show that crumb rubber (CR) can improve the high-temperature elastic properties of asphalt; secondly, CR can have a negative impact on the maximum tensile strength of asphalt and aggregates. CR can improve the SFE parameter of asphalt. The work of adhesion of rubberized asphalt and limestone is the highest, followed by basalt and, finally, granite. Finally, CR can cause the catanaphase in asphalt to gradually break down and become smaller, and the adhesion of rubberized asphalt can be reduced. Overall, CR can reduce the adhesion performance of asphalt, and this work provides a reference for the application of rubberized asphalt.