The incorporation of recycled concrete aggregate (RCA) in permeable asphalt mixtures (PAMs) is an efficient method of utilizing construction demolished waste. It not only conforms to the trend of building sponge cities, but also alleviates the problem of overexploitation of natural aggregate resources. As the performance of PAM containing recycled aggregate is not comparable to natural aggregate, modification treatments and the addition of hybrid fibers are adopted as two enhancement methods to improve the performance of PAM with RAC in this study. It is found that replacing natural aggregate with recycled aggregate increases the optimum asphalt content (OAC) but decreases the residual stability. The OAC is increased by 45% when the RCA ratio is 100%, whereas applying silicone resin can give a 16.2% decrease in the OAC. Enhancing RCA with silicone resin can increase the water stability to be comparable with natural aggregate. Moreover, with modification treatment using calcium hydroxide solution, the mechanical strength of PAM is enhanced to even higher than that of natural coarse aggregate mixture alone. Improvements in both mechanical strength and water stability are also achieved by strengthening recycled aggregate with cement slurry, although the performance is less effective than using silicone resin. With the increase in the content of RCA, the permeability coefficients of PAM first decrease and then exhibit an increasing trend. The results indicate that the PAM with RCA and modification treatments can perform satisfactorily as a pavement material in practice. Applying probable modification, PAM incorporating RCA meets the criteria for use in concrete pavement applications.