Background: Prostate cancer is one of the most prevalent cancers in the male population. To determine the aggressiveness of suspected lesions precisely, predictive models are increasingly being developed using quantitative MRI measurements, and particularly the ADC value. This study aimed to determine whether ADC values could be used to establish the aggressiveness of prostate cancer. Methods: A retrospective single-center study included 398 patients with prostate cancer who underwent a multiparametric MRI prior to radical prostatectomy. DWI ADC values were measured (µm2/s) using b values of 50 and 1000. The dominant lesion best visualized on MRI was analyzed. The ADC values of the index lesion and reference tissue were compared to tumor aggressivity according to the Gleason grade groups based on radical prostatectomy results. Statistical analysis was performed using the Mann–Whitney U test, Kruskal–Wallis H test, Spearman’s rank correlation, and ROC curves. Results: A very strong negative correlation (rs = −0.846, p < 0.001) between ADC and GS was found. ROC analysis revealed an AUC of 0.958 and an ADC threshold value of 758 µm2/s in clinically significant prostate cancer diagnoses using the absolute ADC value, with no advantage of using the ADC ratio over the absolute ADC value being identified. Conclusion: DWI ADC values and the calculated ADC ratio have a significant inverse correlation with GS. The findings indicate a strong capability in determining prostate cancer aggressiveness, as well as the possibility of assisting with assigning PI-RADS categories using ADC as quantitative metrics.