Background Prostate cancer is one of the most frequent malignancies in the world, with significant morbidity and death rates. Identifying novel biomarkers is critical to reducing morbidity and mortality from the illness today. Although lipids have been linked to an increased risk of prostate cancer, it is unclear if the non-high-density lipoprotein cholesterol (NHDL-C) to high-density lipoprotein cholesterol (HDL-C) ratio (NHHR) is related to prostate cancer. Therefore, we assessed the relationship between NHHR and prostate cancer risk. Methods This study comprised 1,718 individuals who attended the First Affiliated Hospital of Xinjiang Medical University between March 2020 and March 2024. A pathological examination of a prostate puncture biopsy was utilized to confirm the diagnosis of prostate cancer. The study collected information on participants’ clinical and laboratory examinations, used propensity score matching to eliminate potential confounders, and analyzed the relationship between NHHR and prostate cancer, using logistic regression models combined with restricted cubic spline (RCS) and receiver operating characteristic (ROC) curve. Furthermore, sensitivity analyses were undertaken to assess the robustness of the results. Results (1) There were statistically significant differences in neutrophils, erythrocytes, triglycerides, alkaline phosphatase, and NHHR between the two groups; (2) after adjusting for all covariates, logistic regression revealed a negative association between NHHR and prostate cancer (0.91 (0.83∼0.99), P = 0.028). Participants in the second quartile had a significantly lower risk of prostate cancer after comprehensive adjustment (0.69 (0.50∼0.96), P = 0.026); (3) the restrictive cubic spline curve shows a non-linear relationship between NHHR and prostate cancer risk, with a turning point of 1.824; (4) the results of the ROC curve showed that the NHHR had an area under the curve (AUC) of 0.73; the results of the decision curve suggested that the model was able to provide a high benefit value for patients, with a threshold range of approximately 0.01–0.98 and a maximum net benefit of 0.52, and the calibration curve demonstrated that the model was accurate and reliable. Conclusions There is a non-linear dose-response relationship between NHHR and prostate cancer risk, which may be associated with a reduced risk of prostate cancer. The finding can be used to detect high-risk groups and prevent prostate cancer.