In this study, we develop an innovative method that assists computer-aided diagnosis in the determination process of the exact location of the femoral neck junction in plain radiographs. Our algorithm consists of two phases, i.e., coarse prediction and fine matching, which are implemented by supervised deep learning method and unsupervised clustering, respectively. In coarse prediction, standard masks are first produced by a specialist and trained in our proposed feature propagation network (FPU-Net) with supervised learning on the femoral neck dataset. In fine matching, the standard masks are first classified into different categories using our proposed three parameters with unsupervised learning. The predicted mask from FPU-Net is matched with each category of standard masks by calculating the values of intersection of union (IOU), and finally the predicted mask is substituted by the standard mask with the largest IOU value. A total of 4320 femoral neck parts in anterior–posterior (AP) pelvis radiographs collected from China Medical University Hospital database were used to test our method. Simulation results show that, on the one hand, compared with other segmentation methods, the method proposed in this paper has a larger IOU value and better suppression of noise outside the region of interest; on the other hand, the introduction of unsupervised learning for fine matching can help in the accurate localization segmentation of femoral neck images. Accurate femoral neck segmentation can assist surgeons to diagnose and reduce the misdiagnosis rate and burden.