Aiming at the problems of the existing coal gangue recognition methods based on image processing, such as the inaccurate positioning of coal gangue targets on the conveyor belt, and the influence of belt background and local surface of coal gangue on the recognition process, a coal gangue recognition method based on deep learning is proposed. This method uses the CornerNet Lite model to locate the coal gangue in the image. According to the CornerNet series key point pooling method, four vertices of the rectangle of the coal gangue position in the frame and a key point with significant gray change of the coal gangue itself are found. The image is divided into four blocks by using these points, and then the targets are classified respectively. Finally, the coal gangue recognition results of the whole image are obtained. This method improves the positioning of coal gangue on the belt, reduces the interference of conveyor belt background and the influence of local surface of coal gangue during target positioning, and then improves the accuracy of coal gangue identification. The experimental results show that this method can effectively improve the recognition accuracy of coal gangue, reaching 91.3%.