Abstract It is an important step for the precision operation of the litchi picking robot to accurately detect litchi fruits and fruiting stems in the natural environment. At present, the visual detection algorithms of litchi fruits and fruiting stems in the natural environment arestill limited by poor accuracy and robustness. This paper proposes a method to detect litchi fruits and fruiting stems at nighttime environment. In this paper, the litchi fruits in the nighttime natural environment are detected based on YOLOv3, then the Regions of Interest (RoI) of the fruiting stems are determined according to the Bounding Boxes of the litchi fruits. Finally fruiting stem is segmented one by one based on U-Net to achieve the detection for nighttime litchi fruits and fruiting stems. Moreover, we design an experiment to evaluate the effects of detecting nighttime litchi fruits and fruiting stems under different illuminations and different cluster number of litchi fruits. The experiment demonstrates that the Average Precision (AP) of the litchi fruits detection model is 96.78%, 99.57% and 89.30% under the high-brightness, the normal brightness and the low-brightness, respectively. Correspondingly, the Mean Intersection over Union (MIoU) of the fruiting stems segmentation model is 79.00%, 84.33% and 78.60% respectively. In addition, the litchi fruits detection model obtains the AP of 100% and 96.52% with single-cluster litchi fruit and multiple-cluster litchi fruits respectively. Therefore, the method to detect nighttime litchi fruits and the fruiting stems based on deep learning shows high precision and robustness at nighttime natural environment and under multiple conditions, which provides technical support for the practical application of the litchi picking robots.