Auditory Brainstem Response (ABR) is an evoked potential in the brainstem's neural centers in response to sound stimuli. Clinically, characteristic waves, especially Wave V latency, extracted from ABR can objectively indicate auditory loss and diagnose diseases. Several methods have been developed for the extraction of characteristic waves. To ensure the effectiveness of the method, most of the methods are time-consuming and rely on the heavy workloads of clinicians. To reduce the workload of clinicians, automated extraction methods have been developed. However, the above methods also have limitations. This study introduces a novel deep learning network for automatic extraction of Wave V latency, named ABR-Attention. ABR-Attention model includes a self-attention module, first and second-derivative attention module, and regressor module. Experiments are conducted on the accuracy with 10-fold cross-validation, the effects on different sound pressure levels (SPLs), the effects of different error scales and the effects of ablation. ABR-Attention shows efficacy in extracting Wave V latency of ABR, with an overall accuracy of 96.76 ± 0.41 % and an error scale of 0.1ms, and provides a new solution for objective localization of ABR characteristic waves.