With the increasing amount of cloud-based speech files, the privacy protection of speech files faces significant challenges. Therefore, integrity authentication of speech files is crucial, and there are two pivotal problems: (1) how to achieve fine-grained and highly accurate tampering detection and (2) how to perform high-quality tampering recovery under high tampering ratios. Tampering detection methods and tampering recovery methods of existing speech integrity authentication are mutually balanced, and most tampering recovery methods are carried out under ideal tampering conditions. This paper proposes an encrypted speech integrity authentication method that can simultaneously address both of problems, and its main contributions are as follows: (1) A 2-least significant bit (2-LSB)-based dual fragile watermarking method is proposed to improve tampering detection performance. This method constructs correlations between encrypted speech sampling points by 2-LSB-based fragile watermarking embedding method and achieves low-error tampering detection of tampered sampling points based on four types of fragile watermarkings. (2) A speech self-recovery model based on residual recovery-based linear interpolation (R2-Lerp) is proposed to achieve tampering recovery under high tampering ratios. This method constructs the model based on the correlation between tampered sampling points and their surrounding sampling points and refines the scenarios of the model according to the tampering situation of the sampling points, with experimental results showing that the recovered speech exhibits improved auditory quality and intelligibility. (3) A scrambling encryption algorithm based on the Lorenz mapping is proposed as the speech encryption method. This method scrambles the speech sampling points several times through 4-dimensional chaotic sequence, with experimental results showing that this method not only ensures security but also slightly improves the effect of tampering recovery.