Abstract This paper presents an optimal robust control method for trajectory tracking of a X-rudder autonomous underwater vehicle (AUV) subjects to velocity sensor failures and uncertainties. Two reduced-order extended state observers (ESOs) are designed to estimate the surge and heave velocities, and the estimated values are used to replace all the linear velocity-related parameters in controller design, which helps releasing the requirements of linear velocities measurement and makes the controller robust against linear velocity sensor failures. In kinematics control loop, line-of-sight (LOS) guidance law and Lyapunov-based control are employed, and the unknown attack angle is calculated based on the estimated linear velocities. In dynamics control loop, a robust disturbance rejection control law is constructed using disturbance observers and modified terminal sliding mode control. Moreover, a multi-objective optimization method is proposed to achieve X-rudder allocation, which is not only energy efficient but also robust against rudder failures, and helps tackling the rudder input saturation problem at the same time. Finally, comparative numerical simulations are provided to demonstrate the robustness and effectiveness of the proposed approach.