Since synthetic aperture radar (SAR) images have complex noise and have no clean reference images, SAR image denoising is very challenging. With the development of deep learning, several denoising algorithms based on deep learning are proposed to achieve a better SAR image denoising effect. However, most networks are prone to gradient disappearance and explosion in the training process. The deep network model will produce an excessive amount of computation. The denoising time is also too long. Since most of the denoising algorithms based on deep learning use simulated images for model training, it is difficult to effectively suppress speckle noise in the real SAR image while a balance between denoising and detail preservation cannot be achieved. To address the mentioned problems, we propose a novel feature refinement attention network named FRANet. In FRANet, a feature refinement network is first used to refine the input noise image to extract more useful features while accelerating network training. Second, a feature attention encoder–decoder network is constructed for deep feature extraction. This network uses an asymmetric encoder–decoder structure to expand the receptive field, which can improve the information extraction ability and reduce the number of parameters effectively. Finally, the final denoised SAR image is obtained by global residual learning. Compared with other denoising algorithms, the proposed algorithm can achieve better results in denoising performance and running time.