Methods based on generative adversarial network (GAN) have been widely used in infrared and visible images fusion. However, these methods cannot perceive the discriminative parts of an image. Therefore, we introduce a multigrained attention module into encoder–decoder network to fuse infrared and visible images (MgAN-Fuse). The infrared and visible images are encoded by two independent encoder networks due to their diverse modalities. Then, the results of the two encoders are concatenated to calculate the fused result by the decoder. To exploit the features of multiscale layers fully and force the model focus on the discriminative regions, we integrate attention modules into multiscale layers of the encoder to obtain multigrained attention maps, and then, the multigrained attention maps are concatenated with the corresponding multiscale features of the decoder network. Thus, the proposed method can preserve the foreground target information of the infrared image and capture the context information of the visible image. Furthermore, we design an additional feature loss in the training process to preserve the important features of the visible image, and a dual adversarial architecture is employed to help the model capture enough infrared intensity information and visible details simultaneously. The ablation studies illustrate the validity of the multigrained attention network and feature loss function. Extensive experiments on two infrared and visible image data sets demonstrate that the proposed MgAN-Fuse has a better performance than state-of-the-art methods.