In recent years, owing to the swift progression in sensor technology and the extensive utilization of remote sensing imagery, obtaining and using high-quality remote sensing images is increasingly important. Among them, it is necessary to address the classification problem of panchromatic remote sensing images and multispectral remote sensing images. In this field of research, cleverly eliminating modal differences, removing redundancy, and better integrating information has become a challenge. In this article, we propose a DBFR-AENet for the multisource remote sensing image classification task. First, the IFFS strategy aims to design different feature branches to pick up the advantageous features of multispectral and panchromatic images separately. It filters redundant information and obtains useful information with higher purity. Second, the Bi-VAE strategy aims to eliminate modal differences by constructing a low-dimensional shared space. The dual-source image is input into the encoder to obtain the latent encoding in the latent space. The goal of feature alignment can be achieved in the potential shared space. Then, perform feature fusion. Finally, classify the image after feature fusion.