In the context of traditional energy shortage and climate warming, the development of solar energy, as a clean and renewable energy, is crucial. As an effective way to utilize solar energy resources, photovoltaic (PV) power generation technology has been widely used around the world. Using remote sensing images to extract PV panel information, including location, area, has a positive effect on understanding the development status, planning and construction of regional PV new energy. In this study, a semantic segmentation network called HCT-Net, combined with the hybrid neural networks and the swarm intelligence optimization algorithms, is designed to segment solar PV panels from remote sensing images automatically and accurately. To address the problem of inconsistent segmentation within PV regions, a hybrid encoder, which combines a convolutional neural network and a Transformer, is designed to extract local features with rich detail information and global features with global context dependencies, resulting in enhanced feature representations. The foreground relation module is designed to solve the problem of mis-segmentation of the background into PVs. This module strengthens the model’s focus on the target object and suppresses the feature representations of non-PVs by explicitly learning the similarity relationship between the global PV feature representation and the feature representations of other objects, and by adaptively assigning weights according to the similarity. The swarm intelligence optimization algorithm is applied to adjust the learning rate and the balance coefficient of the composite loss function of HCT-Net during training. Experimental results show that compared with the current mainstream semantic segmentation network, the method in this study effectively alleviates the problem of inconsistent segmentation within PV regions and mis-segmentation and has advantages in the complete and accurate extraction of PV panels.