It is challenging for textile manufacturers to automate inspection due to a pronounced imbalance in the class of samples, with the majority of samples being defect-free, and the complementary nature of local and global defects. In this study, FusionDefectNet is presented as a novel hybrid architecture. For extracting local features, Convolutional Neural Networks (CNNs) are used, and for understanding global context, Vision Transformers (ViTs) are employed. As part of the adaptive gating mechanism, the weights on branch contributions are modified based on the input attributes. The 68.7:1 class imbalance is addressed by integrating class-weighted loss with focal loss (γ=2). To gain insights into decision-making processes, use interpretable AI technologies like Grad-CAM and ViT attention visualizations. According to the TILDA dataset, we were 97.3% accurate and scored 96.9% higher than pure CNN solutions. When α is below 0.3, CNNs are preferred over other search methods for localized imperfections like holes and stains. However, for structural issues at global scale, it favors ViTs when α exceeds 0.7. According to Explainable AI, CNN identifies minor discrepancies, while ViT identifies patterns within the context that are more substantial.