Fabric defect detection plays a vital role in ensuring the production quality of the textile manufacturing industry. However, in practice, there are relatively few manually annotated defective samples, and considering both performance and parameter quantity, there is still room for optimization in the architecture of detection networks. Therefore, this paper proposes a textile defect detection method called MED-AGNet. Firstly, based on the diffusion model, a mask-embedding data augmentation method, MEDiffusion, is proposed. During the training process, a conditional term (M) that represents the shape of the defect is added, and through supervised learning, the generative model learns the correlation between the background and defects. In the generation stage, it samples from a normal distribution and relies on M guidance to gradually generate corresponding defective textiles, thereby expanding the original sample set. An attention-guided network (AGNet) is a network that utilizes attention to guide information across different scales. Its feature extraction module employs a dual-branch information residual unit (DIRU) as a substitute for the conventional convolution block, which combines the feature extraction capabilities of global pooling and max pooling, reducing the number of parameters while also achieving a certain improvement in detection results. In the feature fusion stage, it utilizes the attention-guided fusion module (AGFM), which can allow the attention information of high-level semantics to guide the low-level semantics, and simultaneously, adds a high-level semantic residual attention module (HSRA) to enhance the perception of defect shapes and improve detection effectiveness. Ultimately, AGNet’s true positive rate (TPR), positive predictive value (PPV), and f-measure exceed those of the state-of-the-art (SOTA) algorithms by 1.88%, 0.05%, and 0.77%, respectively, and with a consistent model architecture, its parameter quantity is reduced by 56%.