In practical engineering, obtaining labeled high-quality fault samples poses challenges. Conventional fault diagnosis methods based on deep learning struggle to discern the underlying causes of mechanical faults from a fine-grained perspective, due to the scarcity of annotated data. To tackle those issue, we propose a novel semi-supervised Gaussian Mixed Variational Autoencoder method, SeGMVAE, aimed at acquiring unsupervised representations that can be transferred across fine-grained fault diagnostic tasks, enabling the identification of previously unseen faults using only the small number of labeled samples. Initially, Gaussian mixtures are introduced as a multimodal prior distribution for the Variational Autoencoder. This distribution is dynamically optimized for each task through an expectation-maximization (EM) algorithm, constructing a latent representation of the bridging task and unlabeled samples. Subsequently, a set variational posterior approach is presented to encode each task sample into the latent space, facilitating meta-learning. Finally, semi-supervised EM integrates the posterior of labeled data by acquiring task-specific parameters for diagnosing unseen faults. Results from two experiments demonstrate that SeGMVAE excels in identifying new fine-grained faults and exhibits outstanding performance in cross-domain fault diagnosis across different machines. Our code is available at https://github.com/zhiqan/SeGMVAE.