Traditional task-oriented learning for pre-stack AVA inversion typically adopts a single-network architecture to handle all elastic parameters. This practice introduces two inherent limitations. First, it struggles to accommodate spatial variations in lithofacies because a single global mapping cannot satisfy the distinct elastic-parameter distributions of different rock types. Second, its outputs may violate petrophysical trends since the elastic parameters are predicted independently. To relieve these limitation, we propose a Variational Bayesian Mixture of Implicitly Localized Experts (VB-MILE) network. We derive two geological priors from the latent facies distribution. One is a lithofacies-conditional elastic-parameter prior from a well-log Gaussian Mixture Model (GMM). This prior constrains each expert's output means and covariances to match the facies-specific statistics observed in well logs. The other is a spatial assignment prior obtained by aligning a low-frequency GMM with the well-log GMM. This prior initializes the gating network and regularizes its dynamic expert assignment via KL divergence. Both priors are integrated into a unified variational Bayesian framework, enabling a divide-and-conquer strategy. In this strategy, different experts specialize in different lithofacies, and the gating network assigns each spatial location to the appropriate expert. The network is trained semi-supervisedly using sparse well logs and abundant unlabeled data to improve generalization. Experiments on the Marmousi2 dataset and field data from the Tahe Oilfield (Xinjiang, China) demonstrate that the proposed method outperforms conventional data-driven approaches in both accuracy and geological plausibility.