PAC-Bayes bound provides a formal framework for deducing the tightest risk bounds of the classifiers. After formulating the concept space as a Reproducing Kernel Hilbert Space (RKHS), the Markov Chain Monte Carlo (MCMC) sampling algorithm for simulating posterior distributions of the concept space can realize the calculation of PAC-Bayes bound. A major issue is the computational complexity in geometric growth when the dimension of concept space increases. In this paper, we store a portion of the sampling data and calculate its variance, after which the variance minimization method is proposed to investigate the support vectors. Finally, we optimize the support vectors coupled with their weight vectors, and compare the PAC-Bayes bounds. The experimental results of our artificial data sets in low-dimensional spaces show that the optimization is reasonable and effective in practice.