During the study of biological viruses, a large number of fluorescent particle images are photographed by ultra-microscopes in order to observe the motion and variation of viruses. However, due to diffraction effects, the luminance values of these fluorescent images are distributed in the form of the Point Spread Function. Considering the popularity and robustness of the Levenberg-Marquardt Algorithm (LMA), in this paper the LMA is adopted to achieve Gaussian fitting. Because plenty of matrix operations are involved in this algorithm, it is appropriate to take advantage of GPU to deal with this parallel problem. Plus, the increase of the image dataset brings us much trouble, such as the growth of total running time and out-of-memory problem. Therefore, we also achieve quick Gaussian fitting on multiple GPUs. Test and analysis results show that near proportional growth of computing speed can be reached as the number of GPUs increases. And this work also has shown in using different quantity of GPUs, all input image data sizes can obtain some speedup and larger data sizes can obtain more efficient speedup.