With the popularity of mobile devices and location-based social network, understanding and modelling the human mobility becomes an important topic in the field of ubiquitous computing. With the model developing from personal models with own information to the joint models with population information, the prediction performance of proposed models become better and better. Meanwhile, the privacy issues of these models come into the view of community and the public: collecting and uploading private data to the centralized server without enough regulation. In this paper, we propose PMF, a privacy-preserving mobility prediction framework via federated learning, to solve this problem without significantly sacrificing the prediction performance. In our framework, based on the deep learning mobility model, no private data is uploaded into the centralized server and the only uploaded thing is the updated model parameters which are difficult to crack and thus more secure. Furthermore, we design a group optimization method for the training on local devices to achieve better trade-off between performance and privacy. Finally, we propose a fine-tuned personal adaptor for personal modelling to further improve the prediction performance. We conduct extensive experiments on three real-life mobility datasets to demonstrate the superiority and effectiveness of our methods in privacy protection settings.