Point of interest (POI) recommendation, a service which can help people discover useful and interesting locations has emerged rapidly with the development of location-based social networks (LBSNs), like Foursquare, Gowalla and Wechat. The large number of check-in histories make it possible to mine the preference of each user and then to provide accurate personalized POI recommendation. In real-world applications, apart from check-in data, there are some other useful information available for making better POI recommendation, such as social relationship among users and geographical influence. In this paper, a new POI recommendation method called Social and Geographical Fusing Model (SGFM) is designed. The basic idea is summarized as follows. Firstly, the users' check-in records and social influence are integrated in a combinative model. Then the global user impact factors generated by the PageRank algorithm are used to improve the combinative model. Secondly, a geographical influence measurement is used to capture the users' physical check-in characters. Finally, the enhanced combinative model and geographical influence are combined together to form a new framework. Extensive experiments have been conducted on a famous dataset, namely Gowalla. The comparison results confirm that the proposed framework outperforms state-of-the-art POI recommendation methods significantly.