In this paper, we propose a method for point of interest (POI) recommendation by extracting the multi-tag influence and modeling the geographical influence. First of all, we extract a user-tag matrix from the initial user-POI rating matrix by analyzing the relations between POI and the related bag of tags. Secondly, we use the probabilistic factor model to predict the missing data of the extracted matrix. Thirdly, an effective method to model the geographical influence is proposed by considering the location of user and POI and the related region center. Finally, the multi-tag and geographical influence are fused in the process of making prediction of missing value of every POI. Then we will get a great result for POI recommendation. The experimental analysis on the large dataset Yelp demonstrates that our method outperform the state-of-art methods.