Description of Points of Interest (POIs) plays an important role to enhance the quality of many location-based services, such as displaying concentrated information of POIs for user-friendly experience and leading to successful POI recommendation. However, only a few popular POIs have enough description on the web. Collecting or writing high-quality descriptions for many unpopular or long-tail POIs remains a huge challenge for online map services, especially considering there are numerous new appeared POIs every day. Unlike existing studies about automatic product description generation, the POI description is quite diverse across different locations over a country, and requires high expert knowledge. To address this issue, we first study the POI description generation problem by proposing a novel model, named as Multi Mode Description Generator (MMDG), to automatically generate description based on POIs’ reviews and other features. To extract key information for POI description generation, MMDG is equipped with a multi-mode encoder and a transformer-based decoder. Besides user reviews, the multi-mode encoder also considers the category and spatial context information of target POIs, and integrate them with a fusion function. We have conducted an extensive experimental evaluation on a large-scale real-world dataset to demonstrate its effectiveness and superiority over state-of-the-art baselines in terms of various metrics.