Abstract An important aspect of Intelligent Public Transportation Systems (IPTS) is providing accurate travel time information. Knowing arrival times of public vehicles in advance can reduce waiting times of passengers and attract more people to take public transport. Existing approaches have two main limitations in the field of bus travel time prediction. First, influenced by increasingly complex real-time traffic factors and sparsity of real-time data, bus travel times can be difficult to predict accurately in modern cities. Second, bus dwelling and transit times are predominantly affected by different factors and hence have different patterns, but little research focuses on how to divide dwelling and transit areas and to build independent models for them. Consequently, we propose a novel segment-based approach to predict bus travel times using a combination of real-time taxi and bus datasets, that can automatically divide bus routes into dwelling and transit segments. Two models are built to predict them separately by incorporating different impact traffic factors. We evaluate our approach using real-world trajectory datasets, collected in Xi’an, China during June 2017. Compared to existing methods, the experimental results reveal that our approach improves the accuracy of bus travel time prediction, especially under abnormal traffic conditions.