Usage-based Insurance (UBI) is regarded as a promising way to offer more accurate insurance premium by profiling driving behaviors. Compared with traditional insurance which considers drivers' history of accidents, traffic violations and etc, UBI focuses on driving data and can give a more reasonable insurance premium based on the current driving behaviors. Insurers use sensors in smartphone or vehicle to collect driving data (e.g. mileage, speed, hark braking) and compute a risk score based on these data to recalculate insurance premium. Many insurance programs, which are advertised as being privacy-preserving, do not directly use the GPS-based tracking, but it is not enough to protect driver's location privacy. In real world, many environment factors such as real-time traffic and traffic regulations can influence driving speed. These factors provide the side-channel information about the driving route, which can be exploited to infer the vehicle's trace. Based on the observation, we propose a novel speed based trajectory inference algorithm which can track drivers only with the speed data and original location. We implement the attack on a public dataset in New Jersey. The evaluation results show that the attacker can recover the route with a high successful rate.