In recent years, visual simultaneous localization and mapping (SLAM) is widely used in robotic applications. Feature tracking is a fundamental problem in visual SLAM. Kanade-Lucas Tomasi (KLT) feature tracker is the most popular intensity-based feature tracking algorithm for its fast speed and easiness of use. However, it is vulnerable to large optical flow and accumulates error over time. To overcome the drawbacks, we propose a novel inertial-aided multi-reference and multi-level patch based feature tracking approach called IMRL feature tracker. A probabilistic approach is used to estimate the feature’s depth, and it is combined with the inertial measurements to provide a good initial feature position, which improves the robustness of our feature tracker to both fast camera rotation and translation. Furthermore, we propose a novel multi-reference and multi-level patch (MRL) based feature alignment method to improve the tracking accuracy. Thorough experiments were carried on open source datasets EuRoC and KITTI. The results show that comparing to the original KLT feature tracker, the proposed IMRL feature tracker achieves better robustness and accuracy with lower computational cost.