The tracking method based on spatio-temporal context has achieved great effects in real-time and robustness, but its tracking results are not satisfying while facing with strong interference such as illumination variation, occlusion, in-plane rotation and fast motion. For solving these problems, we have proposed a real-time detection and selective update spatio-temporal context tracker. The introduced detect-or is based on keypoint matching and consistent voting, so that the improved method can effectively track occluded and in-plane rotating targets; the overlap rate and Hamming distance are introduced to judge the confidence of the output consequence of the detection and tracking modules, and then the best result is selected to update the tracker. This improvement effectively avoids the situation that the tracking template introduces error information due to fast motion, illumination variation, thereby improving the tracker's robustness and precision. A large number of experimental results on the OTB-50 datasets demonstrate that this method has better performance compared with some representative advanced tracking algorithms.