A scale adaptive visual tracking algorithm based on the latent support vector machine (SVM) is proposed. The location of the object to be tracked is predicted by scanning all possible candidate locations and the scale is treated as a latent variable. With the predicted location, the latent SVM is optimised by a coordinate descent approach that optimises the latent variable and SVM parameters in an iterative manner. The separation of location and scale searching makes the tracker less likely to drift. Experimental results on test video sequences demonstrate that the proposed approach shows better accuracy than several state‐of‐the‐art visual tracking algorithms.