In this paper, we address the problem of visual tracking in videos without using a pre-learned model of the object. This type of model-free tracking is a hard problem because of limited information about the object, abrupt object motion, and shape deformation. We propose to integrate an object-agnostic prior, called objectness, which is designed to measure the likelihood of a given location to contain an object of any type, into structured tracking framework. Our objectness prior is based on image segmentation and edges; thus, it does not require training data. By extending a structured tracker with the prior, we introduce a new tracker which we call ObjStruck. We extensively evaluate our tracker on publicly available datasets and show that objectness prior improves tracking accuracy.