Crack detection has drawn much attention in the last two decades, because of dramatic bloom in monitoring images and the urgent need of corresponding crack detection. However, recent methods have not taken advantage of structure information effectively, resulting in low accuracy when dealing with crack-like noises. In this paper, we propose a novel crack detection framework, which is able to identify cracks from noisy background. The main contributions of this paper are as follows: (1) giving a new edge-based crack detection framework to improve the detection performance; (2) proposing a novel mid-level feature, named Crack Token , which captures the local structure information of cracks; (3) introducing a new evaluation strategy for crack detection task, which provides a comprehensive system for approach evaluation and comparison in this area. In addition, we provide a novel definition of pavement crack and verify our framework and evaluation strategy in this real world application. Extensive experiments demonstrate the state-of-the-art results of the proposed framework.