The core of visual processing is the identification and recognition of the objects relevant to cognitive behaviors. In natural environment, visual input is often comprised of highly complex 3-dimensional signals involving multiple visual objects. One critical determinant of object recognition is visual contour. Despite substantial insights on visual contour processing gained from previous findings, these studies have focused on limited aspects or particular stages of contour processing. So far, a systematic perspective of contour processing that comprehensively incorporates previous evidence is still missing. We therefore propose an integrated framework of the cognitive and neural mechanisms of contour processing, which involves three mutually interacting cognitive stages: contour detection, border ownership assignment and contour integration. For each stage, we provide an elaborated discussion of the neural properties, processing mechanism, and its functional interaction with the other stages by summarizing the relevant electrophysiological and human cognitive neuroscience evidence. Finally, we present the major challenges for further unraveling the mechanisms of visual contour processing.