Vessel border detection in IVUS images is essential for coronary disease diagnosis. It helps to obtain the clinical indices on the inner vessel morphology to indicate the stenosis. However, the existing methods suffer the challenge of scale-dependent interference. Early methods usually rely on the hand-crafted features, thus not robust to this interference. The existing deep learning methods are also ineffective to solve this challenge, because these methods aggregate multi-scale features in the top-down way. This aggregation may bring in interference from the non-adjacent scale. Besides, they only combine the features in all scales, and thus may weaken their complementary information. We propose the scale mutualized perception to solve this challenge by considering the adjacent scales mutually to preserve their complementary information. First, the adjacent small scales contain certain semantics to locate different vessel tissues. Then, they can also perceive the global context to assist the representation of the local context in the adjacent large scale, and vice versa. It helps to distinguish the objects with similar local features. Second, the adjacent large scales provide detailed information to refine the vessel boundaries. The experiments show the effectiveness of our method in 153 IVUS sequences, and its superiority to ten state-of-the-art methods.