Tumor segmentation in Magnetic Resonance Imaging (MRI) plays an important role in diagnosis, treatment planning, and disease surveillance. But still there are many hurdles in the process because of low contrast tissues, unclear boundaries and high morphology variations. In this paper, we propose Contour-Aware U-Net (CAU-Net), which uses explicit contour refinement techniques along with multi-level feature fusion. Our framework includes three main components that are as follows: (1) Contour-Aware Decoder with Attention Fusion blocks for contour enhancement, (2) adversarial learning constraint for anatomically plausible results, and (3) combined hybrid loss function using cross entropy loss, dice loss, and sub-differentiable Hausdorff loss. Extensive experiments on tumor datasets have proven that our proposed approach outperforms existing approaches in terms of accuracy by producing Dice Similarity Coefficient score of 0.92 and reducing Hausdorff Distance by 38%. Our model performs exceptionally well in terms of boundary delineation that was the crucial requirement in clinical practice.