Colorectal cancer(CRC) or colon cancer, is fatal cancer seen in males and females. Colorectal polyps usually develop on the mucosal layer of the colon or rectal part of the large intestine. They may later turn malignant and become cancerous. Diagnosis of colorectal polyps in the initial stages is a key factor in reducing the mortality rate due to CRC. Colonoscopy is considered the golden standard in CRC detection. Automation of polyp detection, localization and segmentation in the screening stage can help the clinicians to a great extent. However, detection, localization and segmentation of polyps of various morphological structures and textures have been proved to be very challenging. Deep neural networks (DNNs) have emerged as a powerful subset of machine learning and recorded a tremendous boost in many visual recognition tasks including medical imaging. Deep learning models often need an immense number of annotated images, which is difficult to collect in the medical domain and these models are computationally expensive and memory intensive. Hence a lot of works are going on to have model compression and acceleration in deep neural networks without significantly decreasing the performance. This work suggests a lightweight deep learning model rooted on auto-encoder decoder architecture for the segmentation of colorectal polyps of various morphological structures and textures. This model can be trained at full length from a considerably less number of images and shows par performance in terms of essential metrics used in semantic segmentation.