This paper presents a novel vision based approach for detecting rows of crop in paddy field. The precise detection of crop row enables a farm-tractor to autonomously navigate the field for successful inter-row weeding. While prior works on crop row detection rely primarily on various image based features, a deep neural network based approach for learning semantic graphics to directly extract the crop rows from an input image is used in this work. A deep convolutional encoder decoder network is trained to detect the crop lines using semantic graphics. The detected crop lines are then used to derive control signal for steering the tractor autonomously in the field. The results demonstrate that the proposed method is able to detect the rows of paddy accurately and enable the tractor to navigate autonomously along the crop rows even with a simple proportional only controller.