Gravity surveys in regional geophysical research can be used to estimate the depth of the sediment-basement interface. In this paper, we investigate a novel method using the convolutional neural network (CNN) for depth-to-basement inversion directly from gravity data. Based on the Random-Midpoint-Displacement method (RMD) and the features of the observed gravity data, we can generate a large set of realistic sediment-basement interface models. This new method for model generation can significantly reduce the size of the training data sets which is usually considerably large to train a pervasive network. The application on synthetic models shows that the developed CNN inversion is able to capture the detailed features of the sediment-basement interface for the complex geological model. However, so far, the training set obtained from the proposed method is still continuous and the CNN inversion still cannot effectively recover the models such as abrupt faults. We also successfully applied the developed method and workflow to a field study. The proposed approach opens a new window for estimating the physical contrast interfaces using potential field. The files provided contain the training data set used in the manuscript, as well as the Python scripts to train and predict the 3D basement reliefs.