Despite being an essential prerequisite at the basis of many applications ranging from surveillance to computational photography, the problem of initial background estimation seems to be marginally investigated. In this paper, we present a reliable CNN-based solution to estimate the initial background (BG) of a scene, given not necessarily a whole sequence but just a small set of frames containing foreground objects (FG). The proposed solution is based on a convolutional neural network (CNN) which is trained to estimate BG patches followed by an aggregation/post-processing step of these estimates to form the final BG image. The accuracy of our approach is evaluated visually and numerically using different metrics on the proposed sequences by the scene background modeling contest 2016 (SBMC2016). It demonstrates robustness against very challenging scenarios under extreme conditions such as very short or long sequences, dynamic BG, illumination changes and intermittent object motion. As most deep learning solutions, our approach achieves promising results.