Incorporating optical and polarimetric synthetic-aperture radar (SAR) data to estimate impervious surface is useful but challenging due to their different geometric imaging mechanism and the high diversity of urban land covers. The recent development of deep convolutional networks (DCN) opens a promising opportunity by automatically extracting the deep features from both data sets. In this study, a small-patched DCN (SDCN) was designed to estimate the impervious surface from optical and SAR data. Benchmark methods, e.g., GoogLeNet, VGG16, ResNet50, and the support vector machine were employed for comparison. Two study sites in the most complex metropolitan of China, the Guangdong-Hong Kong-Macau Greater Bay Area, were selected to assess the proposed method. Experimental results indicated the effectiveness of proposed SDCN with a better accuracy outperforming other benchmark methods. Furthermore, we found that 60%–80% of training samples performed comparably with the whole training set, indicating that a large number of training samples may not be necessary in all cases, depending on the settings of some factors (e.g., number of epochs). Generally, SDCN appears more suitable than other methods in terms of combining the optical and SAR data and improved the accuracy of estimating impervious surface.