In this study, a digital twin framework is proposed for damage detection in a civil structure, which consists of a finite element model, neural networks, model updating methods, and signal processing. To verify the proposed framework, we present a case study of slab structure using deflection measurement as input data. The dynamic characteristics of the physical model are used to calibrate the digital twin model. Damage scenarios are created on the digital twin model. The defection of the damaged slab under static loads is analyzed with two-dimensional discrete wavelet theory (DWT), whereas the diagonal wavelets are used to extract images data set used to train the convolutional neural network (CNN). MobileNetV2 uses transfer learning can reduce the number of trained parameters and hence perform fast convergence. The proposed method gives high accuracy about detection of low-severity damage having the severity less than 10%. There is more than 80% accuracy for predicting the damaged location and its severity. The success of using MobileNetV2 and transfer learning helps to improve the methods further on mobile devices and the potential for more applications. Moreover, the proposed framework does not require the data of the intact structures, leading to much wider applications.