High-contrast imaging instruments are today primarily limited by non-common path aberrations appearing between the scientific and wavefront sensing arms. These aberrations can produce quasi-static speckles in the science images that are difficult to distinguish from exoplanet signatures. With the help of recent advances in deep learning, we have developed in previous works a method that implements convolutional neural networks (CNN) to estimate pupil-plane phase aberrations from point spread functions (PSF). Here we take it a step further by incorporating into the deep learning architecture the optical propagation occurring inside the instrument. The motivation behind this is to give a physical meaning to the models and to improve their robustness to various conditions. We explore how a variational autoencoder architecture that contains a differentiable optical simulator as the decoder can be used for that task. Because this unsupervised learning approach reconstructs the PSFs, it is not required to know the true wavefront aberrations in order to train the models, which is particularly promising for on-sky applications. We investigate different configurations of such a physics-based deep learning method and compare their performance to a standard CNN approach.