Human-robot co-manipulation of large but lightweight elements made by soft materials is a challenging operation that presents several relevant industrial applications. This paper proposes using a 3D camera to track the deformation of soft materials for human-robot co-manipulation. Thanks to a Convolutional Neural Network (CNN), the acquired depth image is processed to estimate the element deformation. The output of the CNN is the feedback for the robot controller to track a given set-point of deformation.