Manipulated media is becoming a prominent threat due to the recent advances in realistic image and video synthesis techniques. There have been several attempts at detecting synthetically tampered media using machine learning classifiers. However, such classifiers do not generalize well to black-box image synthesis techniques and have been shown to be vulnerable to adversarial examples. To address these challenges, we introduce FaceSigns —a deep learning-based semi-fragile watermarking technique that allows media authentication by verifying an invisible secret message embedded in the image pixels. Instead of identifying and detecting manipulated media using visual artifacts, we propose to proactively embed a semi-fragile watermark into a real image or video so that we can prove its authenticity when needed. FaceSigns is designed to be fragile to malicious manipulations or tampering while being robust to benign operations such as image/video compression, scaling, saturation, contrast adjustments, and so forth. This allows images and videos shared over the internet to retain the verifiable watermark as long as a malicious modification technique is not applied. We demonstrate that our framework can embed a 128-bit secret as an imperceptible image watermark that can be recovered with a high bit recovery accuracy at several compression levels, while being non-recoverable when unseen malicious manipulations are applied. For a set of unseen benign and malicious manipulations studied in our work, our framework can reliably detect manipulated content with an AUC score of 0.996, which is significantly higher than prior image watermarking and steganography techniques.