High-frequency regions like edges compromise the image denoising performance. In traditional hand-crafted systems, image edges/textures were regularly used to restore the frequencies in these regions. However, this practice seems to be left forgotten in the deep learning era. In this paper, we revisit this idea of using the image gradient and introduce the GradNet. Our major contribution is fusing the image gradient in the network. Specifically, the image gradient is computed from the denoised network input and is subsequently concatenated with the feature maps extracted from the shallow layers. In this step, we argue that image gradient shares intrinsically similar nature with features from the shallow layers, and thus that our fusion strategy is superior. One minor contribution in this work is proposing a gradient consistency regularization, which enforces the gradient difference of the denoised image and the clean ground-truth to be minimized. Putting the two techniques together, the proposed GradNet allows us to achieve competitive denoising accuracy on three synthetic datasets and three real-world datasets. We show through ablation studies that the two techniques are indispensable. Moreover, we verify that our system is particularly capable of removing noise from textured regions.