Rainy degeneration damages an image’s visual effect and influences the performance of subsequent vision tasks. Various deep learning methods for single image deraining have been proposed, obtaining appropriate recovery results. Unfortunately, most existing methods ignore the interaction between rain-layer and rain-free components when extracting relevant features, leading to undesirable results. To break the above limitations, we propose a progressive hybrid-modulated network (PHMNet) for single image deraining based on the two-branch and coarse-to-fine framework. Specifically, a hybrid-modulated module (HMM) with a two-branch framework is proposed to blend and modulate the feature of rain-free layers and rain streaks. After cascading several HMMs in the coarsest reconstructed stage of the PHMNet, a multi-level refined module (MLRM) is adopted to refine the final deraining results in the refined reconstructed stage. By being trained using loss functions such as contrastive learning, the PHMNet can obtain satisfactory deraining results. Extended experiments on several datasets and downstream tasks demonstrate that our method performs favorably against state-of-the-art methods in quantitative evaluation and visual effects.