Continuous symmetries lead to universal slow relaxation of correlation functions in quantum many-body systems. In this work, we study how local symmetry-breaking impurities affect the dynamics of these correlation functions using Brownian quantum circuits, which we expect to apply to generic nonintegrable systems with the same symmetries. While explicitly breaking the symmetry is generally expected to lead to eventual restoration of full ergodicity, we find that approximately conserved quantities that survive under such circumstances can still induce slow relaxation. This can be understood using a super-Hamiltonian formulation, where low-lying excitations determine the late-time dynamics and exact ground states correspond to conserved quantities. We show that in one dimension, symmetry-breaking impurities modify diffusive and subdiffusive behaviors associated with U(1) and dipole conservation at late times, e.g., by increasing power-law decay exponents of the decay of autocorrelation functions. This stems from the fact that for these symmetries, impurities are relevant in the renormalization-group sense, e.g., bulk impurities effectively disconnect the system, completely modifying both temporal and spatial correlations. On the other hand, for an impurity that disrupts strong Hilbert space fragmentation, the super-Hamiltonian only acquires an exponentially small gap, leading to prethermal plateaus in autocorrelation functions which extend for times that scale exponentially with the distance to the impurity. Overall, our approach systematically characterizes how symmetry-breaking impurities affect relaxation dynamics in symmetric systems.