Abstract In this paper we study the Hilbert–Schmidt norm of time-frequency localization operators $$L_{\Omega } :L^2(\mathbb {R}^d) \rightarrow L^2(\mathbb {R}^d)$$ L Ω : L 2 ( R d ) → L 2 ( R d ) , with Gaussian window, associated with a subset $$\Omega \subset \mathbb {R}^{2d}$$ Ω ⊂ R 2 d of finite measure. We prove, in particular, that the Hilbert–Schmidt norm of $$L_\Omega $$ L Ω is maximized, among all subsets $$\Omega $$ Ω of a given finite measure, when $$\Omega $$ Ω is a ball and that there are no other extremizers. Actually, the main result is a quantitative version of this estimate, with sharp exponent. A similar problem is addressed for wavelet localization operators, where rearrangements are understood in the hyperbolic setting.