We study the Hall response of topologically trivial mobile impurities (Fermi polarons) interacting weakly with majority fermions forming a Chern-insulator background. This setting involves a rich interplay between the genuine many-body character of the polaron problem and the topological nature of the surrounding cloud. When the majority fermions are accelerated by an external field, a transverse impurity current can be induced. To quantify this polaronic Hall effect, we compute the drag transconductivity, employing controlled diagrammatic perturbation theory in the impurity-fermion interaction. We show that the impurity Hall drag is not simply proportional to the Chern number characterizing the topological transport of the insulator on its own—it also depends continuously on particle-hole breaking terms, to which the Chern number is insensitive. However, when the insulator is tuned across a topological phase transition, a sharp jump of the impurity Hall drag results, for which we derive an analytical expression. We describe how to experimentally detect the polaronic Hall drag and its characteristic jump, setting the emphasis on the circular dichroism displayed by the impurity's absorption rate.