We present effective field theories for dipole symmetric topological matters that can be described by the Chern-Simons theory. Unlike most studies using higher-rank gauge theory, we develop a framework with both U(1)U(1) and dipole gauge fields. As a result, only the highest multipole symmetry can support the ’t Hooft anomaly. We show that with appropriate point group symmetries, the dipolar Chern-Simons theory can exist in any dimension and, moreover, the bulk-edge correspondence can depend on the boundary. As two applications, we draw an analogy between the dipole anomaly and the torsional anomaly and generalize particle-vortex duality to dipole phase transitions. All of the above are in the flat spacetime limit, but our framework is able to systematically couple dipole symmetry to curved spacetime. Based on that, we give a proposal about anomalous dipole hydrodynamics. Moreover, we show that the fracton-elasticity duality arises naturally from a non-abelian Chern-Simons theory in 3D.