The collective plasmonic modes of a metal comprise a pattern of charge density and tightly-bound electric fields that oscillate in lock-step to yield enhanced light-matter interaction. Here we show that metals with non-zero Hall conductivity host plasmons with a fine internal structure: they are characterized by a current density configuration that sharply departs from that of ordinary zero Hall conductivity metals. This non-trivial internal structure dramatically enriches the dynamics of plasmon propagation, enabling plasmon wavepackets to acquire geometric phases as they scatter. Strikingly, at boundaries these phases accumulate allowing plasmon waves that reflect off to experience a non-reciprocal parallel shift along the boundary displacing the incident and reflected plasmon trajectories. This plasmon Hall shift, tunable by Hall conductivity as well as plasmon wavelength, displays the chirality of the plasmon's current distribution and can be probed by near-field photonics techniques. Anomalous plasmon dynamics provide a real-space window into the inner structure of plasmon bands, as well as new means for directing plasmonic beams.