AbstractHybrid plasmonic devices involve a nanostructured metal supporting localized surface plasmons to amplify light–matter interaction, and a non‐plasmonic material to functionalize charge excitations. Application‐relevant epitaxial heterostructures, however, give rise to ballistic ultrafast dynamics that challenge the conventional semiclassical understanding of unidirectional nanometal‐to‐substrate energy transfer. Epitaxial Au nanoislands are studied on WSe2 with time‐ and angle‐resolved photoemission spectroscopy and femtosecond electron diffraction: this combination of techniques resolves material, energy, and momentum of charge‐carriers and phonons excited in the heterostructure. A strong non‐linear plasmon–exciton interaction that transfers the energy of sub‐bandgap photons very efficiently to the semiconductor is observed, leaving the metal cold until non‐radiative exciton recombination heats the nanoparticles on hundreds of femtoseconds timescales. The results resolve a multi‐directional energy exchange on timescales shorter than the electronic thermalization of the nanometal. Electron–phonon coupling and diffusive charge‐transfer determine the subsequent energy flow. This complex dynamics opens perspectives for optoelectronic and photocatalytic applications, while providing a constraining experimental testbed for state‐of‐the‐art modelling.