We introduce a hybrid plasmonic-photonic device for on-chip all-optical switching and reading of ferrimagnet bits with perpendicular magnetic anisotropy in a racetrack spintronic memory, coupled onto an indium phosphide waveguide. The device comprises V-shaped gold plasmonic nanoantennas coupled with a photonic crystal cavity, which enables switching and reading of the magnetic state of nanoscale bits by enhancing the absorbed energy density and polar magneto-optical Kerr effect locally. Using a finite-difference time-domain method, we show that our device can switch and read targeted bits down to 100 nm in the presence of oppositely magnetized background regions in the racetrack with widths up to 120 nm, clearly outperforming a bare photonic waveguide. Our hybrid device provides the missing link between integrated photonics and nanoscale spintronics by tackling the challenges of nonlinear absorption in the waveguide, weak magneto-optics, and size mismatch, leading to the development of ultrafast and energy-efficient advanced on-chip applications.