In this paper we present the design of CHRoME, a downlink multi-user beamforming (MUBF) protocol that addresses the inherent sensitivity of multi-stream systems to mobility, inter-stream interference, and imperfect channel state information. Our contributions are: (i) a technique for accurately selecting the downlink bit rate in the presence of inter-stream interference via a custom multi-user probe and feedback signal, immediately preceding data transmission, and (ii) a fast retransmission scheme that exploits liberated antenna resources to increase the expected per-user signal-to-interference-plus-noise ratio (SINR) and retransmit without having to re-sound the channel. We implement each mechanism and evaluate via a combination of indoor over-the-air experiments and trace-driven emulation. We demonstrate that CHRoME increases the resilience of MUBF systems to inter-stream interference and achieves multi-fold throughput gains compared to IEEE 802.11ac.