Sidewall emission of a micro-scale light emitting diode (micro-LED) improves the light extraction efficiency, but it causes mismatched angular distributions between AlGaInP-based red micro-LED and InGaN-based blue/green counterparts due to material difference. As a result, color shift of RGB micro-LED displays may become visually noticeable. To address this issue, we first analyze the angular distributions of RGB micro-LEDs and obtain good agreement between simulation and experiment. Next, we propose a device structure with top black matrix and taper angle in micro-LEDs, which greatly suppresses the color shift while keeping a reasonably high light extraction efficiency.