SummaryIn this paper, the performance of transmission techniques on the fixed‐gain amplify‐and‐forward–based asymmetric radio frequency/free space optical (RF/FSO) communication system is studied. The RF link and FSO link are, respectively, modeled by the Nakagami‐m and Gamma‐Gamma fading channels under the effect of zero boresight and non‐zero boresight pointing errors subject to heterodyne detection. Maximum ratio transmission (MRT) and orthogonal space‐time block coding (OSTBC) techniques are the transmission scenarios considered at the system source while selection combining is used for reception at the relay and destination for the signal detection. Moreover, a unified cumulative density distribution (CDF) of end‐to‐end signal‐to‐noise ratio is derived for the system. By utilizing this channel statistical CDF, the closed‐form expressions for the outage probability and average bit error rate for the M‐ary phase shift keying modulation are then obtained for the system. The analysis presented illustrates that both the atmospheric turbulence and pointing error significantly degrade the system performance. Based on this, the MRT transmission technique offers a better performance compared with the OSTBC techniques under the same system conditions. The accuracy of the analytical results is verified by Monte‐Carlo simulations.