We analyze the outage performance of relay-assisted amplify-and-forward systems under uncompensated I/Q imbalance effects and derive accurate analytical approximations for both cases. In addition, we compare the outage probability for amplify-and-forward systems with that of the direct transmission mode. Both analytical and numerical results demonstrate that the direct mode can outperform the amplify-and-forward mode even under moderate levels of uncompensated I/Q imbalance. The cross-over I/Q imbalance levels are determined analytically to be inversely proportional to the cube of the signal constellation size. Furthermore, we propose a low-complexity receiver-based digital baseband I/Q imbalance compensation scheme for the amplify-and-forward mode. Numerical results show its performance superiority to the conventional maximal-ratio combining scheme even under imperfect channel state information (CSI).