In this paper, we consider a non-orthogonal multiple access cognitive radio network, where a full-duplex multi-antenna relay assists transmission from a base station (BS) to a cognitive far user, whereas, at the same time, the BS transmits to a cognitive near user. Our objective is to enlarge the far-near user rate region by maximizing the rate of the near user under a constraint that the rate of the far user is above a certain threshold. To this end, a non-convex joint optimization problem of relay beamforming and the transmit powers at the BS and cognitive relay is solved as a semi-definite relaxation problem, in conjunction with an efficiently solvable line-search approach. For comparisons, we also consider low complexity fixed beamformer design, where the optimum power allocation between the BS and cognitive relay is solved. Our results demonstrate that the proposed joint optimization can significantly reduce the impact of the residual self-interference at the FD relay and inter-user interference in the near user case.

Accepted for IEEE Global Communications Conference (GLOBECOM 2017)