We show theoretically and with the aid of computer simulations that the nonlinear depolarization induced by cross-polarization modulation leads to crosstalk between the polarization-division multiplex (PolDM) subchannels during their demultiplexing. This additive crosstalk can be approximated as additive Gaussian noise, and we derive expressions to estimate its variance. It can lead to significant penalties of up to several decibels for PolDM transmission and must be properly accounted for. We also show why time-interleaving both subchannels can significantly reduce these penalties.