This paper proposes linear precoder-decoder schemes for a multiple-input multiple-output (MIMO) underlay device-to-device (D2D) communication system by considering two D2D modes: two-way relaying based D2D and direct D2D. The D2D communication takes place in the same spectrum as the cellular communication. In the two-way relaying based D2D mode, the relay uses physical layer network coding (PNC). The precoder-decoder design is based on minimizing mean square errors (MSE), which is useful to mitigate interference and to improve the performance of both D2D and cellular communications. Distributed and centralized algorithms are proposed considering bi-directional communication in both D2D and cellular communications. In the direct D2D mode, a similar MSE procedure is adopted, and exact solutions are derived for precoder-decoder matrices. In the numerical results, the optimality and convergence properties of the proposed algorithms are analyzed. Additionally, the system performances are investigated with interference thresholds and maximum available power at the nodes. Two transmit mode selection schemes are considered as dynamic and static selection schemes. Finally, these selection schemes are investigated over an XY grid by varying the position of a given device. The results reveal that the PNC two-way relaying based D2D mode extends the coverage area of D2D communication.