We introduce the concept of n-OU and n-OO matrix sets, a collection of n mutually-orthogonal unitary and real orthogonal matrices under Hilbert-Schmidt inner product. We give a detailed characterization of order-three n-OO matrix sets under orthogonal equivalence. As an application in quantum information theory, we show that the minimum and maximum numbers of an unextendible maximally entangled bases within a real two-qutrit system are three and four, respectively. Further, we propose a new matrix decomposition approach, defining an n-OU (resp. n-OO) decomposition for a matrix as a linear combination of n matrices from an n-OU (resp. n-OO) matrix set. We show that any order-d matrix has a d-OU decomposition. As a contrast, we provide criteria for an order-three real matrix to possess an n-OO decomposition.

16 pages, no figure