The following question is addressed: To what extent the \(n\)-tuple of \(m\times m\) Hermitian matrices is determined by its joint numerical range? The cases \(m=2\), \(n\) arbitrary and \(m=n=3\) are considered in detail. For \(m=2\), the principal result states: The \(n\)-tuple of \(2\times 2\) Hermitian matrices \(A_1, \dots, A_n\) is determined up to unitariy similarity by its joint numerical range \(J\) if and only if the set \(J\) is flat, i.e., when the dimension of the affine hull of~\(J\) is \(\leqslant 2\). For arbitrary~\(m\), the previous result can be generalized in the following way: Let \((A_1, \dots, A_n)\) be an \(n\)-tuple of \(m\times m\) matrics with the joint numerical range \(J\) different from a polytope and having \(m-2\) conical points. Then all the \(n\)-tuples of \(m\times m\) matrics with the same numerical range \(J\) are uitarily similar either to \((A_1, \dots, A_n)\) or to its transposed.