The main object under investigation is the family of the Hermite matrix orthogonal polynomials \(\{H_n(x,A)\}_{n\geq 0}\), which depends on the matrix parameter \(A\) having all its eigenvalues in the open right half plane. The main result (Theorem 1) states that \[ \| H_{2n}(x,A)\| \leq \frac{(2n+1)! 2^{2n}}{n!} \exp(5/2 \| A\| x^2), \] and the similar inequality holds for the odd numbered Hermite matrix polynomials. As an application of this result the authors suggest an algorithm for computing the matrix exponential with prescribed approximation error. The key tool is the formula for generating function \[ \sum_{n\geq 0} \frac{(-1)^n H_{2n}(x,A)}{2^{2n} n!}t^{2n} = (1-t^2)^{-1/2}\exp\left(-\frac{A}2 \frac{x^2t^2}{(1-t^2)}\right). \]