In recent years, in quantum information theory, there has been a remarkable development in the general theoretical framework for studying symmetry in dynamics. This development, called resource theory of asymmetry, is expected to have a wide range of applications, from accurate time measurements to black hole physics. Despite its importance, the foundation of resource theory of asymmetry still has room for expansion. An important problem is in quantifying the amount of resource. When the symmetry prescribed U(1), i.e., with a single conserved quantity, the quantum Fisher information is known as a resource measure that has suitable properties and a clear physical meaning related to quantum fluctuation of the conserved quantity. However, it is not clear what is the resource measure with such suitable properties when a general symmetry prevails for which there are multiple conserved quantities. The purpose of this paper is to fill this gap. Specifically, we show that the quantum Fisher information matrix is a resource measure whenever a connected linear Lie group describes the symmetry. We also consider the physical meaning of this matrix and see which properties that the quantum Fisher information has when the symmetry is described by $U(1)$ can be inherited by the quantum Fisher information matrix.