Berry connections are shown to be invariant under the action of symmetry groups on parameter space. This observation allows one to use the theory of invariant Yang-Mills potentials to evaluate these connections without computing the instantaneous energy eigenstates. Hamiltonians belonging to the algebra of a Lie group are examined in this light. They provide a wide class of systems admitting nontrivial adiabatic holonomy. The special case of the generalized harmonic oscillator is analyzed and the SO(2,1) invariance of the associated Berry connection is exhibited.