We investigate the space of Hermitian metrics on a fixed complex vector bundle. This infinite-dimensional space has appeared in the study of Hermitian-Einstein structures, where a special L2-type Riemannian metric is introduced. We compute the metric spray, geodesics and curvature associated to this metric, and show that the exponential map is a diffeomorphsim. Though being geodesically complete, the space of Hermitian metrics is metrically incomplete, and its metric completion is proved to be the space of L2 integrable singular Hermitian metrics. In addition, both the original space and its completion are CAT(0). In the holomorphic case, it turns out that Griffiths seminegative/semipositive singular Hermitian metric is always "L2 integrable" in our sense. Also, in the Appendix, the Nash-Moser inverse function theorem is used to prove that, for any L2 metric on the space of smooth sections of a given fiber bundle, the exponential map is always a local diffeomorphism, provided that each fiber is nonpositively curved.