The importance of proper initialization in taking into account the history of a system whose time evolution is governed by a differential equation of fractional order, has been established by Lorenzo and Hartley, who also gave the method of properly incorporating the effect of the past (history) by means of an initialization function for the Riemann-Liouville and the Grunwald formulations of fractional calculus. The present work addresses this issue for the Caputo fractional derivative and cautions that the commonly held belief that the Caputo formulation of fractional derivatives properly accounts for the initialization effects is not generally true when applied to the solution of fractional differential equations.