The generalized Langevin equation is widely used to model the influence of a heat bath upon a reactive system. This equation will here be studied from a geometric point of view. A dynamical phase space that represents all possible states of the system will be constructed, the generalized Langevin equation will be formally rewritten as a pair of coupled ordinary differential equations, and the fundamental geometric structures in phase space will be described. It will be shown that the phase space itself and its geometric structure depend critically on the preparation of the system: A system that is assumed to have been in existence forever has a larger phase space with a simpler structure than a system that is prepared at a finite time. These differences persist even in the long-time limit, where one might expect the details of preparation to become irrelevant.