Tilt-to-length coupling was the limiting noise source in LISA Pathfinder between 20 and 200 mHz before subtraction in post-processing. To prevent the adding of sensing noise to the data by the subtraction process, the success of this strategy depended on a previous direct noise reduction by test mass alignment. The exact dependency of the level of tilt-to-length coupling on the set-points of LISA Pathfinder's test masses was not understood until the end of the mission. Here, we present, for the first time, an analytical tilt-to-length coupling model that describes the coupling noise changes due to the realignments. We report on the different mechanisms, namely the lever arm and piston effect as well as the coupling due to transmissive components, and how they contribute to the full coupling. Further, we show that a pure geometric model would not have been sufficient to describe the coupling in LISA Pathfinder. Therefore, we model also the non-geometric tilt-to-length noise contributions. For the resulting coupling coefficients of the full model, we compute the expected error bars based on the known individual error sources. Also, we validated the analytical model against numerical simulations. A detailed study and thorough understanding of this noise are the basis for a successful analysis of the LISA Pathfinder data with respect to tilt-to-length coupling.