Dynamic term structure models price interest rate options based on the model-implied fair values of the yield curve, ignoring any pricing residuals on the yield curve that are either from model approximations or market imperfections. In contrast, option pricing in practice often takes the market observed yield curve as given and focuses exclusively on the specification of the volatility structure of forward rates. Thus, if any errors exist on the observed yield curve, they will be carried over permanently. In this paper, we propose an m+n model structure that bridges the gap between the two approaches and consistently prices both interest rates and interest rate options. The first m factors capture the systematic movement of the yield curve, whereas the latter n factors capture the impacts of the yield curve residuals on option pricing. We estimate a 3+3 Gaussian affine example using eight years of data on U.S. dollar LIBOR, swap rates, and interest rate caps. The model performs well in pricing both interest rates and interest rate caps. Furthermore, estimation shows that small residuals on the yield curve can have large impacts on pricing interest rate caps. Under the estimated model, the three yield curve factors explain over 99 percent of the variation on the yield curve, but account for less than 50 percent of the variation on cap implied volatilities. Incorporating the three residual factors improves the explained variance on cap implied volatility to over 99 percent.