Estimators of regression coefficients are known to be asymptotically normally distributed, provided certain regularity conditions are satisfied. In small samples and if the noise is not normally distributed, this can be a poor guide to the quality of the estimators. The paper addresses this problem for small and medium sized samples and heavy tailed noise. In particular, we assume that the noise is regularly varying, i.e., the tails of the noise distribution exhibit power law behavior. Then the distributions of the regression estimators are heavy tailed themselves. This is relevant for regressions involving financial data which are typically heavy tailed. In medium sized samples and with some dependency in the noise structure, the regression coefficient estimators can deviate considerably from their true values. The relevance of the theory is demonstrated for the highly variable cross country estimates of the expectations coefficient in yield curve regressions.