This paper aims to add further research to the field of downside risk, and downside risk measures’ influence on the average returns in the U.S. stock market. The study also examines and compares how well the Fama-French three-factor model, Carhart four-factor model, Fama-French five-factor Model, q-four factor model, and q-five factor model explain these average returns. This was done by constructing zero-cost portfolios, split into two weight classes of stocks in the portfolios. The study shows relatively strong results for a major group of the downside risk measures. The measures of the major group show significance and good explanatory power; this could lay ground for further research and use of downside risk measures in financial contexts. Regarding the minor group of the downside risk measures, the result gives ambiguous implications about the way the asset pricing models can explain those residual mean returns. Therefore, the minor group could not establish what asset pricing model is preferred over other models.