The purpose of this research was to investigate the reliability of the scores produced and validity of the inferences drawn from the American Association for the Advancement of Science (AAAS, 2018) force and motion sub-topic assessment for middle school students. The assessment of student outcomes in STEM is an international focus in K-12 education. Project 2061, initiated by the AAAS, focuses on addressing challenges related to standards and assessments. This study informs this effort through testing a 14-item multiple-choice test constructed of questions from the AAAS item bank. Two samples of eighth-grade students participated (N = 1777). Rasch analysis applying the dichotomous model (Rasch, 1960) indicated sufficient item separation and reliability. Thirteen items fit the Rasch model and one item was removed for misfit. Further support for construct validity was observed with 78 percent of item ordering aligned with that predicted by physics educators and stability of measures for 11 items across the two samples. One item exhibited significant differential item functioning by gender and minority status in science. After inspection by physics educators, no bias in item wording or context was determined. Recommendation for additional items is made to increase item targeting and variance explained by the Rasch linear measure.