This paper uses a laboratory experiment to study beliefs and their relationship to action and strategy choices in finitely and indefinitely repeated prisoners' dilemma games. We find subjects' beliefs about the other player's action are accurate despite some systematic deviations corresponding to early pessimism in the indefinitely repeated game and late optimism in the finitely repeated game. The data reveals a close link between beliefs and actions that differs between the two games. In particular, the same history of play leads to different beliefs, and the same belief leads to different action choices in each game. Moreover, we find beliefs anticipate the evolution of behavior within a super game, changing in response to the history of play (in both games) and the number of rounds played (in the finitely repeated game). We then use the subjects' beliefs over actions in each round to identify their beliefs over super game strategies played by the other player. We find these beliefs correctly capture the different classes of strategies used in each game. Importantly, subjects using different strategies have different beliefs, and for the most part, strategies are subjectively rational given beliefs. The results also suggest subjects tend to overestimate the likelihood that others use the same strategy as them, while underestimating the likelihood that others use less cooperative strategies.