We formally define Markov quantal response equilibrium (QRE) and prove existence for all finite discounted dynamic stochastic games. The special case of logit Markov QRE constitutes a mapping from precision parameter λ to sets of logit Markov QRE. The limiting points of this correspondence are shown to be Markov perfect equilibria. Furthermore, the logit Markov QRE correspondence can be given a homotopy interpretation. We prove that for all games, this homotopy contains a branch connecting the unique solution at λ = 0 to a unique limiting Markov perfect equilibrium. This result can be leveraged both for the computation of Markov perfect equilibria, and also as a selection criterion.