A multi-agent generation expansion planning (GEP) simulation is useful for electricity market design. However, most preceding studies which assume a deterministic strategy didn't consider important elements of the multi-agent GEP, e.g. probabilistic strategy and multiple determination timing over years. This paper models the multi-agent GEP problem as an uncooperative stochastic strategy dynamic game, and proposes the novel simulation method based on reinforcement learning. Because this simulation method assumes probabilistic strategy, subgame's Nash equilibriums in every situation and every year can be derived. From the simulation results of the proposed method and conventional iterative search method for simple test power system, it's shown that a reasonable probabilistic investment strategy which can get more expected profit than the conventional method by the early investment strategy can be derived by using this method.