Summary: This paper investigates the stability of optimal solution sets to stochastic programs with complete recourse, where the underlying probability measure is understood as a parameter varying in some space of probability measures. \textit{A. Shapiro} [Math. Program. 67A, No. 1, 99-108 (1994; Zbl 0828.90099)] proved Lipschitz upper semicontinuity of the solution set mapping. Inspired by this result, we introduce a subgradient distance for probability distributions and establish the persistence of optimal solutions. For a subclass of recourse models we show that the solution set mapping is (Hausdorff) Lipschitz continuous with respect to the subgradient distance. Moreover, the subgradient distance is estimated above by the Kolmogorov-Smirnov distance of certain distribution functions related to the recourse model. The Lipschitz continuity result is illustrated by verifiable sufficient conditions for stochastic programs to belong to the mentioned subclass and by examples showing its validity and limitations. Finally, the Lipschitz continuity result is used to derive some new results on the asymptotic behavior of optimal solutions when the probability measure underlying the recourse model is estimated via empirical measures (law of iterated logarithm, large deviation estimate, estimate for asymptotic distribution).