This is a survey concerning the most appropriate numerical methods in nonsmooth and stochastic optimization. It contains the main results in the above field obtained by the author from 1974 up to 1992: the combined method of iterative integral penalty and stochastic quasigradients for semi-infinite optimization, the combined method of iterative penalty, iterative approximation and stochastic quasigradients for the solution of convex constrained optimization problems in Hilbert spaces, extensions of the combined method on iterative penalty, iterative approximation and stochastic quasigradients to problems of infinite-dimensional optimization with a finite-dimensional perturbation, optimization methods for uncontrolled factors in Hilbert spaces, many algorithms, important results, significant comments and some applications. The basis for the algorithms is ensured by the combined iterative integral penalty method and the stochastic quasigradients, treating also this method in infinite-dimensional cases. This monograph addresses researchers in stochastic methods for variational problems in Mathematical Physics and Operations Research, professional mathematicians and graduate students and can be used as a veritable textbook on advanced stochastic methods and their recent implications.