In this letter we consider a distributed stochastic optimization framework in which agents in a network aim to cooperatively learn an optimal network-wide policy. The goal is to compute local functions to minimize the expected value of a given cost, subject to individual constraints and average coupling constraints. In order to handle the challenges of the distributed stochastic context, we resort to a Lagrangian duality approach that allows us to derive an associated stochastic dual problem with a separable structure. Thus, we propose a distributed algorithm, without a central coordinator, that exploits consensus iterations and stochastic approximation to find an optimal solution to the problem, with attractive scalability properties. We demonstrate convergence of the proposed scheme and validate its behavior through simulations.