The asymptotic phase is a fundamental quantity for the analysis of deterministic limit-cycle oscillators, and generalized definitions of the asymptotic phase for stochastic oscillators have also been proposed. In this article, we show that the asymptotic phase and also amplitude can be defined for classical and semiclassical stochastic oscillators in a natural and unified manner by using the eigenfunctions of the Koopman operator of the system. We show that the proposed definition gives appropriate values of the phase and amplitude for strongly stochastic limit-cycle oscillators, excitable systems undergoing noise-induced oscillations, and also for quantum limit-cycle oscillators in the semiclassical regime.