The transduction of the serotonin (5-HT) signal in Fundulus heteroclitus ovarian follicles leading to the inhibition of oocyte meiosis reinitiation (oocyte maturation) in vitro induced by the naturally occurring maturation-inducing steroid 17 alpha, 20 beta-dihydroxy-4-pregnen-3-one (17,20 beta P) was investigated. Steroid-induced oocyte maturation was inhibited by 5-HT in a dose-dependent manner; maximum inhibition (90%) was observed with 10(-4) M 5-HT. Groups of follicle-enclosed oocytes were cultured in the presence of the phosphodiesterase inhibitor 3-isobutyl-1-methylxanthine (IBMX) and treated with increasing doses of 5-HT. Serotonin was found to slightly increase the levels of follicular 3',5'-cyclic adenosine monophosphate (cAMP) in a dose-dependent manner; 10(-4) M 5-HT induced approximately a 3-fold increase in cAMP with respect to the controls. The changes in cAMP were then evaluated in follicles treated with 17,20 beta P in IBMX-free culture media in the presence or absence of 10(-4) M 5-HT. The exposure of follicles to 17,20 beta P alone produced a small and transient reduction in cAMP (40%) within 1-3 hr of steroid stimulation, and these early changes in cAMP appeared associated with a high incidence of germinal vesicle breakdown (80% GVBD) by 24 hr of incubation. Under these conditions, treatment of follicles with 5-HT also increased significantly the production of cAMP, and when 5-HT was combined with 17,20 beta P, the steroid-mediated reduction in cAMP was prevented and the levels of GVBD inhibited by 95%. Meiosis also was reinitiated with either the protein kinase A (PKA) inhibitor H8 or the protein kinase C (PKC) activator PMA, and the 5-HT inhibitory action on GVBD was found to be 100-fold reduced or completely ineffective, respectively. Preincubation of follicles with the PKC inhibitor GF109203x abolished PMA-induced GVBD in a dose-dependent manner, whereas this inhibitor had no effect on 17,20 beta P-triggered meiotic maturation, indicating that activation of PKC is apparently sufficient but not necessary to reinitiate meiosis. Taken together, these findings suggest that 5-HT may inhibit 17,20 beta P-induced meiotic reinitiation through the activation of a cAMP-PKA transduction pathway and that PKC possibly induces oocyte maturation by a different pathway than the steroid and thus is not affected by 5-HT.