Intense angiogenesis is critical for the development of the corpus luteum and is tightly regulated by numerous factors. However, the exact role transforming growth factor-β1 (TGFB1) plays during this follicular–luteal transition remains unclear. This study hypothesised that TGFB1, acting through TGFB receptor 1 (TGFBR1) and Smad2/3 signalling, would suppress angiogenesis during the follicular–luteal transition. Using a serum-free luteinising follicular angiogenesis culture system, TGFB1 (1 and 10 ng mL–1) markedly disrupted the formation of capillary-like structures, reducing the endothelial cell network area and the number of branch points (P < 0.001 compared with control). Furthermore, TGFB1 activated canonical Smad signalling and inhibited endothelial nitric oxide synthase (NOS3) mRNA expression, but upregulated latent TGFB-binding protein and TGFBR1, serpin family E member 1 (SERPINE1) and serpin family B member 5 (SERPINB5) mRNA expression. SB431542, a TGFBR1 inhibitor, reversed the TGFB1-induced upregulation of SERPINE1 and SERPINB5. In addition, TGFB1 reduced progesterone synthesis by decreasing the expression of steroidogenic acute regulatory protein (STAR), cytochrome P450 family 11 subfamily A member 1 (CYP11A1) and 3β-hydroxysteroid dehydrogenase (HSD3B1) expression. These results show that TGFB1 regulates NOS3, SERPINE1 and SERPINB5 expression via TGFBR1 and Smad2/3 signalling and this could be the mechanism by which TGFB1 suppresses endothelial networks. Thereby, TGFB1 may provide critical homeostatic control of angiogenesis during the follicular–luteal transition. The findings of this study reveal the molecular mechanisms underlying the actions of TGFB1 in early luteinisation, which may lead to novel therapeutic strategies to reverse luteal inadequacy.