A significant body of evidence has supported a negative impact of endometriosis on ovarian follicles; however, the origin and relevance of this ovarian impairment in endometriosis is still a matter of debate. The ovarian damage can be caused by endometriosis itself or by surgeries aiming to remove endometriotic lesions. In this review, we summarized the existing knowledge on the mechanisms by which endometriosis can impact the ovarian follicles, from molecular to clinical points of view. From a molecular standpoint, the presence of endometriosis or its consequences can induce oxidative stress, inflammation, aberrant mitochondrial energy metabolism and inappropriate steroid production in granulosa cells, phenomena that may impair the quality of oocytes to variable degrees. These alterations may have clinical relevance on the accelerated exhaustion of the ovarian reserve, on the ovarian response to gonadotrophin stimulation in IVF cycles and on the competence of the oocytes. Critical points to be considered in current clinical practices related to fertility issues in endometriosis are discussed.