Extra virgin olive oil (EVOO) is one of the most distinctive ingredients of the Mediterranean diet. There are many properties related to this golden ingredient, from supreme organoleptic characteristics to benefits for human health. EVOO contains in its composition molecules capable of exerting bioactivities such as cardio protection, antioxidant, anti-inflammatory, antidiabetic, and anticancer activity, among others, mainly caused by unsaturated fatty acids and certain minor compounds such as tocopherols or phenolic compounds. EVOO is considered the highest quality vegetable oil, which also implies a high sensory quality. The organoleptic properties related to the flavor of this valued product are also due to the presence of a series of compounds in its composition, mainly some carbonyl compounds found in the volatile fraction, although some minor compounds such as phenolic compounds also contribute. However, these properties are greatly affected by the incidence of certain factors, both intrinsic, such as the olive variety, and extrinsic, such as the growing conditions, so that each EVOO has a particular flavor. Furthermore, these flavors are susceptible to change under the influence of other factors throughout the oil’s shelf-life, such as oxidation or temperature. This work offers a description of some of the most remarkable compounds responsible for EVOO’s unique flavor and aroma, the factors affecting them, the mechanism that lead to the degradation of EVOO, and how flavors can be altered during the shelf-life of the oil, as well as several strategies suggested for the preservation of this flavor, on which the quality of the product also depends.