Cultivar segregation according to the sensory perception of their organoleptic characteristics was attempted by using trained panel data evaluated by principal component analysis of 12 plum and four pluot cultivars as a part of our program to understand plum minimum quality. The perception of the four sensory attributes (sweetness, sourness, plum flavor intensity, plum aroma intensity) was reduced to three principal components, which accounted for 98.6% of the variation in the sensory attributes of the tested cultivars. Using the Ward separation method and PCA analysis (PC1 = 49.8% and PC2 = 25.6%), plum and pluot cultivars were segregated into groups (tart, plum aroma, and sweet/plum flavor) with similar sensory attributes. Fruit source significantly affected cultivar ripe soluble solids concentration (RSSC) and ripe titratable acidity (RTA), but it did not significantly affect sensory perception of plum flavor intensity, sourness, sweetness, and plum aroma intensity by the trained panel on fruit harvested above their physiological maturity. Based on this information, we recommend that validation of these organoleptic groups should be conducted using “in store” consumer tests prior to development of a minimum quality index within each organoleptic group based on ripe soluble solids concentration (RSSC). This organoleptic cultivar classification will help to match consumer preferences and enhance current promotion and marketing programs.