Color harmony of simple color patterns has been widely studied for color design. Rules defined then by psychological experiments have been applied to derive image aesthetic scores, or to re-colorize pictures. But what is harmonious or not in an image? What can the human eye perceive disharmonious? Extensive research has been done in the context of quality assessment to define what is visible or not in images and videos. Techniques based on human visual system models use signal masking to define visibility thresholds. Based on results in both fields, we present a harmony quality assessment method to assess what is harmonious or not in an image. Color rules are used to detect what part of images are disharmonious, and visual masking is applied to estimate to what extent an image area can be perceived disharmonious. The output perceptual harmony quality map and scores can be used in a photo editing framework to guide the user getting the best artistic effects. Results show that the harmony maps reflect what a user perceives and that the score is correlated to the artistic intent.