When a population is engaged in successive prisoner's dilemmas, indirect reciprocity through reputation fosters cooperation through the emergence of moral and action rules. A simplified model has recently been proposed where individuals choose between helping others or not and are judged good or bad for it by the rest of the population. The reputation so acquired will condition future actions. In this model, eight strategies (referred to as "leading eight") enforce a high level of cooperation, generate high payoffs, and are therefore resistant to invasions by other strategies. Here we show that, by assigning each individual one of two labels that peers can distinguish (e.g., political ideas, religion, and skin color) and allowing moral and action rules to depend on the label, intolerant behaviors can emerge within minorities under sufficient economic stress. We analyze the sets of conditions where this can happen and also discuss the circumstances under which tolerance can be restored. Our results agree with empirical observations that correlate intolerance and economic stress and predict a correlation between the degree of tolerance of a population and its composition and ethical stance.