How should we deal with social diversity if we conceive it as cultural diversity? Appeals to cultural relativism and to the collective good of diversity provide inadequate answers. Taking cultural diversity seriously requires that we respond to it fairly or justly and that, in turn, requires an approach that is impartial (or neutral) amongst cultures. Claims of impartiality are often thought peculiarly implausible when applied to cultural diversity, but an impartialist approach is in fact peculiarly appropriate to that form of diversity. The issue is not whether we can be impartial, but whether we are ready to accept the implications of describing a diversity as ‘cultural’ and, if we are, what form our impartiality should take. Attempts to avoid claims of impartiality by dealing with diversity through deliberative processes are misguided since those processes must embody commitments to impartiality if their outcomes are to be just.