Copying the majority is generally an adaptive social learning strategy but the majority does not always know best. Previous work has demonstrated young children's selective uptake of information from a consensus over a lone dissenter. The current study examined children's flexibility in following the majority: do they overextend their reliance on this heuristic to situations where the dissenting individual has privileged knowledge and should be trusted instead? Four- to six- year-olds (N = 103) heard conflicting claims about the identity of hidden drawings from a majority and a dissenter in two between-subject conditions: in one, the dissenter had privileged knowledge over the majority (he drew the pictures); in the other he did not (they were drawn by an absent third party). Overall, children were less likely to trust the majority in the Privileged Dissenter condition. Moreover, 5- and 6- year-olds made majority-based inferences when the dissenter had no privileged knowledge but systematically endorsed the dissenter when he drew the pictures. The current findings suggest that by 5 years, children are able to make an epistemic-based judgment to decide whether or not to follow the majority rather than automatically following the most common view.