In the past, the amygdala has generally been conceptualized as a fear-processing module. Recently, however, it has been proposed to respond to all stimuli that are relevant with respect to the current needs, goals, and values of an individual. This raises the question of whether the human amygdala may differentiate between separate kinds of relevance. A distinction between emotional (vs. neutral) and social (vs. non-social) relevance is supported by previous studies showing that the human amygdala preferentially responds to both emotionally and socially significant information, and these factors might even display interactive encoding properties. However, no investigation has yet probed a full 2 (positive vs. negative valence) × 2 (social vs. non-social content) processing pattern, with neutral images as an additional baseline. Applying such an extended orthogonal factorial design, our fMRI study demonstrates that the human amygdala is (1) more strongly activated for neutral social vs. non-social information, (2) activated at a similar level when viewing social positive or negative images, but (3) displays a valence effect (negative vs. positive) for non-social images. In addition, this encoding pattern is not influenced by cognitive or behavioral emotion regulation mechanisms, and displays a hemispheric lateralization with more pronounced effects on the right side. Finally, the same valence × social content interaction was found in three additional cortical regions, namely the right fusiform gyrus, right anterior superior temporal gyrus, and medial orbitofrontal cortex. Overall, these findings suggest that valence and social content processing represent distinct kinds of relevance that interact within the human amygdala as well as in a more extensive cortical network, likely subserving a key role in relevance detection.