This study uses a self-presentational framework to investigate second language listening anxiety (L2 listening anxiety) among university students learning English in Japan and demonstrate that L2 listening involves social concerns that are specific to L2 settings. Successful performance in aural interaction presupposes mutual understanding, and L2 listeners have good reason to become anxious when it is doubtful whether they properly comprehend what others say. The Shortened Scale of Foreign Language Listening Anxiety, the Revised Interaction Anxiousness Scale, the Penn State Worry Questionnaire as well as a dictation test were administered to 1,177 students in 15 universities for the quantitative part of this study. Introspection verbal report data were collected from 17 students in two universities to investigate the thought processes of L2 listeners for the qualitative part. The profile analysis indicated the following: L2 listening anxiety was (a) specific to L2 situations and (b) linked to L2 proficiency because high and low L2 listening proficiency groups were different in the overall levels of L2 listening anxiety, but not different in general anxiety measures. The exploratory factor analysis and Rasch principle component analyses demonstrated that L2 listening anxiety was a broad construct composed of two related but distinct dimensions, Self-Focused Apprehension and Task-Focused Apprehension. The former is a concern over social evaluative threat, and the latter is worry over effective processing of aural input. The univariate analyses of variance confirmed that L2 listening anxiety was partly socially constructed because social anxiety was linked to both dimensions of L2 listening anxiety. The verbal data suggested that L2 listening anxiety was receiver-specific in that it involved concerns over comprehending and responding appropriately to aural messages. They also indicated that the levels of L2 listening anxiety were (a) susceptible to individual differences, and (b) influenced by different social situations. This study contributes to conceptual developments in the area of L2 learner psychology because understanding others is of profound importance in successful communication, and anxiety over non-understanding or misunderstanding can have significant personal and interpersonal consequences.