Foreign language listening anxiety (FLLA) significantly affects learners’ listening performance but remains less explored compared to other language anxieties. This study surveyed 702 Chinese tertiary EFL learners across 22 faculties to investigate key dimensions of FLLA, gender differences, and their relationship with English proficiency. Through exploratory and confirmatory factor analyses, four distinct dimensions of FLLA were identified: Situational Listening Anxiety (SLA), Self-perceived Cognitive Load (SC), Self-perceived Affective Load (SA), and Listening Processing Anxiety (LP). Among them, LP emerged as the most influential, indicating that real-time processing challenges outweigh emotional or contextual stress. Gender was not a significant predictor of FLLA, suggesting that anxiety stems more from cognitive and situational challenges than from demographic factors. Interestingly, English proficiency was weakly but positively correlated with SLA and SA, implying that more proficient learners may experience slightly heightened anxiety due to increased expectations or task complexity. The findings highlight the need for pedagogical strategies that target cognitive processing and listening-specific training rather than focusing solely on emotional or demographic considerations.