Recognition of others' emotions is a key life ability that guides one's own choices and behavior, and it hinges on the recognition of others' facial cues. Independent studies indicate that facial appearance-based evaluations affect social behavior, but little is known about how facial appearance-based trustworthiness evaluations influence the recognition of specific emotions. We tested the hypothesis that first impressions based on facial appearance affect the recognition of basic emotions. A total of 150 participants completed a dynamic emotion recognition task. In a within-subjects design, the participants viewed videos of individuals with trustworthy-looking, neutral, or untrustworthy-looking faces gradually and continuously displaying basic emotions (happiness, anger, fear, and sadness). The participants' accuracy and speed in recognizing the emotions were measured. Untrustworthy-looking faces decreased participants' emotion recognition accuracy and speed, across emotion types. In addition, faces that elicited a positive inference of trustworthiness enhanced emotion recognition speed of fear and sadness, emotional expressions that signal another's distress and modulate prosocial behavior. These findings suggest that facial appearance-based inferences may interfere with the ability to accurately and rapidly recognize others' basic emotions.