The neural substrates of affective prosody are beginning to be understood. However, temporal processing and differences between emotions have yet to be established. High-density brain electrical activity was recorded while twenty healthy females heard and categorized semantically neutral sentences presented in five emotional prosodies (happy, sad, angry, fearful, and neutral). Accuracy was comparable between emotions, but reaction time was significantly different. Event-related potentials (ERPs) revealed an overall effect of prosody, with a main component peaking at 1 sec, localized in anterior temporal cortex (right greater than left). Main differences between emotions included an early effect peaking at 450 ms with increased activity in response to fear and localized in right dorsal premotor cortex. A later effect, peaking at 950 ms, showed sad-specific activity localized to left inferior premotor cortex. These results suggest that different emotions recruit distinct premotor regions during affective prosody recognition.