Tasks for which people must act together to achieve a goal are a feature of daily life. The present study explored social influences on joint action using a Simon procedure for which participants (n = 44) were confronted with a series of images of hands and asked to respond via button press whenever the index finger wore a ring of a certain color (red or green) regardless of pointing direction (left or right). In an initial joint condition they performed the task while sitting next to another person (friend or stranger) who responded to the other color. In a subsequent individual condition they repeated the task on their own; additionally, they completed self-report tests of empathy. Consistent with past research, participants reacted more quickly when the finger pointed toward them rather than their co-actor (the Simon Effect or SE). The effect remained robust when the co-actor was no longer present and was unaffected by degree of acquaintance; however, its magnitude was correlated positively with empathy only among friends. For friends, the SE was predicted by cognitive perspective taking when the co-actor was present and by propensity for fantasizing when the co-actor was absent. We discuss these findings in relation to social accounts (e.g., task co-representation) and non-social accounts (e.g., referential coding) of joint action.