Patients with right hemisphere damage and visual neglect have severe problems to orient attention towards left-sided objects, often associated with the tendency to produce inappropriate rightward saccades. In its most severe form, this tendency can assume the compulsive character of a rightward deviation of gaze as soon as the visual scene deploys (so-called “magnetic attraction of gaze”). However, little is known about the exact nature of inappropriate rightward saccades, their relation with impaired conscious perception of left-sided stimuli, and their lesional correlates. To explore these issues, we studied three groups of patients with right brain damage: patients with signs of left visual neglect associated to left homonymous hemianopia, neglect patients without hemianopia, and patients without neglect or hemianopia. Participants searched for a gap missing within a target, presented among distractors. Manual responses for target detection were required, while participants were encouraged to move their eyes during search. Endogenous attention could be summoned to the target location by a central cue. All the three groups of patients produced inappropriate rightward saccades, which could not be completely overcome by the endogenous orienting of attention induced by the cues. Anatomical analysis indicated a specific implication of damage to the right frontal eye field and to a long-range white matter tract, the fronto-parietal superior longitudinal fasciculus. Fronto-parietal networks in the right hemisphere appear thus to be essential to integrate covert and overt orienting of attention, and to thoroughly explore space in order to become aware of the multiple competing objects around us.