Accurate motion detection requires neural circuitry that compensates for global visual field motion. Select subtypes of retinal ganglion cells perceive image motion and connect to the accessory optic system (AOS) in the brain, which generates compensatory eye movements that stabilize images during slow visual field motion. Here, we show that the murine transmembrane semaphorin 6A (Sema6A) is expressed in a subset of On direction-selective ganglion cells (On DSGCs) and is required for retinorecipient axonal targeting to the medial terminal nucleus (MTN) of the AOS. Plexin A2 and A4, two Sema6A binding partners, are expressed in MTN cells, attract Sema6A(+) On DSGC axons, and mediate MTN targeting of Sema6A(+) RGC projections. Furthermore, Sema6A/Plexin-A2/A4 signaling is required for the functional output of the AOS. These data reveal molecular mechanisms underlying the assembly of AOS circuits critical for moving image perception.