Cortical feedback connections are extremely numerous but the logic of connectivity between higher and lower areas remains poorly understood. Feedback from higher visual areas to primary visual cortex (V1) has been shown to enhance responses on perceptual figures compared to backgrounds, an effect known as figure-background modulation (FBM). A likely source of this feedback are border-ownership (BO) selective cells in mid-tier visual areas (e.g. V4) which represent the location of figures. We examined the connectivity between V4 cells and V1 cells using noise-correlations and micro-stimulation to estimate connectivity strength. We show that connectivity is consistent with a model in which BO-tuned V4 cells send positive feedback in the direction of their preferred figure and negative feedback in the opposite direction. This connectivity scheme can recreate patterns of FBM observed in previous studies. These results provide insights into the cortical connectivity underlying figure-background perception and establish a link between FBM and BO-tuning.