Mammalian neural progenitor cells divide asymmetrically to self-renew and produce a neuron by segregating cytosolic Numb proteins primarily to one daughter cell. Numb signaling specifies progenitor over neuronal fates but, paradoxically, also promotes neuronal differentiation. Here we report that ACBD3 is a Numb partner in cell-fate specification. ACBD3 and Numb proteins interact through an essential Numb domain, and the respective loss- and gain-of-function mutant mice share phenotypic similarities. Interestingly, ACBD3 associates with the Golgi apparatus in neurons and interphase progenitor cells but becomes cytosolic after Golgi fragmentation during mitosis, when Numb activity is needed to distinguish the two daughter cells. Accordingly, cytosolic ACBD3 can act synergistically with Numb to specify cell fates, and its continuing presence during the progenitor cell cycle inhibits neuron production. We propose that Golgi fragmentation and reconstitution during cell cycle differentially regulate Numb signaling through changes in ACBD3 subcellular distribution and represent a mechanism for coupling cell-fate specification and cell-cycle progression.