The brains of humans and other animals are asymmetrically organized, but we still know little about the ontogenetic and neural fundaments of lateralizations. Here, we review the current state of understanding about the role of genetic and non-genetic factors for the development of neural and behavioral asymmetries in vertebrates. At the genetic level, the Nodal signaling cascade is of central importance, but several other genetic pathways have been discovered to also shape the lateralized embryonic brain. Studies in humans identified several relevant genes with mostly small effect sizes but also highlight the extreme importance of non-genetic factors for asymmetry development. This is also visible in visual asymmetry in birds, in which genes only affect embryonic body position, while the resulting left-right difference of visual stimulation shapes visual pathways in a lateralized way. These and further studies in zebrafish and humans highlight that the many routes from genes to asymmetries of function run through left-right differences of neural pathways. They constitute the lateralized blueprints of our perception, cognition, and action.