In the early 1960s, Hubel and Wiesel reported the first physiological description of cells in cat primary visual cortex. They distinguished two main cell types: simple cells and complex cells. Based on their distinct response properties, they suggested that the two cell types could represent two consecutive stages in receptive-field construction. Since the 1960s, new experimental and computational evidence provided serious alternatives to this hierarchical model. Parallel models put forward the idea that both simple and complex receptive fields could be built in parallel by direct geniculate inputs. Recurrent models suggested that simple cells and complex cells may not be different cell types after all. To this day, a consensus among hierarchical, parallel, and recurrent models has been difficult to attain; however, the circuitry used by all models is becoming increasingly similar. The authors review theoretical and experimental evidence for each line of models emphasizing their strengths and weaknesses.