Since 1972, nearly continuous observations of coronal holes and their associated photospheric magnetic fields have been made using a variety of satellite and ground-based equipment. The results of comparisons of these observations are reviewed and it is demonstrated that the structure and evolution of coronal holes is basically governed by the large-scale distribution of photospheric magnetic flux. Non-polar holes form in the decaying remnants of bipolar magnetic regions in areas with a large-scale flux imbalance. There is strong indirect evidence that the magnetic field in coronal holes is always open to interplanetary space but not all open-field regions have associated coronal holes. The well-observed declining phase of the last solar cycle was characterized by stable magnetic field and coronal hole patterns which were associated with recurrent, high-speed wind streams and interplanetary magnetic field patterns at the Earth. The ascending phase of the current cycle has been characterized by transient magnetic field and coronal hole patterns which tend to occur at high solar latitudes. This shift in magnetic field and coronal hole patterns has resulted in a less obvious and more complicated association with high-speed wind streams at the Earth.