AbstractThanks to the introduction of non‐fullerene acceptors, efficiencies of organic photovoltaics are now approaching 20%. Closing the gap with inorganic photovoltaics requires minimizing voltage losses without penalizing charge extraction, for which microstructure control is crucial. However, the complex interplay between microstructure and charge generation, recombination, and extraction has so far not been unraveled. Here, a systematic study linking device performance to distinct microstructural features via machine learning is presented. Building bi‐layer devices allows to separately study the influence of aggregation and disorder on the energies and lifetimes of bulk and interfacial states. Unambiguous assignments of specific structural motifs to the device photophysics are thus possible. It is found that the control of aggregation‐caused quenching is decisive for the exciton splitting efficiency and thus the carrier generation. Furthermore, the static disorder at the donor–acceptor interface controls the nonradiative recombination by shifting the excited state population from the bulk toward the interface. Finally, the amount of disorder in the bulk is found decisive for charge extraction. The finding that charge generation, recombination, and extraction are controlled by distinct structural features, is the key to optimizing these motifs independently, which will pave the way for organic photovoltaics toward the detailed balance limit.