The dominant view in the social sciences is that, on average, exposure to adversity impairs cognitive abilities, such as planning, goal-directed behavior, and self-control. This view, however, is incomplete: people might also develop intact, or even enhanced, abilities for solving challenges in adverse conditions. Understanding these abilities is critical to close achievement gaps in education and job contexts. Here, we will investigate the associations between two types of adversity—exposure to violence and poverty—and two cognitive abilities that are central to successful goal-directed behavior and self-control: switching attention and ignoring distractions. The central question is: to what extent does adversity impair/enhance abilities, and to what extent does it shape other processes—such as being more cautious? We will answer these questions using a well-developed analytic technique, Drift Diffusion Modeling. The results will have implications for fundamental science and can inform policy and interventions aimed at alleviating the negative impact of adversity.