The performance of older adults on neuropsychological (and cognitive) assessment tests can set the perspective on the capabilities of this population in general, hence contributing to ageism. These tests are administered with two implicit assumptions: (a) the tests are a valid gauge of performance in older adults and (b) cognitive abilities decline in older age. In the current chapter, we challenge these assumptions by discussing two potential sources of threat to test validity: the sensory and the social contexts. First, we review findings that age-related decline in performance may reflect, at least in part, a sensory (visual and auditory) rather than a purely cognitive decline. Next, we discuss the negative effect of age-based stereotype threat on older adults’ cognitive performance. Both contexts may be present in daily testing of older adults in the clinic or the lab. Thus, targeting the sensory and social context in neuropsychological assessment may assist in reducing age-bias (leading to ageism) in the scientific, medical and general community.