Associative recognition and recall depend on structures in the medial temporal lobes (MTLs). There is disagreement about whether associative memory is functionally heterogeneous, whether it is functionally distinct from intra-item associative memory and how the MTLs contribute to this kind of memory. Despite an increase in research on associative memory, work has lacked a theoretical framework to guide design and interpretation of studies. One view provides a useful framework by postulating that associative memories differ in the degree to which their informational components converge in MTL structures that create familiarity-supporting or recollection-supporting memory representations. Using this framework, we consider psychological, lesion and functional imaging evidence, highlighting how informational convergence in the brain underlies the associative nature of both memory and perception.