Navigating the world requires attention to number; however, sets in real-world contexts are rarely homogeneous or presented in isolation, thus the task of determining what constitutes a relevant set for enumeration can be a difficult one. This contextual ambiguity increases the likelihood that irrelevant sets may bias our ability to accurately track number. In the current study, we investigated whether numerical estimates are influenced by irrelevant set information, such as the size of other present subsets or the number of subsets composing the set. Adult observers were shown brief arrays of dots containing 1 or 2 intermixed subsets, differentiated by color, and were asked to estimate the number of either: (1) 1 of the subsets or (2) the superset (total number of dots). When estimating the size of a subset, numerical estimates were greatly influenced by the size of the other, irrelevant subset, suggesting that the presence of extraneous sets may hinder accurate number judgments. Furthermore, when asked to judge the total number of items (the superset), observers judged supersets comprising 2 subsets as more numerous compared to those comprising only a single subset. Importantly, both trends were apparent even when observers had prior information about the identity of the target set, suggesting adults preattentively parse the world into sets, causing numerical biases that are not under conscious control. Potential explanations for this pattern of results, including simultaneous numerical contrast effects and superset summation, are discussed.