Working memory (WM) is often tested through immediate serial recall of word lists. Performance in such tasks is negatively influenced by phonological similarity: People more often get the order of words wrong when they are phonologically similar to each other (e.g., cat, fat, mat). This phonological-similarity effect shows that phonology plays an important role for the representation of serial order in these tasks. By contrast, semantic similarity usually does not impact performance negatively. To resolve and understand this discrepancy, we tested the effects of phonological and semantic similarity for the retention of positional information in WM. Across six experiments (all Ns = 60 young adults), we manipulated between-item semantic and phonological similarity in tasks requiring participants to form and maintain new item-context bindings in WM. Participants were asked to retrieve items from their context, or the contexts from their item. For both retrieval directions, phonological similarity impaired WM for item-context bindings across all experiments. Semantic similarity did not. These results demonstrate that WM encodes phonological and semantic information differently. We propose a WM model accounting for semantic-similarity effects in WM, in which semantic knowledge supports WM through activated long-term memory.