In areas of machine learning such as cognitive modeling or recommendation, user feedback is usually context-dependent. For instance, a website might provide a user with a set of recommendations and observe which (if any) of the links were clicked by the user. Similarly, there is growing interest in the so-called "odd-one-out" learning setting, where human participants are provided with a basket of items and asked which is the most dissimilar to the others. In both of those cases, the presence of all the items in the basket can influence the final decision. In this article, we consider a classification task where each input consists of three items (a triplet), and the task is to predict which of the three will be selected. Our aim is not only to return accurate predictions for the selection task, but also to additionally provide interpretable feature representations for both the context and for each individual item. To achieve this, we introduce CARE, a specialized neural network architecture that yields Context-Aware REpresentations of items based on observations of triplets of items alone. We demonstrate that, in addition to achieving state-of-the-art performance at the selection task, our model can produce meaningful representations both for each item, as well for each context (triplet of items). This is done using only triplet responses: CARE has no access to supervised item-level information. In addition, we prove parameter counting generalization bounds for our model in the i.i.d. setting, demonstrating that the apparent sample sparsity arising from the combinatorially large number of possible triplets is no obstacle to efficient learning.