In this paper we demonstrate that a recently introduced shape descriptor, the "shape context", can be used to quickly prune a search for similar shapes. Our representation for a shape is a discrete set of n points sampled from its internal and external contours. For each of these points, the shape context is a histogram of the relative positions of the n - 1 remaining points. We present two methods for rapid shape retrieval: one that does comparisons based on a small number of shape contexts and another that uses vector quantization in the space of shape contexts. We verify the discriminative power of these methods with tests on the Columbia (COIL-100) 3D object database and the Snodgrass and Vanderwart line drawings. The shape context-based methods are shown to quickly produce an accurate shortlist of candidates suitable for a more exact matching engine in spite of pose variation and occlusion.