We address the problem of estimating and removing localized image blur, as it for example arises from moving objects in a scene, or when the depth of field is insufficient to sharply render all objects of interest. Unlike the case of camera shake, such blur changes abruptly at the object boundaries. To cope with this, we propose an automated sharp image recovery method that simultaneously determines blurred regions and estimates their responsible blur kernels. To address a wide range of different scenarios, our model is not restricted to a discrete set of candidate blurs, but allows for arbitrary, non-parametric blur kernels. Moreover, our approach does not require specialized hardware, an alpha matte, or user annotation of the blurred region. Unlike previous methods, we show that localized blur estimation can be accomplished by incorporating a pixel-wise latent variable to indicate the active blur kernel. Furthermore, we generalize the marginal likelihood technique of blind deblurring to the case of localized blur. Specifically, we integrate out the latent image derivatives to permit marginal density estimates of both blur kernels and their regions of influence. We obtain sharp images in applications to both object motion blur and defocus blur removal. Quantitative results on two novel datasets as well as qualitative results comparing to a range of specialized methods demonstrate the versatility and effectiveness of our non-parametric approach.