Metasurfaces have been used to realize optical functions such as focusing and beam steering. They use subwavelength nanostructures to control the local amplitude and phase of light. Here we show that such control could also enable a new function of artificial neural inference. We demonstrate that metasurfaces can directly recognize objects by focusing light from an object to different spatial locations that correspond to the class of the object.