Neutron imaging is an invaluable noninvasive technique for exploring new science and assisting industrial manufacture. However, state-of-the-art neutron facilities are extremely expensive and inconvenient to access, while the flux of portable neutron sources is not strong enough to form even a static image within an acceptable time frame. It is hard to obtain images with both high spatial resolution and energy resolution together. Here, based on classical amplitude modulation, we demonstrate single-pixel imaging with neutrons with specially designed masks and, further, obtain energy-selective images with a spallation neutron source. Images of real complex objects with 100 μm spatial resolution and 10 μs time resolution (corresponding to 0.4% at 1 Å) have been obtained using a 3He single-pixel detector. Even when the neutron counts in the detector plane were lowered to 1000 per modulation pattern on average, a clear image was still obtained. The experimental setup is simple, inexpensive and easy to operate, thus our scheme points to a new path for neutron imaging, especially for portable radioactive neutron sources of low intensity, which should be of great benefit for diagnostic analysis in biology, materials science, and industrial processes.