Affine moment invariant (AMI) is a kind of hand-crafted image feature, which is invariant to affine transformations. This property is precisely what the standard convolution neural network (CNN) is difficult to achieve. In this letter, we present a kind of network architecture to introduce AMI into CNN, which is called AMI-Net. We achieved this by calculating AMI on the feature maps of the hidden layers. These AMIs will be concatenated with the standard CNN's FC layer to determine the network's final output. By calculating AMI on the feature maps, we can not only extend the dimension of AMIs, but also introduce affine transformation invariant into CNN. Two network architectures and training strategies of AMI-Net are illuminated, one is two-stage, and the other is end-to-end. To prove the effectiveness of the AMI-Net, several experiments have been conducted on common image datasets, MNIST, MNIST-rot, affNIST, SVHN, and CIFAR-10. By comparing with the corresponding standard CNN, respectively, we verify the validity of AMI-net.