We propose a fast Bag-of-Words (BoW) method for image classification, inspired by the mechanism that arrangement of neurons in visual cortex can preserve the topology of mapping from inputs, and the fact that human brain can retrieve information almost instantly. We propose algorithms for accelerating both Self-Organizing Map (SOM) training and BoW coding. First, we modify the traditional SOM based on the matrix factorization form of K-means. Utilizing the topology-preserving property of dictionary learned by SOM, the coding process of BoW can be accelerated by fast search of k-nearest neighbor codewords in the grid of SOM dictionary. We evaluate the proposed method in different coding scenarios for image classification task on MNIST and CIFAR-10 datasets. The results show that the proposed method accelerates BoW classification greatly with little loss of classification accuracy.