Understanding how a machine learns is a pressing topic as machine learning becomes more complex enabled by more powerful computers. This paper presents a visualization of neural networks to make them trackable during the operation of learning for pattern recognition, as well as testing for patterns. Specifically, our implementation includes fully connected neural networks, convolutional neural networks, and networks with memories. This will help us understand the insight of neural networks for pattern recognition to ensure full human control of the machines and to eliminate public's concern of recent leap in AI and machine learning. The visualization also helps to measure and identify performance bottleneck for future improvement.