Visual terrain classification can provide crucial and important information for motion control and autonomous navigation for mobile robots in complex terrain environment, becoming an important but challenging task. This paper uses a novel hybrid coding architecture, Deep Filter Banks (DFB), combining stacked denoising sparse autoencoder (SDSAE) and Fisher Vector (FV) for visual terrain classification. Then, we propose a terrain dataset, termed "Terrain8", which is the first publicly available benchmark for visual terrain classification. This dataset contains 2400 terrain images, covering 8 terrain classes with 300 images in each class. Our method achieves superior performance on the Terrain8 dataset. Moreover, we design the framework to deal with terrain videos and carry out the field experiments in arc-legged mobile robot. The field experimental results also indicate the effectiveness of our proposed methods.