In this paper, we propose a novel deep networks, multi-feature fusion deep networks (MFFDN), based on denoising autoencoder. MFFDN significantly reduces the classification error while giving the interpretability of the hidden-layer feature representation in learning process. Comparing with the traditional denoising autoencoder, MFFDN mainly shows the following advantages: (1) minimally retaining a certain amount of information constrained to a given form about its input; (2) explicitly interpreting the meaning of the feature representation in one hidden layer; (3) enhancing discriminativeness of the whole networks. At last, the experiments analysis on MNIST, CIFAR-10 and SVHN prove the state-of-the-art performance improvement of MFFDN with the advantages minimally retaining information constraint and the interpreted hidden feature representation.