A Novel Hybrid Approach for Combining Deep and Traditional Neural Networks

Abstract

Over last fifty years, Neural Networks (NN) have been important and active models in machine learning and pattern recognition. Among different types of NNs, Back Propagation (BP) NN is one popular model, widely exploited in various applications. Recently, NNs attract even more attention in the community because a deep learning structure (if appropriately adopted) could significantly improve the learning performance. In this paper, based on a probabilistic assumption over the output neurons, we propose a hybrid strategy that manages to combine one typical deep NN, i.e., Convolutional NN (CNN) with the popular BP. We present the justification and describe the detailed learning formulations. A series of experiments validate that the hybrid approach could largely improve the accuracy for both CNN and BP on two large-scale benchmark data sets, i.e., MNIST and USPS. In particular, the proposed hybrid method significantly reduced the error rates of CNN and BP respectively by 11.72% and 28.89% on MNIST.