Skip to main content
SpringerLink
Log in
Book cover

International Conference on Artificial Intelligence and Soft Computing

ICAISC 2016: Artificial Intelligence and Soft Computing pp 136–147Cite as

Estimation of Deep Neural Networks Capabilities Using Polynomial Approach

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 9692))

Abstract

Currently very popular trend in artificial intelligence is the use of deep neural networks. The power of such networks are very large, but the main difficulty is learning these networks. The article presents a analysis of deep neural network nonlinearity with polynomial approximation of neuron activation functions. It is shown that nonlinearity grows exponentially with the depth of the neural network. The effectiveness of the approach is demonstrated by several experiments.

This work was supported by the National Science Centre, Cracow, Poland under Grant No. 2013/11/B/ST6/01337.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Deng, L., Li, X.: Machine learning paradigms in speech recognition: An overview. IEEE Trans. Audio Speech Lang. Process. 21(5), 1060–1089 (2013)

    Article  Google Scholar 

  2. Lee, H., Grosse, R., Ranganath, R., Ng, A.: Unsupervised learning of hierarchical representations with convolutional deep belief networks. Commun. Assoc. Comput. Mach. (ACM) 54(10), 95–103 (2011)

    Google Scholar 

  3. Wilamowski, B.M.: Neural network architectures and learning algorithms - how not to be frustrated with neural networks. IEEE Ind. Electron. Mag. 3(4), 56–63 (2009)

    Article  Google Scholar 

  4. Hunter, D., Yu, H., Pukish, M.S., Kolbusz, J., Wilamowski, B.M.: Selection of proper neural network sizes and architectures–A comparative study. IEEE Trans. Industr. Inf. 8, 228–240 (2012)

    Article  Google Scholar 

  5. Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–127 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  6. Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Unc. Fuzz. Knowl. Based Syst. 06, 107 (1998)

    Article  MATH  Google Scholar 

  7. Erhan, D., Bengio, Y., Courvelle, A., Manzagol, P., Vencent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res. 9, 201–208 (2010)

    MathSciNet  MATH  Google Scholar 

  8. Schmidhuber, J.: Deep learning in neural networks: An overview. Neural Netw. 61, 85–117 (2015)

    Article  Google Scholar 

  9. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  10. Olas, T., Mleczko, W.K., Nowicki, R.K., Wyrzykowski, R., Krzyzak, A.: Adaptation of RBM learning for intel MIC architecture. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) Artificial Intelligence and Soft Computing. LNCS, vol. 9119, pp. 90–101. Springer, Heidelberg (2015)

    Chapter  Google Scholar 

  11. Wilamowski, B.M., Hunter, D., Malinowski, A.: Solving parity-N problems with feedforward neural networks. In: Proceedings of the 2003 IEEE IJCNN, pp. 2546–2551. IEEE Press (2003)

    Google Scholar 

  12. Xu, Y.: An experimental study on speech enhancement based on deep neural networks. IEEE Signal Process. Lett. 21(1), 65–68 (2011)

    Article  Google Scholar 

  13. Gao, J., et al.: Modeling interestingness with deep neural networks. U.S. Patent No. 20,150,363,688, 17 December 2015

    Google Scholar 

  14. Erhan, D., et al.: Scalable object detection using deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)

    Google Scholar 

  15. Zhang, X., et al.: Improving deep neural network acoustic models using generalized maxout networks. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE (2014)

    Google Scholar 

  16. Montufar, G.F., et al.: On the number of linear regions of deep neural networks. In: Advances in Neural Information Processing Systems (2014)

    Google Scholar 

  17. Oquab, M., et al.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)

    Google Scholar 

  18. Szegedy, C., Toshev, A., Erhan, D.: Deep neural networks for object detection. In: Advances in Neural Information Processing Systems (2013)

    Google Scholar 

  19. Wilamowski, B.M., Yu, H.: Neural network learning without backpropagation. IEEE Trans. Neural Netw. 21(11), 1793–1803 (2010)

    Article  Google Scholar 

  20. Wilamowski, B.M., Yu, H.: NNT - Neural Networks Trainer. http://www.eng.auburn.edu/wilambm/nnt/. Accessed 15 January 2016

  21. Alvarez-Sanchez, J.R.: Injecting knowledge into the solution of the two-spiral problem. Neural Comput. Appl. 8, 265–272 (1999)

    Article  Google Scholar 

  22. Różycki, P., Kolbusz, J., Bartczak, T., Wilamowski, B.M.: Using parity-N problems as a way to compare abilities of shallow, very shallow and very deep architectures. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) Artificial Intelligence and Soft Computing. LNCS, vol. 9119, pp. 112–122. Springer, Heidelberg (2015)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Information Technology and Management in Rzeszow, Sucharskiego 2, 35-225, Rzeszow, Poland

    Pawel Rozycki, Janusz Kolbusz & Roman Korostenskyi

  2. Auburn University, Auburn, AL, 36849-5201, USA

    Bogdan M. Wilamowski

Authors
  1. Pawel Rozycki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Janusz Kolbusz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Roman Korostenskyi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bogdan M. Wilamowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pawel Rozycki .

Editor information

Editors and Affiliations

  1. Częstochowa University of Technology, Czestochowa, Poland

    Leszek Rutkowski

  2. Częstochowa University of Technology, Czestochowa, Poland

    Marcin Korytkowski

  3. Częstochowa University of Technology, Czestochowa, Poland

    Rafał Scherer

  4. AGH University of Science and Technology, Krakow, Poland

    Ryszard Tadeusiewicz

  5. University of California, Berkeley, California, USA

    Lotfi A. Zadeh

  6. University of Louisville, Louisville, Kentucky, USA

    Jacek M. Zurada

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Rozycki, P., Kolbusz, J., Korostenskyi, R., Wilamowski, B.M. (2016). Estimation of Deep Neural Networks Capabilities Using Polynomial Approach. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L., Zurada, J. (eds) Artificial Intelligence and Soft Computing. ICAISC 2016. Lecture Notes in Computer Science(), vol 9692. Springer, Cham. https://doi.org/10.1007/978-3-319-39378-0_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-39378-0_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-39377-3

  • Online ISBN: 978-3-319-39378-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation