Skip to main content
Log in

Time series prediction using deep echo state networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Artificial neural networks have been used for time series modeling and forecasting in many domains. However, they are often limited in their handling of nonlinear and chaotic data. More recently, reservoir-based recurrent neural net systems, most notably echo state networks (ESN), have made substantial improvements for time series modeling. Their shallow nature lends themselves to an efficient training method, but has limitations on nonstationary, nonlinear chaotic time series, particularly large, multidimensional time series. In this paper, we propose a novel approach for forecasting time series data based on an additive decomposition (AD) applied to the time series as a preprocessor to a deep echo state network. We compare the performance of our method, AD-DeepESN, on popular neural net architectures used for time series prediction. Stationary and nonstationary data sets are used to evaluate the performance of the methods. Our results are compelling, demonstrating that AD-DeepESN has superior performance, particularly on the most challenging time series that exhibit non-stationarity and chaotic behavior compared to existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

References

  1. Bar-Joseph Z, Gerber GK, Gifford DK, Jaakkola TS, Simon I (2003) Continuous representations of time-series gene expression data. J Comput Biol 10(3–4):341–356

    Article  Google Scholar 

  2. Taylor SJ (2007) Modeling financial time series

  3. Gottman JM (1981) Time-series analysis: a comprehensive introduction for social scientists, vol 400. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  4. Billinton R, Chen H, Ghajar R (1996) Time-series models for reliability evaluation of power systems including wind energy. Microelectron Reliab 36(9):1253–1261

    Article  Google Scholar 

  5. Ghil M, Vautard R (1991) Interdecadal oscillations and the warming trend in global temperature time series. Nature 350:324–327

    Article  Google Scholar 

  6. Maqsood I, Khan MR, Abraham A (2004) An ensemble of neural networks for weather forecasting. Neural Comput Appl 13(2):112–122

    Article  Google Scholar 

  7. Taylor JW, Buizza R (2002) Neural network load forecasting with weather ensemble predictions. IEEE Trans Power Syst 17(3):626–632

    Article  Google Scholar 

  8. Lin X, Yang Z, Song Y (2009) Short-term stock price prediction based on echo state networks. Expert Syst Appl 36(3):7313–7317

    Article  Google Scholar 

  9. Bernal A, Fok S, Pidaparthi R (2012) Financial market time series prediction with recurrent neural networks the challenge of time series prediction echo state network implementation, pp 1–5

  10. Jaeger H (2001) The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. German National Research Institute for Computer Science

  11. Zhang PG (2003) Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing. https://doi.org/10.1016/S0925-2312(01)00702-0

    Article  MATH  Google Scholar 

  12. Nason GP (2006) Stationary and non-stationary time series. Stat Volcanol 1994:29

    Google Scholar 

  13. Palachy S (2019) Stationarity in time series analysis. Towards Data Science. https://towardsdatascience.com/stationarity-in-time-series-analysis-90c94f27322

  14. Zaiontz C (2018) Dickey–Fuller test. http://www.real-statistics.com/time-series-analysis/stochastic-processes/dickey-fuller-test/

  15. Boshnakov GN (2010) Introductory time series with R. J Time Ser Anal 3:1. https://doi.org/10.1111/j.1467-9892.2009.00647.x

    Article  Google Scholar 

  16. Hochreiter S (1998) The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int J Uncertainty Fuzziness Knowl Based Syst 6(2):107–116. https://doi.org/10.1142/S0218488598000094

    Article  MathSciNet  MATH  Google Scholar 

  17. Cummins F, Gers FA, Schmidhuber J (2000) Learning to forget: continual prediction with LSTM. Neural Comput 2:850–855. https://doi.org/10.1197/jamia.M2577

    Article  Google Scholar 

  18. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling, pp 1–9. Retrieved from http://arxiv.org/abs/1412.3555

  19. Lukoševičius M, Jaeger H (2009) Reservoir computing approaches to recurrent neural network training. Comput Sci Rev 3(3):127–149. https://doi.org/10.1016/J.COSREV.2009.03.005

    Article  MATH  Google Scholar 

  20. Lukoševičius M (2012) A practical guide to applying Echo State Networks. In: Montavon G, Orr GB, Müller KR (eds) Neural networks: tricks of the trade. Lecture Notes in Computer Science, vol 7700. Springer, Berlin

    Google Scholar 

  21. Gallicchio C, Micheli A, Pedrelli L (2018) Deep Echo State Networks for diagnosis of Parkinson’s disease. Retrieved from http://arxiv.org/abs/1802.06708

  22. Li D, Han M, Wang J (2012) Chaotic time series prediction based on a novel robust echo state network. IEEE Trans Neural Netw Learn Syst 23(5):787–797. https://doi.org/10.1109/TNNLS.2012.2188414

    Article  Google Scholar 

  23. Sheng C, Zhao J, Liu Y, Wang W (2012) Prediction for noisy nonlinear time series by echo state network based on dual estimation. Neurocomputing 82:186–195. https://doi.org/10.1016/J.NEUCOM.2011.11.021

    Article  Google Scholar 

  24. Sun X, Li T, Li Q, Huang Y, Li Y (2017) Deep belief echo-state network and its application to time series prediction. Knowl-Based Syst 130:17–29. https://doi.org/10.1016/J.KNOSYS.2017.05.022

    Article  Google Scholar 

  25. Lin X, Yang Z, Song Y (2009) Short-term stock price prediction based on echo state networks. Expert Syst Appl. https://doi.org/10.1016/j.eswa.2008.09.049

    Article  Google Scholar 

  26. Gallicchio C, Micheli A, Pedrelli L (2017) Deep reservoir computing: a critical experimental analysis. Neurocomputing 268:87–99. https://doi.org/10.1016/j.neucom.2016.12.089

    Article  Google Scholar 

  27. Gallicchio C, Micheli A (2017) Deep Echo State Network (DeepESN): a brief survey, pp 1–13. Retrieved from http://arxiv.org/abs/1712.04323

  28. Ma Q, Shen L, Cottrell GW (2017) Deep-ESN: a multiple projection-encoding hierarchical reservoir computing framework. 14(8), 1–15. Retrieved from http://arxiv.org/abs/1711.05255

  29. Yahoo Finance S&P 500 data. https://finance.yahoo.com/quote/%5EGSPC/history/

  30. Stolfi DH, Alba E, Yao X (2017) Predicting car park occupancy rates in Smart Cities. In: Smart Cities: second international conference, Smart-CT 2017, Spain, 14–16 June 2017, pp 107–117

  31. Weeks E (2015) Chaotic time series analysis. Department of Physics. Emory University. http://www.physics.emory.edu/faculty/weeks/research/tseries1.html

  32. The Pennsylvania State Climatologist. http://www.climate.psu.edu/

  33. Fanaee-T H, Gama J (2015) Event labeling combining ensemble detectors and background knowledge. Prog Artifi Intell 2:113–127

    Article  Google Scholar 

Download references

Acknowledgements

The study was funded in part from the McKenna Environmental Internship Program and Bucknell University Presidential Fellowship and Program for Undergraduate Research.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Brian R. King.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 459 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, T., King, B.R. Time series prediction using deep echo state networks. Neural Comput & Applic 32, 17769–17787 (2020). https://doi.org/10.1007/s00521-020-04948-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-04948-x

Keywords

Navigation