Abstract
Artificial neural networks have been used for time series modeling and forecasting in many domains. However, they are often limited in their handling of nonlinear and chaotic data. More recently, reservoir-based recurrent neural net systems, most notably echo state networks (ESN), have made substantial improvements for time series modeling. Their shallow nature lends themselves to an efficient training method, but has limitations on nonstationary, nonlinear chaotic time series, particularly large, multidimensional time series. In this paper, we propose a novel approach for forecasting time series data based on an additive decomposition (AD) applied to the time series as a preprocessor to a deep echo state network. We compare the performance of our method, AD-DeepESN, on popular neural net architectures used for time series prediction. Stationary and nonstationary data sets are used to evaluate the performance of the methods. Our results are compelling, demonstrating that AD-DeepESN has superior performance, particularly on the most challenging time series that exhibit non-stationarity and chaotic behavior compared to existing methods.
Similar content being viewed by others
References
Bar-Joseph Z, Gerber GK, Gifford DK, Jaakkola TS, Simon I (2003) Continuous representations of time-series gene expression data. J Comput Biol 10(3–4):341–356
Taylor SJ (2007) Modeling financial time series
Gottman JM (1981) Time-series analysis: a comprehensive introduction for social scientists, vol 400. Cambridge University Press, Cambridge
Billinton R, Chen H, Ghajar R (1996) Time-series models for reliability evaluation of power systems including wind energy. Microelectron Reliab 36(9):1253–1261
Ghil M, Vautard R (1991) Interdecadal oscillations and the warming trend in global temperature time series. Nature 350:324–327
Maqsood I, Khan MR, Abraham A (2004) An ensemble of neural networks for weather forecasting. Neural Comput Appl 13(2):112–122
Taylor JW, Buizza R (2002) Neural network load forecasting with weather ensemble predictions. IEEE Trans Power Syst 17(3):626–632
Lin X, Yang Z, Song Y (2009) Short-term stock price prediction based on echo state networks. Expert Syst Appl 36(3):7313–7317
Bernal A, Fok S, Pidaparthi R (2012) Financial market time series prediction with recurrent neural networks the challenge of time series prediction echo state network implementation, pp 1–5
Jaeger H (2001) The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. German National Research Institute for Computer Science
Zhang PG (2003) Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing. https://doi.org/10.1016/S0925-2312(01)00702-0
Nason GP (2006) Stationary and non-stationary time series. Stat Volcanol 1994:29
Palachy S (2019) Stationarity in time series analysis. Towards Data Science. https://towardsdatascience.com/stationarity-in-time-series-analysis-90c94f27322
Zaiontz C (2018) Dickey–Fuller test. http://www.real-statistics.com/time-series-analysis/stochastic-processes/dickey-fuller-test/
Boshnakov GN (2010) Introductory time series with R. J Time Ser Anal 3:1. https://doi.org/10.1111/j.1467-9892.2009.00647.x
Hochreiter S (1998) The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int J Uncertainty Fuzziness Knowl Based Syst 6(2):107–116. https://doi.org/10.1142/S0218488598000094
Cummins F, Gers FA, Schmidhuber J (2000) Learning to forget: continual prediction with LSTM. Neural Comput 2:850–855. https://doi.org/10.1197/jamia.M2577
Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling, pp 1–9. Retrieved from http://arxiv.org/abs/1412.3555
Lukoševičius M, Jaeger H (2009) Reservoir computing approaches to recurrent neural network training. Comput Sci Rev 3(3):127–149. https://doi.org/10.1016/J.COSREV.2009.03.005
Lukoševičius M (2012) A practical guide to applying Echo State Networks. In: Montavon G, Orr GB, Müller KR (eds) Neural networks: tricks of the trade. Lecture Notes in Computer Science, vol 7700. Springer, Berlin
Gallicchio C, Micheli A, Pedrelli L (2018) Deep Echo State Networks for diagnosis of Parkinson’s disease. Retrieved from http://arxiv.org/abs/1802.06708
Li D, Han M, Wang J (2012) Chaotic time series prediction based on a novel robust echo state network. IEEE Trans Neural Netw Learn Syst 23(5):787–797. https://doi.org/10.1109/TNNLS.2012.2188414
Sheng C, Zhao J, Liu Y, Wang W (2012) Prediction for noisy nonlinear time series by echo state network based on dual estimation. Neurocomputing 82:186–195. https://doi.org/10.1016/J.NEUCOM.2011.11.021
Sun X, Li T, Li Q, Huang Y, Li Y (2017) Deep belief echo-state network and its application to time series prediction. Knowl-Based Syst 130:17–29. https://doi.org/10.1016/J.KNOSYS.2017.05.022
Lin X, Yang Z, Song Y (2009) Short-term stock price prediction based on echo state networks. Expert Syst Appl. https://doi.org/10.1016/j.eswa.2008.09.049
Gallicchio C, Micheli A, Pedrelli L (2017) Deep reservoir computing: a critical experimental analysis. Neurocomputing 268:87–99. https://doi.org/10.1016/j.neucom.2016.12.089
Gallicchio C, Micheli A (2017) Deep Echo State Network (DeepESN): a brief survey, pp 1–13. Retrieved from http://arxiv.org/abs/1712.04323
Ma Q, Shen L, Cottrell GW (2017) Deep-ESN: a multiple projection-encoding hierarchical reservoir computing framework. 14(8), 1–15. Retrieved from http://arxiv.org/abs/1711.05255
Yahoo Finance S&P 500 data. https://finance.yahoo.com/quote/%5EGSPC/history/
Stolfi DH, Alba E, Yao X (2017) Predicting car park occupancy rates in Smart Cities. In: Smart Cities: second international conference, Smart-CT 2017, Spain, 14–16 June 2017, pp 107–117
Weeks E (2015) Chaotic time series analysis. Department of Physics. Emory University. http://www.physics.emory.edu/faculty/weeks/research/tseries1.html
The Pennsylvania State Climatologist. http://www.climate.psu.edu/
Fanaee-T H, Gama J (2015) Event labeling combining ensemble detectors and background knowledge. Prog Artifi Intell 2:113–127
Acknowledgements
The study was funded in part from the McKenna Environmental Internship Program and Bucknell University Presidential Fellowship and Program for Undergraduate Research.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Kim, T., King, B.R. Time series prediction using deep echo state networks. Neural Comput & Applic 32, 17769–17787 (2020). https://doi.org/10.1007/s00521-020-04948-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-020-04948-x