Elsevier

Journal of Hydrology

Volume 515, 16 July 2014, Pages 330-344
Journal of Hydrology

Assessing artificial neural networks and statistical methods for infilling missing soil moisture records

https://doi.org/10.1016/j.jhydrol.2014.04.068Get rights and content

Highlights

  • Assessed statistical and neural network methods to infill missing soil moisture.

  • Found that only nonlinear autoregressive network offer high infilling accuracy.

  • Showed that rough sets method provides a high accurate estimation of soil moisture.

  • The rough sets method offers a pattern-based narrative of soil moisture variation.

  • The rough sets method accounts for seasonality of rank stability of soil moisture.

Summary

Soil moisture information is critically important for water management operations including flood forecasting, drought monitoring, and groundwater recharge estimation. While an accurate and continuous record of soil moisture is required for these applications, the available soil moisture data, in practice, is typically fraught with missing values. There are a wide range of methods available to infilling hydrologic variables, but a thorough inter-comparison between statistical methods and artificial neural networks has not been made. This study examines 5 statistical methods including monthly averages, weighted Pearson correlation coefficient, a method based on temporal stability of soil moisture, and a weighted merging of the three methods, together with a method based on the concept of rough sets. Additionally, 9 artificial neural networks are examined, broadly categorized into feedforward, dynamic, and radial basis networks. These 14 infilling methods were used to estimate missing soil moisture records and subsequently validated against known values for 13 soil moisture monitoring stations for three different soil layer depths in the Yanco region in southeast Australia. The evaluation results show that the top three highest performing methods are the nonlinear autoregressive neural network, rough sets method, and monthly replacement. A high estimation accuracy (root mean square error (RMSE) of about 0.03m3/m3) was found in the nonlinear autoregressive network, due to its regression based dynamic network which allows feedback connections through discrete-time estimation. An equally high accuracy (0.05 m3/m3 RMSE) in the rough sets procedure illustrates the important role of temporal persistence of soil moisture, with the capability to account for different soil moisture conditions.

Introduction

Moisture in the upper layers of the soil is a vital component of the total water balance in the Earth-atmosphere system, playing a crucial role in several hydrological processes. Soil moisture is one of the main factors influencing the partitioning of rainfall into infiltration and runoff (Mahmood, 1996, Thornthwaite, 1961), controlling the exchange of water and energy between the land surface and the atmosphere (Legates et al., 2010, Berg and Mulroy, 2006, Trenberth and Guillemot, 1998, Houser et al., 1998, Reynolds et al., 2002), and the subsurface water drainage that influences the leaching of contaminants to groundwater (Langevin and Panday, 2012, Legates et al., 2010). The reliability of the above mentioned applications usually depends on the availability of a continuous time series of soil moisture record. Typically, soil moisture data acquired through ground (or in situ) measurements have missing values due to equipment malfunction, logger storage overruns, data retrieval problems, and/or severe weather conditions (Dumedah and Coulibaly, 2011, Coulibaly and Evora, 2007). Consequently, the infilling of missing soil moisture values becomes a necessary procedure to generate a continuous time series record.

Several studies have infilled hydrologic variables including precipitation (Mwale et al., 2012, Nkuna and Odiyo, 2011, Coulibaly and Evora, 2007, French et al., 1992, Luck et al., 2000, Abebe et al., 2000, ASCE Task Committee on Application of Artificial Neural Networks in Hydrology, 2000b), streamflow (Mwale et al., 2012, Ng and Panu, 2010, Ng et al., 2009, Elshorbagy et al., 2000, ASCE Task Committee on Application of Artificial Neural Networks in Hydrology, 2000b), evapotranspiration (Abudu et al., 2010), air temperature (Coulibaly and Evora, 2007, Schneider, 2001), and soil moisture (Gao et al., 2013, Wang et al., 2012, Dumedah and Coulibaly, 2011). The infilling methods employed in the above studies ranged from statistical methods (Gao et al., 2013, Wang et al., 2012, Dumedah and Coulibaly, 2011) to artificial neural networks (Mwale et al., 2012, Nkuna and Odiyo, 2011, Coulibaly and Evora, 2007), with varying levels of accuracy. While several studies have explored different infilling approaches, very few studies have been undertaken to actually reconstruct soil moisture records using both statistical and artificial neural network methods. As a result, this study investigates 5 statistical and 9 artificial neural network methods, a total of 14 methods to estimate missing soil moisture records. The soil moisture monitoring network located in the Yanco region of southeast Australia (Smith et al., 2012) is used as the demonstration data set.

The statistical methods include monthly replacement, weighted Pearson correlation, station relative difference, and a weighted merger of the three statistical methods. Moreover, a method based on the concept of rough sets (Pawlak, 1997, Pawlak et al., 1995, Pawlak, 1982) was used to determine patterns of temporal stability of soil moisture to account for different moisture conditions. The artificial neural networks (ANNs) evaluated in this stud are broadly categorized into feedforward group, dynamic group and radial basis group. Detailed descriptions for the statistical and ANN methods are provided in the methods section. The selected approaches constitute a varied range of methodologies to facilitate a comprehensive inter-comparison between a range of statistical and ANNs with the potential to identify high performing methods to infill missing soil moisture. The infilling methods have been evaluated for their estimation accuracy across 13 soil moisture monitoring stations independently at three different soil layer depths in the Yanco area. Moreover, an evaluation of the soil moisture across the 13 monitoring stations in space and their persistence of relative moisture conditions over several time periods was demonstrated. These space–time distributions are presented for the entire period of the chosen soil moisture data, and also on a month-by-month basis.

Section snippets

Study area and soil moisture data

The Yanco area shown in Fig. 1 is a 60km×60km area, located in the western plains of the Murrumbidgee Catchment in southeast Australia where the topography is flat with very few geological outcroppings. Soil texture types are predominantly sandy loams, scattered clays, red brown earths, transitional red brown earth, sands over clay, and deep sands. According to the Digital Atlas of Australian Soils, the dominant soil is characterized by plains with domes, lunettes, and swampy depressions, and

Methods for infilling missing soil moisture records

Using the raw soil moisture data, a complete data set was retrieved by removing all periods with missing records, such that all soil moisture records were temporally consistent (or common) to all stations for a specific soil layer (Dumedah and Coulibaly, 2011, Coulibaly and Evora, 2007). In other words, the complete data set is spatially complete in a way that each record in the data set at any one station has corresponding records available across the remaining 12 stations for the specified

Results and discussion

The application of the 14 infilling methods is presented in two stages. The first stage illustrates the spatial–temporal variation of soil moisture for the 13 monitoring stations; whereas the second stage presents the evaluation of the infilling methods to estimate the soil moisture records. The infilling methods are evaluated in three phases: surface soil layer, second soil layer, and the third soil layer independently.

Conclusions

This study has evaluated 14 infilling methods including artificial neural network and statistical techniques, to estimate missing soil moisture records at 13 monitoring stations independently for three different soil layers. An evaluation of the estimated soil moisture values against known records showed that the top three highest performing methods are the nonlinear autoregressive neural network, the rough sets method, and the monthly replacement. The high estimation accuracy (RMSE of 0.03 m3/m

Acknowledgements

This work was supported by funding from the Australian Research Council (DP0879212). The program for the infilling methods used herein is available upon request from the first author at: [email protected].

References (53)

  • Z. Pawlak

    Rough set approach to knowledge-based decision support

    Eur. J. Oper. Res.

    (1997)
  • C. Rüdiger et al.

    Towards a general equation for frequency domain reflectometers

    J. Hydrol.

    (2010)
  • G. Wang et al.

    A three-dimensional gap filling method for large geophysical datasets: application to global satellite soil moisture observations

    Environ. Model. Softw.

    (2012)
  • A. Abebe et al.

    Application of adaptive fuzzy rule – based models for reconstruction of missing precipitation events

    Hydrol. Sci. J.

    (2000)
  • S. Abudu et al.

    Infilling missing daily evapotranspiration data using neural networks

    J. Irrig. Drain. Eng.

    (2010)
  • M. Ananda Rao et al.

    Neural Networks: Algorithms and Applications

    (2003)
  • ASCE Task Committee on Application of Artificial Neural Networks in Hydrology

    Artificial neural networks in hydrology I: preliminary concepts

    J. Hydrol. Eng.

    (2000)
  • ASCE Task Committee on Application of Artificial Neural Networks in Hydrology

    Artificial neural networks in hydrology II: hydrologic applications

    J. Hydrol. Eng.

    (2000)
  • M.H. Beale et al.

    Neural Network Toolbox: Users Guide

    (2012)
  • A.A. Berg et al.

    Streamflow predictability given macro-scale estimates of the initial soil moisture status

    Hydrol. Sci. J.

    (2006)
  • L. Brocca et al.

    Spatial–temporal variability of soil moisture and its estimation across scales

    Water Resour. Res.

    (2010)
  • D.S. Broomhead et al.

    Multivariable functional interpolation and adaptive networks

    Complex Syst.

    (1988)
  • T.W.S. Chow et al.

    Nonlinear autoregressive integrated neural network del for short-term load forecasting

    IEEE Proc. – Gener., Trans. Distrib

    (1996)
  • G. Dumedah et al.

    Evaluation of statistical methods for infilling missing values in high-resolution soil moisture data

    J. Hydrol.

    (2011)
  • G. Dumedah et al.

    Minimizing the effects of inaccurate sediment description in borehole data using rough set theory and transition probability

    J. Geogr. Syst.

    (2008)
  • G. Dumedah et al.

    Minimizing effects of scale distortion for spatially grouped census data using rough sets

    J. Geogr. Syst.

    (2008)
  • Cited by (26)

    • In-use measurement of ultrasonic flowmeter based on Machine learning

      2023, Measurement: Journal of the International Measurement Confederation
    • Comparison of gap-filling techniques applied to the CCI soil moisture database in Southern Europe

      2021, Remote Sensing of Environment
      Citation Excerpt :

      There were hardly any differences between the RMSE and cRMSE due to the low bias; its medians range from 0.036 to 0.026 m3m−3 for SVM with PCA and SVM-Tdh, respectively. These results are in line with those reported by Dumedah et al. (2014). The simplest methods resulted in the lowest accuracy, and the best results were obtained using a nonlinear autoregressive neural network, comparable to SVM-Tdh.

    • Application of artificial neural network modeling techniques to signal strength computation

      2021, Heliyon
      Citation Excerpt :

      Neural networks have been very effective for modeling and for characterisation of complex systems for a number of applications [29, 30]. The successful application of ANN to a number of scientific and engineering problems abounds [31, 32, 33, 34, 35], including the prediction of signal strength in the VHF and UHF bands [2, 36, 37, 38, 39, 40, 41]. One of the most widely used type of ANN is the feedforward network.

    • Using machine learning to produce near surface soil moisture estimates from deeper in situ records at U.S. Climate Reference Network (USCRN) locations: Analysis and applications to AMSR-E satellite validation

      2016, Advances in Water Resources
      Citation Excerpt :

      In addition to isolating locations that are consistently wetter or drier than the spatial average within a test watershed, these analyses can provide indices for temporal patterns of soil moisture to be utilized in subsequent models (Brocca et al., 2010). While the current analysis aims to fill missing soil moisture values in the vertical dimension, other works have deployed various regression techniques (e.g. Dumedah et al., 2011) or artificial neural networks (Dumedah et al., 2014) to fill missing values in the temporal dimension. Meng and Quiring (2008) examined model accuracy for replacing soil moisture at in situ stations.

    View all citing articles on Scopus
    View full text