INTRODUCTION

A quantitative assessment of changes in the content and composition of soil organic carbon potentially caused by a change in the types of land use and tillage practices requires convenient and cost-effective methods for measuring and monitoring. The standard analytical procedures may be very laborious and expensive, especially in the case of numerous samples [49]. The advance in the area of proximal soil sensing, in particular, soil spectroscopy, as well as instrumental engineering are also beneficial for assessment of soil properties [71].

Soil spectroscopy (both laboratory and field) is regarded as an alternative (not a substitute but a supplement) aimed at improvement the traditional methods for analysis of soil properties, as “dry chemistry”. This is a rapid, precise, environmentally safe, reproducible, and cost-effective analytical method for proximal soil sensing [35, 46, 69]. Soil spectroscopy can also form the background for further soil identification and recognition using satellite remote data: the models are first calibrated using the laboratory or in situ field spectrometry data to proceed to satellite data for the most informative wavelengths (in an ideal case, to hyperspectral data) to construct map models that reflect the spatial differentiation of the target soil properties at the of level of a field. This synergistic analysis is referred to as bottom to top [16, 17].

Specific spectral features were comprehensively studied with the use of aerial methods in soil science starting from the 1950s by Andronikov [1], Liverovskii [6], Tolchel’nikov [11], and Karmanov [3, 4]. It has been shown that different soils, minerals, and plant species reflect, absorb, and emit electromagnetic waves of different spectral regions and, consequently, each soil has its own “imprint” depending on the composition and state. At approximately the same time, Kononova [5] headed the pilot testing of spectroscopy approach in the ultraviolet and visible regions of the spectrum and succeeded in detecting the specific humus formation features in the soils of different zones. Orlov [8] developed a classification of the reflectance spectra of main soils and their genetic horizons. The first studies in soil spectroscopy abroad were conducted in the 1970s–1980s [23, 83] and it took additional 30 years for soil spectroscopy to become a hot topic [41].

The main used spectroscopy methods are visible–near infrared (Vis-NIR), infrared (NIR), and mid-infrared (MIR) spectroscopies; the last includes Fourier transform infrared (FTIR) and Raman spectroscopies.

According to the scientific literature, the use of MIR spectroscopy (2500–25 000 nm or 4000–400 cm–1) for the assessment and prediction of soil properties overperforms Vis-NIR spectroscopy [12, 54, 60], which is associated with a more distinctly pronounced fundamental absorption bands of chemical bonds (N–H, C–H, and C–O) in the midrange of wavelengths that result from stretching and bending vibrations [62]. The absorption bands in this spectral range originate from the oscillations of individual atoms or atomic groups and the rotation of the overall molecule. The absorption bands in the Vis-NIR region are the result of the overtones (and their combinations) of the mentioned oscillations, are not so distinct (smaller and wider), overlap, and are less distinguishable. In particular, soil organic matter has a weak absorption band in the region of 1650–1700 nm (NIR), determined by the first overtone of the bending(fundamental) vibrations of methyl (–CH2) and methylene (=CH2) groups in the region of 3400 nm (MIR). However, MIR spectroscopy almost has not been implemented as portable devices applicable under field conditions. A lower cost of Vis-NIR spectroscopy, making it more available, is another advantage of this technology.

With regard to soil analysis, diffuse reflectance spectroscopy is of the highest interest because the energy of electromagnetic spectrum in diffuse reflectance enters soil and interacts there with physical and chemical soil components; moreover, the penetration depth is directly proportional to wavelength. Diffuse reflectance spectroscopy is used to record the spectra of heterogeneous systems, powders, or solid substances; note that the radiation diffusely reflected from a sample is collected at a wide angle and transmitted to a detector unlike the traditionally used transmission spectroscopy, using cuvettes, compaction of samples, and glasses.

Vis-NIR (400–2500 nm or 25 000–4000 cm–1) diffuse reflectance spectroscopy is successfully used for predicting a wide range of physical and chemical soil properties, including color [51], particle-size composition [22, 27, 28], moisture [10, 43, 57], content of soil organic [26, 59, 63] and inorganic [29, 31] carbon species, pools and fractions of soil organic carbon [33, 61, 69, 76], exchangeable bases [52, 73], pH [59, 75], degree of salinization [36, 82], mineralogical composition [40, 68], different iron compounds [67, 78], and contents of heavy metals and other pollutants [25, 47]. Vis-NIR spectroscopy supplements the molecular approaches to characterization of the diversity and biogeography of soil fungi and microbial communities [79, 80]. The soil properties that both directly influence the spectrum of electromagnetic radiation (organic and inorganic carbon compounds, mineralogical composition, and iron oxides) and do not cause active spectral absorption but correlate with spectrally active properties (pH, composition of exchangeable cations, and cation-exchange capacity) are assessable and predictable.

Soil organic carbon as a most important component in the formation of soil color has a considerable effect on the shape and type of reflectance spectra and displays evident specific spectral features [7, 8]. However, the interpretation of a recorded spectral signal with respect to soil organic carbon is not a trivial task because carbon compounds are complex and may be overlain by absorption bands the other components of soil matrix [62]. That is why, the quantitative analysis requires the calibration models (together with the preprocessing of spectral curves) able to extract complex absorption schemes generated by soil components from the recorded spectra.

METHODS

Meta-analysis is widely used and is an efficient tool for analysis of research papers; this procedure sums up the data collected from a number of papers and draws an overall statistical conclusion the basis of these data.

The review and meta-analysis in this work comprise the following stages: formulation of the research goal, search for the research papers using key words, extraction of the necessary data, discard of the non-relevant or duplicated papers, and formal analysis.

The databases of scientific periodicals Science Direct, Scopus, and Google Scholar were searched for the papers by the key word combination “Vis-NIR spectroscopy AND soil organic carbon”. The RSCI database was also searched by the key word “soil spectroscopy”. Part of the papers was discarded based on their titles and part by analyzing the abstracts with respect to inappropriate topics. As a result, the data for meta-analysis were extracted from the remaining 134 papers (published in 1986–2022).

The papers involved in the meta-analysis used the Vis-NIR spectroscopy approaches with subsequent comparison of the results of the standard laboratory procedures with the help of routine quantitative metrics of the predictive power of models. The following values were extracted as the metrics: coefficient of determination (\(R_{{{\text{cv/val}}}}^{2}\)), root mean square error (RMSE), and the ratio of performance to deviation (RPD). Higher values of the coefficient of determination and RPD represent the best predictions, while lower RMSE values indicate higher accuracy. We did not extract the coefficients of regression equation that demonstrate the power and character of the effect of independent variables on a dependent one and the degree of significance of individual variables because they were reported only in single cases.

Nonparametric one-sided Kruskal–Wallis variance analysis (Wilkinson–Mann–Whitney test for two groups) in conjunction with nonparametric pairwise comparison (Dunn’s post-hoc test) was used to determine the presence of a statistically significant difference between the medians of three or more independent groups of values. The results were visualized using free computing environment R and “ggstatsplot” software package [48].

RESULTS AND DISCUSSION

The number of papers reporting the use of Vis-NIR spectroscopy for the assessment of soil organic carbon and its content noticeably increased over last several years. In particular, 66 research papers of those included into meta-analysis were published in 2018–2022, which is 49% of the total number of papers. This may be associated with the demand in a cost-effective and convenient method for measuring soil properties as well as with ever-increasing attention and interest of the scientific community to this approach.

In most of the analyzed papers (84%), the soil spectroscopy approaches were successfully tested under laboratory conditions and the remaining 16% of the papers used in situ or on-the-go measurements.

The analyzed studies cover the levels from field conditions to large-scale continent-wide and global estimates for the content of soil organic carbon.

Preprocessing of spectral curves. Typically, the spectral reflectance curves are similar and not often allow the specific features of particular soils to be detected [8]. The selection of a reliable method for preprocessing the spectral curves may enhance the construction of a more accurate model predicting a target property. This is the first and important stage in the analysis of spectral data. Different methods for preprocessing the initial spectral data are used to decrease the noise and artifacts, improve the signal to noise ratio, enhance absorption characteristics, decrease the effect of light scattering, and reduce the dimensionality of the initial unprocessed or raw spectra. The following methods of preprocessing the spectral data are most frequently used [15, 72]: moving average filtering (MA), Savitsky–Golay smoothing (SG), calculation of first-order derivative (FD), multiple scattering correction (MSC), standard normal variate (SNV) normalization, wavelet transform (WT), continuum removal (CR), transformation of reflectance (R) values into absorption (A = log(1/R)), and removal of any linear trend from the spectrum (detrending, DT). In a number of papers, the researches preferred to use a combination of several preprocessing methods, in particular, Savitsky–Golay smoothing with calculation of first-order derivative (SGD), Savitsky–Golay filtering and transformation into absorption values (A*), and SNV normalization and transformation into absorption values (A**); other papers directly assert that any preprocessing of spectra does not improve the prediction [45]. Any golden standard for the use of preprocessing (or a sequence of preprocessing procedures) is currently absent; correspondingly, several methods are tested to select the best one [21, 24, 44]. Of the 134 analyzed papers, 14 (≈10%) assessed the effects of different preprocessing variants. Figure 1 shows the results of using several methods for preprocessing of spectral data.

Fig. 1.
figure 1

Examples of reflectance spectra in the Vis-NIR range with different data preprocessing methods: (a) initial unprocessed data (R); (b) absorption spectra (A); (c) spectra after Savitsky–Golay smoothing (SG); (d) first-order derivative spectra (FD); (e) standard normal variate spectra (SNV); and (f) spectra with removal of linear trend (DT). The initial spectra are extracted from open ICRAF-ISRIC Soil VNIR Spectral Library. Preprocessing was performed with the help of free computing environment R and “prospectr” software package [64].

Full size image

Figure 2 shows the comparison of the median values of the coefficient of determination and prediction error (RMSE) for different used spectral curve preprocessing methods. In particular, the following methods show the best performance (in terms of the coefficient of determination): wavelet transformation (WT) with R2 = 0.81, Savitsky–Golay filtering (SG) in combination with the transformation into absorption values (A*) with R2 = 0.78, Savitsky–Golay filtering (SG) with R2 = 0.77, and calculation of first-order derivative (FD) with R2 = 0.74 (Fig. 2a).

Fig. 2.
figure 2

Comparison of the median values of the coefficient of determination (R2) and prediction error (RMSE) for different methods used for preprocessing of spectral curves. Horizontal lines show the medians of the groups of values differing in a statistically significant manner. See text for abbreviations.

Full size image

Wavelet transformation (WT) is a useful tool for processing the spectra from the standpoint of filtering and a decrease in the dimensionality (data compression). WT is ideally appropriate for the extraction of spectral characteristics, first and foremost, because it performs multiresolution analysis of the signal [37]; further analysis at different scales (or resolutions) may well improve the identification of different specific spectral features [58]. SG filtering is widely applied to preprocess the soil spectra recorded by MIR spectroscopy. This is a universal method and it is appropriate for smoothing, filtering, and noise reduction. In terms of mathematics, it works as a weighted sum of adjacent values. This approach has several parameters for fine-tuning, namely, the size of smoothing window, differentiation parameter, and order of the polynomials that determine the degree of filtering [56]. Calculation of the first-order derivative (FD) removes the additive and multiple (for example, particle size) effects in spectra [72]. Other advantages are the removal of overlapping of absorption regions, boosting of weak absorption regions, and elimination of instrumental drift. As for its disadvantages, they comprise an increase in noise (correspondingly, smoothing is required), complication of the interpretation of spectra, and the risk of “overfitting” the calibration model [30]. Therefore, it is necessary to be careful when using this preprocessing approach to prevent the distortion of calibration model by generating the correlation with noise rather than with a real soil property.

The meta-analysis demonstrates that the authors use either SG filtering or its combination with the calculation of first-order derivative (SGD) in every tenth paper.

The calculation of first-order derivative (FD, RMSE = 0.26), absorption spectra (A, RMSE = 0.34), and initial spectra (R, RMSE = 0.40) gives the least error of prediction when assessing the content of soil organic carbon with Vis-NIR spectroscopy.

Note that the initial unprocessed spectra (R) give better model performance as compared with several preprocessing approaches, thereby confirming the assumption on the need in iterative heuristic analysis of spectral data. This assumption consists in that individual testing of a number of preprocessing methods and selection of a particular method that considerably improves the model performance by detection and identification of characteristic absorption bands for the soils differing in organic carbon content is necessary in the case of different soils and bioclimatic conditions. It is also likely that a certain preprocessing procedure removes the effect of some soil components, thereby increasing the sensitivity to the estimation of soil organic carbon content. The most appropriate methods will depend on the variation of the assessed property, quality of the initial data, errors of spectral radiometer, conditions of measurements, and method of multidimensional analysis.

The list of the preprocessing methods extends further than the mentioned variants. There are specialized methods for preprocessing the spectral data, for example, splice correction technique. This technique is used to remove the errors when using ASD FieldSpec Pro (Malvern Panalytical, Malvern, Worcestershire, United Kingdom) spectral radiometer involving different detectors. The switch between detectors usually takes place at wavelengths of 1000 and 1830 nm [64]. An example is the Gaussian pyramid representation (widely used for analysis of raster images), which is a hierarchical approach of data compression similar to wavelet transformation [13]. External parameter orthogonalization, which divides the spectrum into “useful” information containing soil properties and “external” part influenced by moisture, was proposed as an efficient method for smoothing the effect of soil moisture [21, 32, 53]. However, such methods are not often used (even just in very few cases) and are beyond the goal of this meta-analysis.

Multidimensional data analysis. The factors, such as overlapping absorption bands of soil component, instrumental noise, and scattering effect, result in intricate absorption patterns; correspondingly, the problem when collecting the data consists in a mathematical extraction of the information that correlates with soil properties from the recorded spectra. This problem belongs to the interdisciplinary area of chemometrics, residing at the interface of chemistry and mathematics, and applies mathematical and statistical methods for construction of the optimal measurement methods and extraction of the most important information when analyzing experimental data [9, 38].

The most widely used methods for calibrating models of the correlation between spectral characteristics (x1, x2 …, xn) and soil properties (y) are multiple linear regression (MLR), principal component regression (PCR), partial least square regression or projection to latent structures (PLSR), artificial neural networks (NN), random forest (RF), support vector machine (SVM), cubist regression model, and local modeling approaches (memory-based learning, MBL), which are similar in their principle to the k-nearest neighbor approach [72]. Of the 134 analyzed papers, 107 (≈80%) used the partial least square regression.

The prediction accuracy for soil organic carbon content with the help of Vis-NIR spectroscopy is illustrated in Fig. 3 (independently of the method of multidimensional analysis and preprocessing technique). The median value of the coefficient of determination (R2) over all analyzed papers amounts to 0.67; prediction error, to 0.48; and RPD, to 1.99. According to the Chaddock scale and [70], predictions are regarded as unsuccessful when RPD is lower than 1.5, and the coefficient of determination is below 0.5; the RPD values in the range of 1.5 to 2 and of R2 from 0.5 to 0.7 suggest satisfactory predictions (but allowing the high and low values to be separated); the RPD values of 2 to 2.5 and R2 of 0.7 to 0.9 indicate sufficiently accurate predictions with a good quantitative estimate; and the RPD values over 2.5 and R2 over 0.9 are excellent predictions.

Fig. 3.
figure 3

Prediction accuracy of the soil organic carbon content with the help of Vis-NIR spectroscopy. Colors from red to green denote unsuccessful, satisfactory, good, and excellent predictions, respectively. In addition, the histograms of the values of the coefficient of determination (R2) and RPD are shown at the axes.

Full size image

Figure 3 (in particular, distribution histograms) demonstrates that the majority of predictions fall into the area of satisfactory and good estimates. The main criticism related to soil spectroscopy consists in that this approach fails to give similarly accurate results as compared with the traditional laboratory methods [39]. However, the routine laboratory assays are the actual standards and any other approaches aimed at obtaining the same results are inevitably less accurate because they are based on the calibration of spectral data and are subject to statistical error. The advantage of Vis-NIR spectroscopy consists in that it is possible to perform the number of measurements exceeding the routine laboratory assays by one order of magnitude, the cost being equal; thus, the variation of the estimate will be lower as compared with the laboratory assays. A larger number of measurements will also decrease the effect of random errors.

Figure 4 shows the comparison of the median values of the coefficient of determination and prediction error (RMSE) for different multidimensional approaches of data analysis. The following methods (Fig. 4a) displayed the best performance (in terms of the coefficient of determination): neural networks (NN) with R2 = 0.77, multiple linear regression (MLR) with R2 = 0.71, principal correlation regression with R2 = 0.70, and partial least square regression (PLSR) with R2 = 0.69. The last two methods are widely used in chemometrics for the quantitative analysis of diffuse reflectance spectra. Both methods are used to construct calibration models in the case of a large number of individual variables (predictors) with strong collinearity. Both methods cope with the curse of dimensionality and compress the data before regression analysis into a series of orthogonal variables (principal components in PCR or hidden variables in PLSR). The difference between these approaches lies in the construction of new variables. In PCR, it is unimportant in what way the predictor variables correlate with the dependent variable. Unlike PCR, the PLSR regression algorithm unites the stages of compression and regression and selects the sequential orthogonal variables that maximize the covariance between the variable predictors (x1, x2 …, xn) and response (y).

Fig. 4.
figure 4

Comparison of the median values of the coefficient of determination (R2) and prediction error (RMSE) for different methods used for multidimensional data analysis. Horizontal lines show the medians of the groups of values differing in a statistically significant manner. See text for abbreviations.

Full size image

Neural networks (NN) are a powerful simulation method making it possible to reproduce extremely complex dependences, especially nonlinear ones. This is especially relevant for the problems in which a linear approximation (MLR) shows unsatisfactory performance. In addition, neural networks cope with the dimensionality reduction. However, this method does not always allow for understanding how a particular model makes its prediction (black box problem), thereby interfering with valid conclusions and discrediting such models. However, certain approaches to interpret complex models are currently used (for example, Shapley vector) aiming to take into account the effect of individual variables on the result [42, 81].

The following methods give the least prediction error in assessing the soil organic carbon content with the help of Vis-NIR spectroscopy (Fig. 4b): cubist regression model with RMSE = 0.29, support vector machine (SVM) with RMSE = 0.33, and partial least square regression (PLSR) with RMSE= 0.38. The first method is very popular and is currently easily interpretable and explainable. This method uses “if, then” type conditions to separate the spectral data into interrelated branches and approximates well the understandable linear regression models to the results of each separation [34].

A larger error may be associated with the fact that the soils considerably different in their properties are included into the calibration and test subsets in the studies at regional, national, and global scales. For example, humus hydrometamorphic and alluvial dark-humus soils containing 6–9% of organic carbon are more frequently included into the calibration subset and the textural calcareous chernozems with 3–4% of organic carbon, to the test subset. One of the possible solutions is construction of the models at a certain taxonomic level, that is, at the level of soil type. Another possible solution is the assessment of methods for selecting the subsets intended for training and validation of model (cluster analysis), estimation of data representativeness, and rejection of outliers.

Recently, considerable efforts were directed towards the development and estimation of different ranking procedures that objectively identify the spectral variables with a large contribution and concurrently remove the uninformative variables introducing noise to the model. This simplifies the model, prevents its overfitting, and increases interpretability. Ranking procedures include some classical approaches, such as expert knowledge and recursive feature elimination, as well as more complex approaches, such as genetic algorithm, regularized trees, and ant colony optimization [77].

We cannot but mention the fact that the initial variation in the content of soil organic matter at different scale levels in many respects determines the accuracy of prediction models. At the level of national and global spectral soil libraries comprising thousands of observations, the variation in soil organic carbon content is higher and consequently results in a higher bias (the mathematical expectation of the difference between true and predicted values) in the models. As for the regional scale, the same soil type may well have different soil organic carbon contents because of specific facies features, erosion processes, and degree of exhaustion by cultivation and, correspondingly, differ in their reflectance spectra. On the other hand, the soils at the level of a field with close soil organic carbon contents but residing on different soil-forming rocks will also differ in their spectral patterns. The pooling of the regional-scale data with the numerous local data in spectral libraries of a nation-wide scale and subsequent construction of general models give biased results. The general correlations obtained by using such models fail to shed light on the local changes in soil organic carbon content. Currently, the spiking approaches, for example, the supplementing of regional samples with a limited number of data (most similar in terms of spectral characteristics and geographic positions) from local samples become ever more popular.

Laboratory spectroscopy versus in situ spectroscopy. Figure 5 shows the comparison of the estimates for soil organic carbon content between the spectroscopy in laboratory (right plot) and directly in field.

Fig. 5.
figure 5

Comparison of the estimates for the soil organic carbon content produced by laboratory and in situ spectroscopies.

Full size image

The laboratory data demonstrates better performance (in terms of the coefficient of determination): the R2 values are higher by 20% as compared with in situ spectroscopy (Fig. 5a). However, the in situ variant gives higher uncertainty as is suggested by the larger intervals of prediction error values (Fig. 5b), which is associated with a considerably larger amount of data from the pool of research data using laboratory spectroscopy.

The better performance of laboratory spectroscopy is explainable by the fact that the assayed samples represent an “ideal” variant, being dried, ground, and preliminarily freed from plant residues and other material. This pretreatment is part of the standards and protocols on diffuse reflectance spectroscopy [14]. However, this can increase the reflectance coefficients and change the spectral curve patterns and the manifestation of local extrema [55]. The common advantages of laboratory spectroscopy are the measurements under controlled conditions, for example, at a constant source of light and the absence of the effect of atmosphere.

The in situ spectroscopy saves additional resources by removing the need in sample preparation. However, the prediction accuracy suffers of the effect of moisture and roughness of the surface. For example, an increase in soil moisture in the case of in situ field spectroscopy results in pronounced specific absorption features at 1400 and 1900 nm and decreases the total albedo [62]. An increase in soil moisture content displays also the trend of masking the spectral characteristics of soil components.

In the case when spectroscopy is used for further mapping of soil organic carbon content based on the remote sensing data for open soil surface, it is necessary to use the in situ spectrometric data (ideally, of a subsatellite level) to detect the target dependences. Such approaches to the prediction of soil organic carbon content has been successfully applied to the APEX [19] and AsiaFenix [65] satellite data as well as Sentinel-2 [16, 18] and Landsat [74] multispectral data. Note that models frequently lose their accuracy when turning to satellite sensing data (as compared with calibration models based on the in situ spectroscopy data), which results from several factors, including artificial noise, specific engineering features of sensors, landscape characteristics, atmospheric conditions, and state of the soil surface. Recent studies show that addition of the information about the state of surface to the analysis compensates for a negative influence of external factors and gives more stable and accurate predictions [50].

CONCLUSIONS

Vis-NIR diffuse scattering spectroscopy is a beneficial variant of rapid and less expensive alternative to the traditional methods for analysis of soil properties with an adequate accuracy. In future, the results of this approach that are more convincing are expectable with the advance of this technology, improvement of equipment, and development of methods for data analysis.

The meta-analysis of a total sample of 709 values of quantitative metrics has shown that the median value of the coefficient of determination (R2) over all papers amounts to 0.67; of prediction error (RMSE), to 0.48; and of RPD, to 1.99.

The analysis also suggests that the amount of data is currently insufficient for systematic confirmation of the accuracy of the field data on the measured soil organic carbon content using Vis-NIR spectroscopy. This area requires further studies.