Skip to main content

Advertisement

Log in

Assessing the Predictability of Solar Energetic Particles with the Use of Machine Learning Techniques

  • Editors’ Choice
  • Published:
Solar Physics Aims and scope Submit manuscript

Abstract

A consistent approach for the inherently imbalanced problem of solar energetic particle (SEP) events binary prediction is being presented. This is based on solar flare and coronal mass ejection (CME) data and combinations of both thereof. We exploit several machine learning (ML) and conventional statistics techniques to predict SEPs. The methods used are logistic regression (LR), support vector machines (SVM), neural networks (NN) in the fully connected multi-layer perceptron (MLP) implementation, random forests (RF), decision trees (DTs), extremely randomized trees (XT) and extreme gradient boosting (XGB). We provide an assessment of the methods employed and conclude that RF could be the prediction technique of choice for an optimal sample comprised by both flares and CMEs. The best-performing method gives a Probability of Detection (POD) of 0.76(±0.06), False Alarm Rate (FAR) of 0.34(±0.10), true skill statistic (TSS) 0.75(±0.05), and Heidke skill score (HSS) 0.69(±0.04). We further show that the most important features for the identification of SEPs, in our sample, are the CME speed, width and flare soft X-ray (SXR) fluence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6

Similar content being viewed by others

Notes

  1. https://github.com/SolarML/SEP-ML.

  2. https://numpy.org/.

  3. https://pandas.pydata.org/.

  4. https://scikit-learn.org/stable/.

  5. https://xgboost.readthedocs.io/en/latest/.

  6. https://www.cawcr.gov.au/projects/verification/.

  7. E.g. class weight \(\{c_{-}:1, c_{+}:500\}\).

  8. Class weight \(\{c_{-}:1, c_{+}:1\}\) in hyper-parametrizations of all 5 folds.

  9. We note that the aforementioned process does not lead to over-fitting, since trial and error iterations are performed strictly on the training set, while the selected models are finally assessed on unseen test sets.

References

  • Alberti, T., Laurenza, M., Cliver, E.W., Storini, M., Consolini, G., Lepreti, F.: 2017, Solar activity from 2006 to 2014 and short-term forecasts of solar proton events using the ESPERTA model. Astrophys. J. 838(1), 59. DOI. ADS.

    Article  ADS  Google Scholar 

  • Anastasiadis, A., Papaioannou, A., Sandberg, I., Georgoulis, M., Tziotziou, K., Kouloumvakos, A., Jiggens, P.: 2017, Predicting flares and solar energetic particle events: The FORSPEF tool. Solar Phys. 292(9), 134. DOI. ADS.

    Article  ADS  Google Scholar 

  • Anastasiadis, A., Lario, D., Papaioannou, A., Kouloumvakos, A., Vourlidas, A.: 2019, Solar energetic particles in the inner heliosphere: Status and open questions. Phil. Trans. Roy. Soc. London Ser. A 377(2148), 20180100. DOI. ADS.

    Article  ADS  Google Scholar 

  • Aran, A., Sanahuja, B., Lario, D.: 2005, Fluxes and fluences of SEP events derived from SOLPENCO. Ann. Geophys. 23(9), 3047. DOI. ADS.

    Article  ADS  Google Scholar 

  • Balch, C.C.: 2008, Updated verification of the space weather prediction center’s solar energetic particle prediction model. Space Weather 6(1), S01001. DOI.

    Article  ADS  Google Scholar 

  • Barth, J.L.: 2004, Prevention of spacecraft anomalies—The role of space climate and space weather models. In: Daglis, I.A. (ed.) Effects of Space Weather on Technology Infrastructure, Springer, Dordrecht, 123. DOI.

    Chapter  Google Scholar 

  • Bein, B.M., Berkebile-Stoiser, S., Veronig, A.M., Temmer, M., Vršnak, B.: 2012, Impulsive acceleration of coronal mass ejections. II. Relation to soft X-ray flares and filament eruptions. Astrophys. J. 755(1), 44. DOI. ADS.

    Article  ADS  Google Scholar 

  • Belov, A., Garcia, H., Kurt, V., Mavromichalaki, H., Gerontidou, M.: 2005, Proton enhancements and their relation to the X-ray flares during the three last solar cycles. Solar Phys. 229(1), 135. DOI. ADS.

    Article  ADS  Google Scholar 

  • Bishop, C.M.: 2006, Pattern Recognition and Machine Learning (Information Science and Statistics), Springer, Berlin. ISBN 0387310738.

    MATH  Google Scholar 

  • Brieman, L.: 2001, Random forests. Mach. Learn. 45, 5.

    Article  Google Scholar 

  • Brueckner, G.E., Howard, R.A., Koomen, M.J., Korendyke, C.M., Michels, D.J., Moses, J.D., Socker, D.G., Dere, K.P., Lamy, P.L., Llebaria, A., Bout, M.V., Schwenn, R., Simnett, G.M., Bedford, D.K., Eyles, C.J.: 1995, The Large Angle Spectroscopic Coronagraph (LASCO). Solar Phys. 162(1–2), 357. DOI. ADS.

    Article  ADS  Google Scholar 

  • Camporeale, E.: 2019, The challenge of machine learning in space weather: Nowcasting and forecasting. Space Weather 17(8), 1166. DOI. ADS.

    Article  ADS  Google Scholar 

  • Cane, H., Richardson, I., Von Rosenvinge, T.: 2010, A study of solar energetic particle events of 1997–2006: Their composition and associations. J. Geophys. Res. Space Phys. 115(A8), A08101. DOI.

    Article  ADS  Google Scholar 

  • Carrasco Kind, M., Brunner, R.J.: 2013, TPZ: Photometric redshift PDFs and ancillary information by using prediction trees and random forests. Mon. Not. Roy. Astron. Soc. 432, 1483. DOI.

    Article  ADS  Google Scholar 

  • Chen, T., Guestrin, C.: 2016, Xgboost: A scalable tree boosting system. In: KDD’16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785. ISBN 978-1-4503-4232-2. DOI.

    Chapter  Google Scholar 

  • Cliver, E.W.: 2016, Flare vs. shock acceleration of high-energy protons in solar energetic particle events. Astrophys. J. 832(2), 128. DOI. ADS.

    Article  ADS  Google Scholar 

  • Crosby, N.B.: 2007, Major radiation environments in the heliosphere and their implications for interplanetary travel. In: Bothmer, V., Daglis, I.A. (eds.) Space Weather- Physics and Effects, 131. DOI. ADS.

    Chapter  Google Scholar 

  • Defazio, A., Bach, F., Lacoste-Julien, S.: 2014, Saga: A fast incremental gradient method with support for non-strongly convex composite objectives.

  • Desai, M., Giacalone, J.: 2016, Large gradual solar energetic particle events. Living Rev. Solar Phys. 13(1), 3. DOI. ADS.

    Article  ADS  Google Scholar 

  • Domingos, P.: 2000, A unified bias-variance decomposition and its applications. In: Langley, P. (ed.) Proceedings of the Seventeenth International Conference on Machine Learning, Morgan Kaufmann Publishers Inc., San Francisco, 231.

    Google Scholar 

  • Fan, R.-E., Chang, K.-W., Hsieh, C.-J., Wang, X.-R., Lin, C.-J.: 2008, Liblinear: A library for large linear classification. J. Mach. Learn. Res. 9, 1871. DOI.

    Article  MATH  Google Scholar 

  • Filali Boubrahimi, S., Aydin, B., Martens, P., Angryk, R.: 2017, On the prediction of >100 MeV solar energetic particle events using GOES satellite data. arXiv. ADS.

  • Garcia, H.A.: 1994, Temperature and emission measure from goes soft X-ray measurements. Solar Phys. 154(2), 275. DOI. ADS.

    Article  ADS  Google Scholar 

  • Garcia, H.A.: 2004, Forecasting methods for occurrence and magnitude of proton storms with solar soft X rays. Space Weather 2(2), S02002. DOI. ADS.

    Article  ADS  Google Scholar 

  • Gopalswamy, N., Yashiro, S., Michalek, G., Stenborg, G., Vourlidas, A., Freeland, S., Howard, R.: 2009, The SOHO/LASCO CME catalog. Earth Moon Planets 104, 295. DOI. ADS.

    Article  ADS  Google Scholar 

  • Kahler, S.: 2001, The correlation between solar energetic particle peak intensities and speeds of coronal mass ejections: Effects of ambient particle intensities and energy spectra. J. Geophys. Res. 106(A10), 20947. DOI.

    Article  ADS  Google Scholar 

  • Kahler, S.W., Ling, A.G.: 2018, Forecasting Solar Energetic Particle (SEP) events with Flare X-ray peak ratios. J. Space Weather Space Clim. 8, A47. DOI. ADS.

    Article  ADS  Google Scholar 

  • Kingma, D., Ba, J.: 2014, Adam: A method for stochastic optimization. In: International Conference on Learning Representations.

    Google Scholar 

  • Lario, D., Aran, A., Gómez-Herrero, R., Dresing, N., Heber, B., Ho, G.C., Decker, R.B., Roelof, E.C.: 2013, Longitudinal and radial dependence of solar energetic particle peak intensities: STEREO, ACE, SOHO, GOES, and MESSENGER observations. Astrophys. J. 767(1), 41. DOI. ADS.

    Article  ADS  Google Scholar 

  • Laurenza, M., Alberti, T., Cliver, E.W.: 2018, A short-term ESPERTA-based forecast tool for moderate-to-extreme solar proton events. Astrophys. J. 857(2), 107. DOI. ADS.

    Article  ADS  Google Scholar 

  • Laurenza, M., Cliver, E., Hewitt, J., Storini, M., Ling, A., Balch, C., Kaiser, M.: 2009, A technique for short-term warning of solar energetic particle events based on flare location, flare size, and evidence of particle escape. Space Weather 7(4), S04008. DOI.

    Article  ADS  Google Scholar 

  • Mertens, C.J., Slaba, T.C.: 2019, Characterization of solar energetic particle radiation dose to astronaut crew on deep-space exploration missions. Space Weather 17(12), 1650. DOI. ADS.

    Article  ADS  Google Scholar 

  • Mishev, A.L., Usoskin, I.G.: 2018, Assessment of the radiation environment at commercial jet-flight altitudes during GLE 72 on 10 September 2017 using neutron monitor data. Space Weather 16(12), 1921. DOI. ADS.

    Article  ADS  Google Scholar 

  • Núñez, M., Paul-Pena, D.: 2020, Predicting >10 MeV SEP events from solar flare and radio burst data. Universe 6(10), 161. DOI. ADS.

    Article  ADS  Google Scholar 

  • Pacheco, D.: 2019, Analysis and Modelling of the Solar Energetic Particle Radiation Environment in the Inner Heliosphere in Preparation for Solar Orbiter, Universitat de Barcelona, Facultat de Física. http://hdl.handle.net/10803/667033.

    Google Scholar 

  • Panasyuk, M.I.: 2001, Cosmic ray and radiation belt hazards for space missions. In: Daglis, I.A. (ed.) Space Storms and Space Weather Hazards, 251. ADS.

    Chapter  Google Scholar 

  • Papaioannou, A., Sandberg, I., Anastasiadis, A., Kouloumvakos, A., Georgoulis, M.K., Tziotziou, K., Tsiropoula, G., Jiggens, P., Hilgers, A.: 2016, Solar flares, coronal mass ejections and solar energetic particle event characteristics. J. Space Weather Space Clim. 6, A42. DOI. ADS.

    Article  ADS  Google Scholar 

  • Papaioannou, A., Anastasiadis, A., Kouloumvakos, A., Paassilta, M., Vainio, R., Valtonen, E., Belov, A., Eroshenko, E., Abunina, M., Abunin, A.: 2018a, Nowcasting solar energetic particle events using principal component analysis. Solar Phys. 293(7), 100. DOI. ADS.

    Article  ADS  Google Scholar 

  • Papaioannou, A., Anastasiadis, A., Sandberg, I., Jiggens, P.: 2018b, Nowcasting of Solar Energetic Particle Events using near real-time Coronal Mass Ejection characteristics in the framework of the FORSPEF tool. J. Space Weather Space Clim. 8, A37. DOI. ADS.

    Article  ADS  Google Scholar 

  • Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: 2011, Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 12, 2825.

    MathSciNet  MATH  Google Scholar 

  • Pisacane, V.L.: 2008, The Space Environment and Its Effects on Space Systems, American Institute of aeronautics and Astronautics, Reston. ISBN 978-1-62410-353-7. DOI.

    Book  Google Scholar 

  • Pomoell, J., Aran, A., Jacobs, C., Rodríguez-Gasén, R., Poedts, S., Sanahuja, B.: 2015, Modelling large solar proton events with the shock-and-particle model. Extraction of the characteristics of the MHD shock front at the cobpoint. J. Space Weather Space Clim. 5, A12. DOI. ADS.

    Article  ADS  Google Scholar 

  • Reames, D.V.: 2015, What are the sources of solar energetic particles? Element abundances and source plasma temperatures. Space Sci. Rev. 194(1–4), 303. DOI.

    Article  ADS  Google Scholar 

  • Robbins, H., Monro, S.: 1951, A stochastic approximation method. Ann. Math. Stat. 22(3), 400. DOI.

    Article  MathSciNet  MATH  Google Scholar 

  • Salas-Matamoros, C., Klein, K.-L.: 2015, On the statistical relationship between CME speed and soft X-ray flux and fluence of the associated flare. Solar Phys. 290(5), 1337. DOI. ADS.

    Article  ADS  Google Scholar 

  • Salim, M., Ahmed, A.-R.: 2018, A family of quasi-Newton methods for unconstrained optimization problems. Optimization 67, 1717. DOI.

    Article  MathSciNet  MATH  Google Scholar 

  • Schmidt, M., Roux, N., Bach, F.: 2017, Minimizing finite sums with the stochastic average gradient. Math. Program. 162, 83. DOI.

    Article  MathSciNet  MATH  Google Scholar 

  • Schrijver, C.J., Kauristie, K., Aylward, A.D., Denardini, C.M., Gibson, S.E., Glover, A., Gopalswamy, N., Grande, M., Hapgood, M., Heynderickx, D., Jakowski, N., Kalegaev, V.V., Lapenta, G., Linker, J.A., Liu, S., Mandrini, C.H., Mann, I.R., Nagatsuma, T., Nandy, D., Obara, T., Paul O’Brien, T., Onsager, T., Opgenoorth, H.J., Terkildsen, M., Valladares, C.E., Vilmer, N.: 2015, Understanding space weather to shield society: A global road map for 2015-2025 commissioned by COSPAR and ILWS. Adv. Space Res. 55(12), 2745. DOI. ADS.

    Article  ADS  Google Scholar 

  • Shea, M.A., Smart, D.F.: 2012, Space weather and the ground-level solar proton events of the 23rd solar cycle. Space Sci. Rev. 171(1–4), 161. DOI. ADS.

    Article  ADS  Google Scholar 

  • Steyn, R., Strauss, D.T., Effenberger, F., Pacheco, D.: 2020, The soft x-ray Neupert effect as a proxy for solar energetic particle injection-a proof-of-concept physics-based forecasting model. J. Space Weather Space Clim. 10, 64.

    Article  ADS  Google Scholar 

  • Swalwell, B., Dalla, S., Walsh, R.W.: 2017, Solar energetic particle forecasting algorithms and associated false alarms. Solar Phys. 292(11), 173. DOI. ADS.

    Article  ADS  Google Scholar 

  • Trottet, G., Samwel, S., Klein, K.-L., Dudok de Wit, T., Miteva, R.: 2015, Statistical evidence for contributions of flares and coronal mass ejections to major solar energetic particle events. Solar Phys. 290(3), 819. DOI. ADS.

    Article  ADS  Google Scholar 

  • Unzicker, A., Donnelly, R.F.: 1974, Calibration of x-ray ion chambers for the space environment monitoring system. Technical report com-75-10667, National Oceanic and Atmospheric Administration, Boulder, Colo. (USA), Space Environment Lab.

  • Vainio, R., Desorgher, L., Heynderickx, D., Storini, M., Flückiger, E., Horne, R.B., Kovaltsov, G.A., Kudela, K., Laurenza, M., McKenna-Lawlor, S., Rothkaehl, H., Usoskin, I.G.: 2009, Dynamics of the Earth’s particle radiation environment. Space Sci. Rev. 147(3–4), 187. DOI. ADS.

    Article  ADS  Google Scholar 

  • Van Rossum, G., Drake, F.L. Jr.: 1995, Python Tutorial, Centrum voor Wiskunde en Informatica Amsterdam, The Netherlands.

    Google Scholar 

  • Vapnik, V.: 2000, The Nature of Statistical Learning Theory 8, 1. ISBN 978-1-4419-3160-3. DOI.

    Book  MATH  Google Scholar 

  • Vlahos, L., Anastasiadis, A., Papaioannou, A., Kouloumvakos, A., Isliker, H.: 2019, Sources of solar energetic particles. Phil. Trans. Roy. Soc. London Ser. A 377, 20180095. DOI.

    Article  ADS  Google Scholar 

  • Vršnak, B., Sudar, D., Ruždjak, D.: 2005, The cme-flare relationship: Are there really two types of cmes? Astron. Astrophys. 435(3), 1149. DOI.

    Article  ADS  Google Scholar 

  • Vršnak, B., Ruždjak, D., Sudar, D., Gopalswamy, N.: 2004, Kinematics of coronal mass ejections between 2 and 30 solar radii. Astron. Astrophys. 423(2), 717. DOI.

    Article  ADS  Google Scholar 

  • Winter, L.M., Ledbetter, K.: 2015, Type II and type III radio bursts and their correlation with solar energetic proton events. Astrophys. J. 809(1), 105. DOI. ADS.

    Article  ADS  Google Scholar 

  • Yashiro, S., Gopalswamy, N.: 2009, Statistical relationship between solar flares and coronal mass ejections. In: Gopalswamy, N., Webb, D.F. (eds.) Universal Heliophysical Processes 257, 233. DOI. ADS.

    Chapter  Google Scholar 

  • Youssef, M.: 2012, On the relation between the CMEs and the solar flares. NRIAG J. Astron. Geophys. 1, 172. DOI. ADS.

    Article  ADS  Google Scholar 

Download references

Acknowledgements

The CME Catalog used in this work is generated and maintained at the CDAW Data Center by NASA and The Catholic University of America in cooperation with the Naval Research Laboratory. Funding for the early phase of the catalog was provided by AFOSR and NSF. Currently, the catalog resides at the CDAW Data Center at Goddard Space Flight Center and is supported by NASA’s Living with a Star program and the SOHO project. SOHO is a project of international cooperation between ESA and NASA. Angels Aran and Athanasios Papaioannou acknowledge the support from the project MDM-2014-0369 of ICCUB (Unidad de Excelencia “María de Maeztu”). Athanasios Papaioannou and Anastasios Anastasiadis further gratefully acknowledge International Space Science Institute (ISSI) support to the ISSI International Team 441: High EneRgy sOlar partICle events analysis (HEROIC, https://www.issibern.ch/teams/heroic/) In addition, Athanasios Papaioannou acknowledges ISSI support to the ISSI International Team 464: The Role Of Solar And Stellar Energetic Particles On (Exo)Planetary Habitability (ETERNAL, https://www.issibern.ch/teams/exoeternal/). Angels Aran acknowledges the support by the Spanish Ministerio de Ciencia e Innovación (MICINN) under grant PID2019-105510GB-C31 and through the “Center of Excellence María de Maeztu 2020-20” award to the ICCUB (CEX2019-000918-M). The authors would further like to thank the anonymous referee for the valuable comments that improved the context of the paper.

Author information

Authors and Affiliations

Authors

Ethics declarations

Disclosure of Potential Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A: Classification Methods and Utilized Implementations

1.1 A.1 Logistic Regression

Logistic regression is a linear classification method, in the sense that the input feature space \(\mathcal{R}^{l}\) is divided into decision regions, bounded by linear decision surfaces (hyper-planes) of l-1 dimensions. In case of binary classification, a single hyper-plane partitions input variable space in two decision regions, corresponding to the positive and negative class. This hyper-plane can be mathematically expressed as:

$$ \boldsymbol{f}(\boldsymbol{\Theta} ; \mathbf{x}) = \boldsymbol{\theta }^{\top}\mathbf{x} + \theta_{0} = 0, \ \ \boldsymbol{\theta},\mathbf{x} \in\mathcal{R}^{l}, \ \theta_{0} \in\mathcal{R}, $$
(20)

where the weight vector \(\boldsymbol{\Theta}\) is defined in a compact form composed of the direction (\(\boldsymbol{\theta}\)) and offset (\(\theta_{0}\)) as follows:

$$ \boldsymbol{\Theta} = [\boldsymbol{\theta}, \theta_{0}], \ \ \boldsymbol {\theta} \in\mathcal{R}^{l}, \ \ \theta_{0} \in\mathcal{R}. $$
(21)

In case classes are linearly separable in the given feature space (Figure 7), the above formulation implies that:

$$ \boldsymbol{f}(\boldsymbol{\Theta}; \mathbf{x}) \textstyle\begin{cases} < 0, \ \text{if} \ \mathbf{x} \in c_{-}, \\ > 0, \ \text{if} \ \mathbf{x} \in c_{+}. \end{cases} $$
(22)
Figure 7
figure 7

Logistic regression for binary classification on linearly separable classes, under L2 (left) or L1 (right) penalization. Black and yellow dots represent synthetic data in two dimensions, as random samples drawn from two non-overlapping normal distributions (std=1). The solutions obtained by imposing strong and weak regularization, are displayed as green and purple lines respectively. The regularization strength is controlled by the free parameter \(C\), where small and large values correspond to strong and weak regularization respectively.

The cost function to be minimized in algorithms implementing Logistic Regression classifiers, is generally based on the logistic sigmoid function:

$$ f(a) = \frac{1}{1 + e^{-a}}. $$
(23)

The logistic sigmoid function’s characteristic property is that it squashes all possible outcomes into the interval (0, 1), for all input values of \(a \in\mathcal{R}\). The primary term of the cost function is formed as follows:

$$ \mathcal{J}(\boldsymbol{\Theta}, \mathbf{x}) = C \sum_{i=1}^{N} \log (1+\exp{(-y_{i}({\mathbf{x}_{i}}^{\top}\boldsymbol{\theta} + \theta_{0}))}), $$
(24)

where C is the inverse of regularization strength, often set to the inverse of the total number of training samples N. An additional penalization term is usually included in the general form Equation 24.

Hyper-parameters of the logistic regression classifier in the sci-kit learn implementation include:

  1. i)

    inverse regularization strength C: smaller values imply stronger regularization,

  2. ii)

    penalty: specifies L1 / L2 penalization term,

  3. iii)

    solver: the minimization algorithm implemented to solve the optimization problem, such as ‘sag’ Stochastic Average Gradient (Schmidt, Roux, and Bach, 2017), ‘saga’ (Defazio, Bach, and Lacoste-Julien, 2014) another incremental gradient algorithm based on Stochastic Gradient Descent (Robbins and Monro, 1951) and ‘liblinear’ which implements a Newton method (Fan et al., 2008),

  4. iv)

    random state: used to shuffle the data, when the solver is ‘sag’, ‘saga’ or ‘liblinear’,

  5. v)

    maximum number of iterations,

  6. vi)

    class weight: to control the strength of penalization applied to the wrong prediction of each class.

1.2 A.2 Support Vector Machines

This is a set of supervised learning methods that use a subset of training vectors, called support vectors, in fitting the discriminant function (Vapnik, 2000). Support vectors are instance vectors that lie within a pre-defined distance from either side of the decision boundary, named the functional margin. In case a linear form is adopted to describe the discriminant function, the goal is to define a hyper-plane as Equation 20 that assigns most instances to the correct class, as indicated by sign(\(\boldsymbol{\theta}^{\top}\mathbf{x} + \theta_{0}\)) (Equation 22). The distance from the nearest instance vectors of all the classes to the decision boundary is maximized at the same time, aiming to increase model’s generalization ability. The distance d from a point \(\mathbf{x}_{*}\) to a hyper-plane \(\boldsymbol {f}(\boldsymbol {\Theta}, \mathbf{x})\) defined by Equation 20 is:

$$ d = \frac{\boldsymbol{f}(\boldsymbol{\Theta}, \mathbf{x}_{*})}{\| \boldsymbol {\theta}\|}. $$
(25)

Maximum distance is achieved by minimizing \(\|\boldsymbol{\theta}\| ^{2}=\boldsymbol{\theta}^{\top}\boldsymbol{\theta}\). In addition, weights are scaled so that the values of the decision function \(\mathcal{C}\) at the nearest points to the decision surface are:

$$ \boldsymbol{f}(\boldsymbol{\Theta}; \mathbf{x}_{near}) = \boldsymbol {\theta }^{\top}\mathbf{x}_{near} + \theta_{0} = \textstyle\begin{cases} -1, & \text{if}\ \mathbf{x}_{near} \in c_{-},\\ +1, & \text{if}\ \mathbf{x}_{near} \in c_{+}. \end{cases} $$
(26)

Taking into account Equation 22, it follows that in the case of perfect linear separability, the optimal solution would satisfy:

$$ y_{i} (\boldsymbol{\theta}^{\top}\boldsymbol{\theta}_{i} + \theta_{0}) \geq 1, \ \ i=1,\ldots,N. $$
(27)

Given that usually classes are not perfectly separable with a hyper-plane, each point is allowed a maximum distance \(\xi_{i}\) to its correct margin. The resulting optimization problem is formulated as:

$$\begin{aligned} \begin{aligned}\min_{\boldsymbol{\theta}, \theta_{0}, \xi} \frac{1}{2} \boldsymbol{\theta}^{\top}\boldsymbol{\theta} + C \sum_{i=1}^{N} \xi_{i}\\ \begin{aligned}\textrm{subject to } & y_{i} (\boldsymbol{\theta}^{\top}\mathbf{x}_{i} + \theta_{0}) \geq1 - \xi_{i},\\ & \xi_{i} \geq0, \ i=1, \ldots, N. \end{aligned} \end{aligned} \end{aligned}$$
(28)

The variables \(\xi_{i}\), called slack variables, define the maximum distance allowed for samples to their correct margin boundary. The coefficient C controls the strength of penalization posed when an instance vector \(\mathbf{x}_{i}\) is misclassified or within the functional margin. Small values correspond to wider margins and vice versa (Figure 8).

Figure 8
figure 8

Linear SVM for binary classification, where classes are not linearly separable in the given feature space. Black and yellow dots represent synthetic data in two dimensions, as random samples drawn from two overlapping normal distributions (std=1). Solid line indicates the decision boundary, whereas dashed lines indicate the functional margin. Support vectors are in circles. Weak regularization, corresponding to small values of \(C\), results in a relatively wide margin wherein some samples are allowed to be misclassified (left panel). Strong regularization (large \(C\)) results in a more strict solution hence a narrow margin, wherein very few or no samples are found.

A common technique utilized under the SVM approach, in the case the problem is highly non-linear in the original input space, is the kernel method. This method maps the problem to a higher dimensional space where linear separability of classes becomes more feasible. Instead of learning an explicit discriminant function that maps support vectors \(\mathbf{x}_{i}\) to their target labels \(y_{i}\), a similarity function or kernel is used to learn the particular weight of each training sample (\(\mathbf{x}_{i}, y_{i}\)). Unlabeled vectors \(\mathbf {x}_{j}\) can subsequently be classified according to their similarity to training vectors \(\mathbf{x}_{i}\), expressed as a proper inner product through the kernel function:

$$ K(x_{i}, x_{j}) = \phi(x_{i})^{\top}\phi(x_{j}) = \langle\phi(x_{i}), \phi (x_{j})\rangle, $$
(29)

where training vectors are mapped to a higher (maybe infinite) feature space \(\mathcal{V}\) through feature mapping function \(\phi\):

$$ \phi: \mathbf{X} \Rightarrow\mathcal{V}. $$
(30)

The optimization problem can be formulated as a dual problem to the primal Equation 28, using the kernel trick:

$$\begin{aligned} \begin{aligned}\min_{\alpha} \frac{1}{2} \alpha^{T} Q \alpha- e^{T} \alpha \\ \begin{aligned} \textrm{subject to } & y^{T} \alpha= 0\\ & 0 \leq\alpha_{i} \leq C, i=1, \ldots, N, \end{aligned} \end{aligned} \end{aligned}$$
(31)

where:

  1. i)

    e is a \(1 \times N\) vector of ones,

  2. ii)

    \(Q=[Q_{ij}]=[y_{i} y_{j} K(x_{i},x_{j})]_{i,j=1,\ldots,N}\) is a \(N \times N\) positive semi-definite matrix,

  3. iii)

    K is the kernel described by Equation 29,

  4. iv)

    coefficients \(\alpha_{i}\), called the dual coefficients, are upper-bounded by C.

Common kernel functions utilized in SVM classification are:

  1. i)

    linear: \(\mathbf{K}=\langle x_{i}, x_{j}\rangle\), where <.> denotes the dot product between two l-dimensional vectors,

  2. ii)

    polynomial: \(\mathbf{K}=(\gamma\langle x_{i}, x_{j} \rangle+ r)^{d}\), where d indicates the polynomial degree,

  3. iii)

    Radial Basis Function (RBF): \(\mathbf{K}=\exp(-\gamma\| x_{i}-x_{j}\|^{2})\),

  4. iv)

    sigmoid: \(\mathbf{K}=\tanh(\gamma\langle x_{i}, x_{j} \rangle + r)\).

The solution to the optimization problem Equation 31 corresponds to the equation describing the decision surface, so that the label of a new sample \(\mathbf{x}_{j}\) can be predicted according to the sign of decision function’s output for this sample, using only the support vectors:

$$ \sum y_{i} \alpha_{i} K(x_{i}, x) + b. $$
(32)

Hyper-parameters used to describe an SVM classifier with a linear kernel in the sci-kit learn implementation include:

  1. i)

    penalty: specifies L1 / L2 penalization term,

  2. ii)

    random state: to be used for shuffling the data in the dual problem optimization (Equation 31),

  3. iii)

    C: the penalization strength controlling the margin,

  4. iv)

    class weight: to control the strength of penalization applied to the wrong prediction of each class,

  5. v)

    maximum number of iterations.

Hyper-parameters used to describe an SVM classifier with a non-linear kernel in the sci-kit learn implementation include:

  1. i)

    the kernel function,

  2. ii)

    \(\gamma\): the regularization coefficient of the kernel,

  3. iii)

    \(coef_{0}\): the independent term in the kernel function,

  4. iv)

    C: the penalization strength controlling the margin,

  5. v)

    the polynomial degree, in the case of a polynomial kernel,

  6. vi)

    class weight: to control the strength of penalization applied to the wrong prediction of each class,

  7. vii)

    maximum number of iterations.

Examples of SVM binary classification using different kernel functions are presented in Figure 9.

Figure 9
figure 9

SVM implementation for binary classification, with the use of different kernel functions. Black and yellow dots represent synthetic data in two dimensions, as random samples drawn from two overlapping normal distributions (std=1). The solid line in each plot indicates the decision boundary, projected onto the original feature space. Dashed lines indicate the functional margin. Support vectors are denoted in circles.

1.3 A.3 Neural Network (Multi-Layer Perceptron)

Neural networks (NN) sequentially transform the input space into higher dimensional spaces, where classes can be linearly discriminated (Figure 10). This classifier is actually constructed as an ensemble of logistic regression classifiers (Bishop, 2006). The dimension of the original feature space corresponds to the number of nodes (neurons) of the input layer. For binary classification problems, the output layer has a unique node, providing the final classification result (0/1). The number of transformations needed to achieve the best possible separability of classes, defines the number of hidden layers. The number of nodes at each hidden layer indicates the dimensionality in each transformed space. Nodes are interconnected via the synapses, indicating transformations of variables, so that transformation coefficients to a given layer from the previous one are referred to as the synaptic weights.

Figure 10
figure 10

Geometric view of the Multi-Layer Perceptron, for binary classification on artificially generated, two-dimensional data (top left). Samples from the two classes, displayed as red and green stars, can be perfectly separated by three lines (l1, l2, l3). The input variable space (x1, x2) is transformed by the 1st hidden layer into a 3D space. The new coordinates are defined with respect to each sample’s position above (1) or below (0) each line, i.e. using the step function for simplification instead of the logistic (sigmoid) function. All samples are now embedded in the vertices of a 3D cube with unity edge (top right). Surfaces S1, S2, S3 can further isolate green samples. The corresponding transformation with respect to S1, S2, S3 is performed by the 2nd hidden layer, so that all red stars end up in the (0, 0, 0) vertex (bottom left). Surface h can finally separate the two classes as positive (red, above the surface h) or negative (green, below h), corresponding to the final binary decision by the single node of the output layer.

Once we define a specific architecture for the NN classifier, it can be parametrized on the synaptic weights \(\boldsymbol{\Theta}\) that describe sequential transformations between layers. The cost as well can be expressed as a function of \(\boldsymbol{\Theta}\) and the classification error between predicted/actual results, subsequently to be minimized by a certain optimization algorithm, as the stochastic average gradient (Schmidt, Roux, and Bach, 2017).

Hyper-parameters of the NN classifier, in the sci-kit learn implementation of MLP, include:

  1. i)

    number of hidden layers and their dimensionalities: described as tuples (\(d_{1}\), …, \(d_{k}\)), \(d_{i}\) = dimensions of i-th hidden layer, k = number of hidden layers,

  2. ii)

    solver: the method for weight optimization,

  3. iii)

    learning rate: the method/schedule for synaptic weight updates (e.g., constant, gradual, adaptive),

  4. iv)

    the initial learning rate: controls the step for updating the weights,

  5. v)

    maximum number of iterations.

Commonly used optimization methods in the MLP implementation include:

  1. i)

    sgd: stochastic gradient descent (Robbins and Monro, 1951),

  2. ii)

    lbfgs: an optimizer in the family of quasi-Newton methods (Salim and Ahmed, 2018),

  3. iii)

    adam: a stochastic gradient-based optimizer proposed by Kingma and Ba (2014).

Neural Networks are in general black-box models, as high dimensional spaces are usually involved so that the model’s structure becomes too complex to interpret or visualize. On the other hand, NNs can tackle problems involving highly non-linear underlying relations.

1.4 A.4 Decision Tree Classifier

Decision tree is a non-parametric supervised classification method, in the sense that the model infers decision rules to optimally classify instance vectors, by learning from the vectors themselves. Input feature space \(\mathcal{R}^{l}\) is sequentially partitioned so that instance vectors of the same class are grouped together, within the created subspaces (Figure 11). Partition is performed under a set of sequential if-then-else decision rules. Each decision rule results in a split of the training data on a particular feature and corresponds to a decision node in the tree’s structure. Leaf nodes represent the tree’s final decisions/predictions, so that they correspond to either homogeneous subsets (including instances of the same class) if classes are linearly separable, or to the purest possible subsets otherwise. The first split is performed on the most informative feature in the given feature space and the corresponding node is the tree’s root node.

Figure 11
figure 11

Decision Tree structure, for binary classification on artificial 2D data. Samples from the two classes are represented as green and red stars, in the (\(x_{1}, x_{2}\)) input space (left panel). The decision tree classifier sequentially splits the input space into more pure subspaces. The 1st split is performed at \(x_{1}=4\), because all samples above this value belong to the red class. This maximizes purity, resulting in a (red) leaf node, where samples with \(x_{1}>4\) end up (right panel). In the opposite case, i.e. for samples with \(x_{1}<4\), an additional split is performed at \(x_{2}=3\), since samples above and below that value belong to the green and red class respectively.

Subsequent splits are performed in a sequential manner, while the rule for a certain decision node is based on the subset of vectors that satisfy the criteria defined by its predecessor nodes. More specifically, for each feature and each possible feature value, the corresponding split is evaluated according to a specified criterion. The combination (feature, value) that optimizes the adopted criterion is selected to perform the split. The commonly used criteria to quantify the quality of each split are Gini Impurity and Entropy (information gain).

Gini Impurity measures the probability for a randomly chosen data vector to be wrongly classified. This actually equals the probability for any vector to be randomly chosen, times the probability for it to be misclassified, summed over all possible classes (two in our case).

$$ G = \sum_{i=0}^{1} p_{i} \sum_{j\neq i} p_{j} = \sum_{i=0}^{1} p_{i}(1-p_{i}) = \sum_{i=0}^{1} p_{i} - \sum_{i=0}^{1} p_{i}^{2} = 1-\sum_{i=0}^{1} p_{i}^{2}. $$
(33)

Entropy is similarly defined as

$$ E = - \sum_{i=0}^{1} p_{i} \log p_{i}, $$
(34)

where the information gain associated to an arbitrary event with probability of occurrence \(p_{i}\) is

$$ I \equiv\log(1/p_{i}) = - \log p_{i}. $$
(35)

Minimization of either Gini impurity or entropy results in more uniform (pure) nodes (Figure 12).

Figure 12
figure 12

Gini impurity and entropy for binary classification, as a function of the percentage of data vectors belonging to one of two classes. Both impurity indices are equal to zero when only one class is present, while they achieve maximum values (Gini=0.5, Entropy=1) when classes are equally split (50:50), for details see Carrasco Kind and Brunner (2013).

Hyper-parameters used to modify the decision tree classifier in the sci-kit learn implementation include:

  1. i)

    split criterion: Gini impurity, information gain,

  2. ii)

    minimum number of samples in a leaf node,

  3. iii)

    minimum number of samples required to perform a split,

  4. iv)

    maximum depth of the tree, indicating the maximum number of decisions/splits to be performed,

  5. v)

    class weight: to control the strength of penalization applied to the wrong prediction of positive and negative class events.

Decision Trees produce white-box models, as their interpretation and visualization is straightforward. Nonetheless, the creation of over-complex trees that cannot generalize well to new data is always possible. Over-fitting is avoided by finding the tree with the minimum possible depth that can solve a particular classification problem. This purpose can be met by fine-tuning the maximum allowed depth and the minimum number of samples required for the tree to make decisions, to relatively low and high values, respectively. Over-fitting can also be tackled a posteriori, that is, after fitting the tree to the training data. Post-pruning is applied to achieve the best possible balance of tree complexity and misclassification cost.

1.5 A.5 Random Forest Classifier

Random Forest is formed as an ensemble of multiple decision trees, trained on different parts of the dataset. The bootstrap aggregating technique is used, to extract random samples with replacement and fit trees to these samples, using a random subset of features (\(\sim\sqrt {n_{features}}\)). Prediction of unseen samples is performed by extracting the “majority vote” of the predictions made by individual trees. This technique reduces variance without increasing the bias, since individual trees are independently trained.

Hyper-parameters of the Random Forest classifier in the sci-kit learn implementation include:

  1. i)

    split criterion, to be used in individual trees (estimators),

  2. ii)

    minimum number of samples in estimator leaf node,

  3. iii)

    minimum number of samples required to perform a split in estimator,

  4. iv)

    maximum depth of estimators,

  5. v)

    class weight ⇒ to control the strength of penalization applied to the wrong prediction of positive and negative class events,

  6. vi)

    total number of estimators to be constructed,

  7. vii)

    maximum number of features to be utilized in constructing individual estimators.

1.6 A.6 Extremely Randomized Trees (Extra Trees)

Extremely randomized trees are also ensembles of multiple decision trees. In contrast to random forest, each tree in Extra trees is fit on the whole training dataset. Decision rules are established in each tree using a randomly selected cut point, instead of searching for the one that optimizes a split criterion, resulting in reduced correlation amongst trees.

Hyper-parameters of the Extremely randomized trees classifier used in the present study are similar to those of the random forest classifier, described in the previous subsection.

1.7 A.7 Extreme Gradient Boosting (XGBoost)

Another classifier built as an ensemble of multiple decision trees is Extreme gradient boosting Chen and Guestrin (2016). XGBoost (XGB) trees are not independently but sequentially constructed, learning from previous trees’ mistakes through a clever penalization of trees. Newton boosting technique is employed, resulting in faster convergence to cost minima than gradient descent. Additionally, an extra randomization parameter is used to reduce correlation amongst trees, so that boosting results in reduced variance and bias.

Hyper-parameters of the XGBoost classifier include:

  1. i)

    booster optimization algorithm to be used, mostly methods in the Newton family,

  2. ii)

    number of estimators to be constructed,

  3. iii)

    maximum depth of each estimator,

  4. iv)

    subsample of the training data to be used in individual estimators,

  5. v)

    gamma parameter: minimum loss reduction required to make a further partition on a leaf node of estimator, larger values correspond to more conservative algorithm,

  6. vi)

    lambda parameter: L2 regularization term on weights, larger values employ more conservative algorithms,

  7. vii)

    alpha: L1 regularization term on weights, larger values employ more conservative algorithms,

  8. viii)

    scale applied to positive weights: to control the strength of penalization applied to the wrong prediction of positive class events, relative to the one applied to the negative class.

Appendix B: Tabulated Information on Experimental Results

Table 5 Logistic regression models performance.
Table 6 Linear SVM Models Performance.
Table 7 Non-linear SVM models performance.
Table 8 Neural network models performance.
Table 9 Decision tree models performance.
Table 10 Random forest models performance.
Table 11 Extra trees models performance.
Table 12 XGBoost models performance.
Table 13 Random forest models performance and optimal hyper-parameters, in the imbalanced setting of the Flare_CME_noiseless configuration.
Table 14 Logistic regression models performance and optimal hyper-parameters, in the imbalanced setting of the Flare_CME_Noiseless data configuration.

Appendix C: Hyper-Parameter Spaces

3.1 C.1 Logistic Regression (LR)

  1. i)

    [penalty:] ‘l2’, ‘l1’

  2. ii)

    [solver:] ‘saga’, ‘liblinear’

  3. iii)

    [random_state:] 42

  4. iv)

    [C:] logspace(start=-4, end=3, step=1)

  5. v)

    [max_iter:] 5000, 10000, 20000

  6. vi)

    [class_weight:] ‘balanced’, {0: 1, 1: 1}, {0: 1, 1: 10}, {0: 1, 1: 20}, {0: 1, 1: 50}, {0: 1, 1: 100}, {0: 1, 1: 150}, {0: 1, 1: 300}, {0: 1, 1: 400}, {0: 1, 1: 500}

3.2 C.2 Support Vector Machines with Linear Kernel (linSVM)

  1. i)

    [penalty:] ‘l2’, ‘l1’

  2. ii)

    [random_state:] 42

  3. iii)

    [C:] logspace(start=-4, end=3, step=1)

  4. iv)

    [max_iter:] 5000, 10000, 20000

  5. v)

    [class_weight:] ‘balanced’, {0: 1, 1: 1}, {0: 1, 1: 10}, {0: 1, 1: 20}, {0: 1, 1: 50}, {0: 1, 1: 100}, {0: 1, 1: 150}, {0: 1, 1: 300}, {0: 1, 1: 400}, {0: 1, 1: 500}

3.3 C.3 Support Vector Machines with Non-linear Kernel (SVM)

  1. i)

    [kernel:] ‘rbf’, ‘poly’, ‘sigmoid’

  2. ii)

    [C:] logspace(start=-4, end=3, step=1)

  3. iii)

    [gamma:] ‘scale’, ‘auto’, 0.001, 0.01, 0.1, 1, 10

  4. iv)

    [degree:] 2, 3, 4

  5. v)

    [coef0:] -10, -1, -0.1, -0.01, -0.001, 0.0, 0.001, 0.01, 0.1, 1, 10

  6. vi)

    [max_iter:] 5000, 10000, 20000

  7. vii)

    [class_weight:] ‘balanced’, {0: 1, 1: 1}, {0: 1, 1: 10}, {0: 1, 1: 20}, {0: 1, 1: 50}, {0: 1, 1: 100}, {0: 1, 1: 150}, {0: 1, 1: 300}, {0: 1, 1: 400}, {0: 1, 1: 500}

3.4 C.4 Neural Network (NN)

  1. i)

    [hidden_layer_sizes:] (50,), (40,), (30,), (20,), (10,), (8,), (6,), (50, 2), (40, 2), (30, 2), (20, 2), (10, 2), (8, 2), (6, 2), (50, 3), (40, 3), (30, 3), (20, 3), (10, 3), (8, 3), (6, 3), (50, 4), (40, 4), (30, 4), (20, 4), (10, 4), (8, 4), (6, 4), (50, 20), (50, 10), (50, 5), (60,), (60, 20), (60, 10), (60, 5), (60, 4), (60, 3), (60, 2), (4,), (3,), (2,), (4, 4), (4, 2), (2, 1), (2, 2), (3, 1), (3, 2), (60, 30), (50, 25), (40, 20), (30, 15), (20, 10), (10, 5)

  2. ii)

    [learning_rate:] ‘constant’, ‘invscaling’, ‘adaptive’

  3. iii)

    [learning_rate_init:] 0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05

  4. iv)

    [max_iter:] 5000, 10000, 20000

  5. v)

    [solver:] ‘adam’, ‘lbfgs’

3.5 C.5 Decision Tree (DT)

  1. i)

    [criterion:] ‘gini’, ‘entropy’

  2. ii)

    [min_samples_leaf:] range(start=2, end=100, step=2)

  3. iii)

    [min_samples_split:] range(start=5, end=130, step=5)

  4. iv)

    [max_depth:] 2, 4, 6, 8, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, None

  5. v)

    [class_weight:] ‘balanced’, {0: 1, 1: 1}, {0: 1, 1: 10}, {0: 1, 1: 20}, {0: 1, 1: 50}, {0: 1, 1: 100}, {0: 1, 1: 150}, {0: 1, 1: 300}, {0: 1, 1: 400}, {0: 1, 1: 500}

3.6 C.6 Random Forest (RF)

  1. i)

    [criterion:] ‘gini’, ‘entropy’

  2. ii)

    [n_estimators:] 10, 20, 40, 60, 80, 100, 200, 400, 600, 800, 1000

  3. iii)

    [min_samples_leaf:] range(start=2, end=100, step=2)

  4. iv)

    [min_samples_split:] range(start=5, end=130, step=5)

  5. v)

    [max_depth:] 2, 4, 6, 8, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, None

  6. vi)

    [max_features:] ‘auto’, ‘sqrt’, ‘log2’

  7. vii)

    [class_weight:] ‘balanced’, {0: 1, 1: 1}, {0: 1, 1: 10}, {0: 1, 1: 20}, {0: 1, 1: 50}, {0: 1, 1: 100}, {0: 1, 1: 150}, {0: 1, 1: 300}, {0: 1, 1: 400}, {0: 1, 1: 500}

3.7 C.7 Extremely Randomized Trees (XT)

  1. i)

    [random_state:] 42

  2. ii)

    [criterion:] ‘gini’, ‘entropy’

  3. iii)

    [n_estimators:] 10, 20, 40, 60, 80, 100, 200, 400, 600, 800, 1000

  4. iv)

    [min_samples_leaf:] range(start=2, end=100, step=2)

  5. v)

    [min_samples_split:] range(start=5, end=130, step=5)

  6. vi)

    [max_depth:] 2, 4, 6, 8, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, None

  7. vii)

    [max_features:] ‘auto’, ‘sqrt’, ‘log2’, None

  8. viii)

    [class_weight:] ‘balanced’, {0: 1, 1: 1}, {0: 1, 1: 10}, {0: 1, 1: 20}, {0: 1, 1: 50}, {0: 1, 1: 100}, {0: 1, 1: 150}, {0: 1, 1: 300}, {0: 1, 1: 400}, {0: 1, 1: 500}

3.8 C.8 Extreme Gradient Boosting (XGB)

  1. i)

    [max_depth:] range(start=2, end=60, step=2)

  2. ii)

    [n_estimators:] 10, 20, 40, 60, 80, 100, 200, 400, 600, 800, 1000

  3. iii)

    [scale_pos_weight:] range(start=1, end=400, step=50)

  4. iv)

    [alpha:] 0.00001, 0.0001, 0.001, 0.01, 0.1, 1, 10

  5. v)

    [gamma:] 0.00001, 0.0001, 0.001, 0.01, 0.1, 1, 10

  6. vi)

    [lambda:] range(start=1, end=22, step=1)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lavasa, E., Giannopoulos, G., Papaioannou, A. et al. Assessing the Predictability of Solar Energetic Particles with the Use of Machine Learning Techniques. Sol Phys 296, 107 (2021). https://doi.org/10.1007/s11207-021-01837-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11207-021-01837-x

Keywords

Navigation