Evaluating eXplainable artificial intelligence tools for hard disk drive predictive maintenance

Abstract

In the last years, one of the main challenges in Industry 4.0 concerns maintenance operations optimization, which has been widely dealt with several predictive maintenance frameworks aiming to jointly reduce maintenance costs and downtime intervals. Nevertheless, the most recent and effective frameworks mainly rely on deep learning models, but their internal representations (black box) are too complex for human understanding making difficult explain their predictions. This issue can be challenged by using eXplainable artificial intelligence (XAI) methodologies, the aim of which is to explain the decisions of data-driven AI models, characterizing the strengths and weaknesses of the decision-making process by making results more understandable by humans. In this paper, we focus on explanation of the predictions made by a recurrent neural networks based model, which requires a tree-dimensional dataset because it exploits spatial and temporal features for estimating remaining useful life (RUL) of hard disk drives (HDDs). In particular, we have analyzed in depth as explanations about RUL prediction provided by different XAI tools, compared using different metrics and showing the generated dashboards, can be really useful for supporting predictive maintenance task by means of both global and local explanations. For this aim, we have realized an explanation framework able to investigate local interpretable model-agnostic explanations (LIME) and SHapley Additive exPlanations (SHAP) tools w.r.t. to the Backblaze Dataset and a long short-term memory (LSTM) prediction model. The achieved results show how SHAP outperforms LIME in almost all the considered metrics, resulting a suitable and effective solution for HDD predictive maintenance applications.

Appendices

Appendix

In this Appendix, we provide further details about our evaluation related to the explanation of HDD health status prediction using SHAP in terms of global and local explanations.

Appendix 1: global explanation

For each class we draw the Dependence Plots of the first four most important features of that class, in which we highlight the interaction effect with the feature we are plotting and another one through the colours (Fig. 24).

Fig. 24

Dependence Plot: $Good_{correct}$ rank 0–3

Fig. 25

Dependence Plot: $VeryFair_{correct}$ rank 0–3

Fig. 26

Dependence Plot: $Warning_{correct}$ rank 0–3

Fig. 27

Dependence Plot: $Alert_{correct}$ rank 0–3

If we observe the plot at the bottom left, we can notice that the two samples that have the highest SHAP values are those included between 0.4 and 0.8 as regards the value of Raw Read Error Rate (RRER) and between 0.2 and 0.4 for SER. The contribution of the two samples is 0.020 and 0.025. In particular, the highlighted samples (in yellow) have the same values of both RRER and SER but both have negative SHAP values as shown in Fig. 28.

Fig. 28

Dependence Plot RRER-SER $VeryFair_{correct}$

Appendix 2: local explanation

1.1 Alert

1.1.1 Force plot single element

The Force plot for the individual item shows the features that help push the model output from the base value (the average model output over the training dataset) to the model output. In particular, features that contribute most to the prediction are shown in red while the others in blue.

Figure 29 shows the Force plot for the 8223 sample, in which it is easy to note that TC, RRER and POH are the three main features that contribute positively to the output. In turn, Fig. 30 indicates that TC and POH features influenced the misclassification of the 8371 sample as Alert despite the fact that SER and RRER are rowing in the right direction.

Fig. 29

Force Plot: $Alert_{correct}$, element 8223

Fig. 30

Force Plot: $Alert_{misclassified}$, element 8371

1.1.2 Bar plot single element

The last plot we are going to analyze is the Bar Plot (see Fig. 31a, b), that does not bring more information content w.r.t. the above described plots but it is more easy to analyze. The features are ordered by importance, that are depicted in red or blue if they provide a positive contribution to the model output or not, respectively. In particular, the Shapley value of each features is reported on the x-axis, showing how they impact on the model output.

Fig. 31

Bar plot for Alert class by using SHAP

1.2 Warning

1.2.1 Force plot single element

Fig. 32

Force Plot: $Warning_{correct}$, element 479

The Fig. 32 shows how TC, RRER, POH and SER are the most important feature for predicting the warning class whilst the ones, that provides negative contribution, have such a low impact that they are not visible. In turn, RRER, SUT and POH provide negative contribution in order to misclassify 1135 sample as Warning class (see Fig. 33).

Fig. 33

Force Plot: $Warning_{misclassified}$, element 1135

1.2.2 Bar plot single element

Figure 34a and b investigates how our model predicts the health level status about two samples, being respectively correctly classified (479) and misclassified (1135), through the bar plots.

Fig. 34

Bar plot for Warning class by using SHAP

1.3 Very fair

1.3.1 Force plot single element

Despite the fact that almost all features (except RRER) contribute positively to prediction, the main important one is undoubtedly SUT, as shown in Fig. 35. Nevertheless, as is even more evident in Fig. 36, TC definitely pushes the model in the wrong direction by classifying the 904 sample as Very Fair.

Fig. 35

Force Plot: $VeryFair_{correct}$, element 721

Fig. 36

Force Plot: $VeryFair_{misclassified}$, element 904

1.3.2 Bar plot single element

We can observe through the bar plots (Fig. 37a, b) the contribution of each feature in terms of Shapley value.

Fig. 37

Bar plot for Very Fair class by using SHAP

1.4 Good

1.4.1 Force plot single element

Figure 38 provides a clear picture of which are the main features (reported in red) contributing to the prediction made by the model, in which the main important features are TC, POH, SER and SUT. In Fig. 39, we can clearly understand why the model made a wrong prediction classifying the 7425 sample as belonging to the Good class. The main feature is POH which has a very high value (looking at the decision plot in Fig. 15b) and corresponds to an extremely large SHAP value as we can see from the force plot.

Fig. 38

Force Plot: $Good_{correct}$, element 28

Fig. 39

Force Plot: $Good_{misclassified}$, element 7425

1.4.2 Bar plot single element

We show the Bar Plots of the two samples, 28 correctly classified and 7425 misclassified, in Fig. 40a and b.

Fig. 40

Bar plot for good class by using SHAP