Performance discrepancy mitigation in heart disease prediction for multisensory inter-datasets

Previous works on Cleveland dataset

El-Hasnony et al. (2022) applied Active Learning-based ML to predict heart disease. This study employed MMC, Random, Adaptive, QUIRE, and AUDI as selection strategies, which also reduce labeling costs for multi-label active learning. They evaluated their proposed experiment regarding performance metric accuracy and the f1-score with or without hyperparameter tuning and cross-validation. Regarding accuracy, the adaptive method advanced others, scoring $0.514\pm 0.032$ before and $0.574\pm 0.020$ after hyperparameter optimization. Also, for the f1-score, the adaptive method showed its superiority for the heart disease prediction score of $0.623\pm 0.040$ before and $0.6062\pm 0.036$ after hyperparameter tuning. Selvi & Muthulakshmi (2021) in 2021 developed an optimal artificial neural network (OANN) by combining distance based misclassified instance removal (DBMRI), teaching and learning based optimization (TLBO), and artificial neural network (ANN). DBMRI is used to extract incorrectly classified instances from the data, and TLBO is an optimization algorithm. Moreover, the model is evaluated utilizing the Cleveland dataset in the offline phase. Real-time data is cleaned, preprocessed, and streamed to the model using Apache Spark in the online phase to detect the potentiality of heart problems. During the test phase, the OANN model surpassed other models by scoring 95.31% accuracy. Motarwar et al. (2020) also proposed an ML-based cognitive technique for predicting heart disease based on the Cleveland heart disease dataset. The stated study made use of five classifiers: logistic model tree (LMT), SVM, Hoeffdpin decision tree (HDT), RF, as well as GNB. They applied feature selection approaches to their work. GNB had a 93% accuracy prediction, SVM had a 90% prediction, RF had a 95% prediction, HDT had an 81% prediction, and LMT had an 81% prediction. Mohan, Thirumalai & Srivastava (2019) used the Hybrid Random Forest Linear Model (HRFLM) on the Cleveland dataset to obtain 88.7% accuracy. They also tried out KNN, DT, genetic algorithm (GA), and naive Bayes (NB), among other machine-learning models. HRFLM performed the best out of all of them. Hasan et al. (2018) proposed a performance analysis of classification approaches to predict heart disease. They employed the Cleveland heart disease dataset. Five different classifiers, namely KNN, DT (ID3), GNB, LR, and RF are used. Using 14 attributes, KNN provided 70%, DT (ID3) provided 88%, GNB provided 88%, LR provided 89%, and RF provided 89% accuracy. After performing RF feature selection, the ten most important features are selected. KNN provided 71%, DT (ID3) provided 91%, GNB provided 91%, LR provided 93%, and RF provided 92% accuracy. In 2017, Uyar & İlhan (2017) presented a method for predicting heart disease utilizing a GA-based Recurrent fuzzy neural network (RFNN). Their proposed model, GA-based RFNN, utilizes the Cleveland heart disease dataset. Their proposed model predicted 98% accuracy. Dun, Wang & Majumder (2016) in 2016 used RF, LR, SVM, and neural networks (NN) to evaluate the prevalence of heart disease. They also performed hyperparameter tuning and feature selection. RF provided 77%, LR provided 75%, linear SVM provided 75%, SVM provided 69%, and NN provided 78% accuracy. Their study also utilized the benchmark Cleveland heart disease dataset.

Previous works on UCI heart disease dataset

For heart disease prediction, Bharti et al. (2021) coupled ML and deep learning algorithms. They achieved an accuracy of 83% using LR, 85% using KNN, 83% using SVM, 80% using RF, 82% using DT, and 94% using deep learning by employing the UCI heart disease dataset and also Lasso as a feature selection technique. In 2021, Katarya & Meena (2021) compared nine different ML and deep learning classifiers for heart disease prediction. Two different datasets from the UCI repository were utilized for their study. Also, data were preprocessed by doing normalization, missing value replacement, and feature selection. Moreover, state-of-the-art RF performed the best, achieving 95.60% accuracy, followed by LR, SVM, and artificial neural networks (ANN). Mehmood et al. (2021) developed Cardio Help. This framework uses a convolutional neural network (CNN) to determine the chance of a subject developing CVD by utilizing the heart disease dataset from the UCI repository. Furthermore, the LASSO technique was applied to pick and normalize relevant features. On top of that, two distinct classifiers were created, one for binary and the other for four classes. The binary classifier achieved 97% accuracy, whereas the four-class classifier achieved 86% accuracy. In 2020, Farzana & Veeraiah (2020) used several different ML and data mining classification techniques, such as GNB, SVM, RF, KNN, and XGBoost. They employed the UCI heart disease dataset, SVM provided 82%, RF provided 89%, GNB provided 82%, KNN provided 67%, and XGBoost provided 79% accuracy. In 2019, Alotaibi (2019) presented the utilization of an ML model to identify heart disease based on the UCI heart disease dataset and applied DT, LR, RF, NB, and SVM as classifiers. The DT provided 93%, LR provided 87%, RF provided 89%, NB provided 87%, and SVM provided 92% prediction accuracy. Repaka, Ravikanti & Franklin (2019) developed sophisticated heart disease estimations using the UCI dataset. They implemented four different techniques; among them, NB gave the best accuracy of 89.77%.

Previous works on other single datasets

Dritsas & Trigka (2023), proposed a ML-based CVD risk prediction model. They balanced the imbalanced CVD dataset using the Synthetic Minority Oversampling Technique (SMOTE). In the context of feature selection, they employed three feature selection methods: RF, information gain, and gain ratio. Also, they employed several classifiers for CVD prediction and evaluated their experimental results with or without the application of SMOTE with 10-fold cross-validation and compared them using different performance metrics. The study outcomes revealed that the Stacking ensemble model followed SMOTE with 10-fold cross-validation, advancing of others, reaching an accuracy of 87.8%, a recall of 88.3%, a precision of 88%, and an AUC of 98.2%. Manimurugan et al. (2022) presented a two-stage heart disease prediction model. Its first step categorized data collected from medical sensors attached to the subject’s body; the second stage classified echocardiography images to predict heart disease. Furthermore, a hybrid linear discriminant analysis with enhanced ant lion optimization (HLDA-MALO) approach was utilized for sensor data identification. Moreover, a hybrid Faster R-CNN with SE-ResNet-101 model was applied for echocardiography image classification. The HLDA-MALO approach detected healthy sensor data with 96.85% accuracy and abnormal sensor data with 98.31% accuracy. Their proposed hybrid Faster R-CNN with SE-ResNeXt-101 transfer learning model performed better in identifying echocardiography pictures, with 98.06% precision, 98.95% recall, 96.32% specificity, a 99.02% F-score, and 99.15% maximum accuracy. Chowdhury et al. (2019) proposed heart disease prediction using a smart digital stethoscope system. They used phonocardiogram (PCG) signal data containing normal and abnormal heart sounds for this analysis. They preprocessed the signal data, segmented it, and extracted different features from the segmented data, which will be fed to ML classifiers. The DT, discriminant analysis, SVM, KNN, and ensemble classifiers with five-fold cross-validation are employed for heart disease prediction. The cost-adjusted optimal ensemble approach can diagnose abnormal and normal HS with 97% and 88% accuracy, respectively, among employed classifiers. Raza (2019) attempted to forecast heart disease in 2019 using the ensemble learning method utilizing the Statlog heart disease dataset available in the UCI ML repository. They used a majority voting rule to combine LR, NB, and ANN. This approach had an accuracy of 88.88%. In 2019, Beunza et al. (2019) proposed comparing ML classifiers for diagnostic coronary heart disease prediction. This study used a dataset from the Framingham heart study. Also, they employed six classifiers: DT, Boosted DT, SVM, NN, RF, and LR. DT provided 84%, Boosted DT provided 84%, RF provided 74%, SVM provided 68%, NN provided 71% and LR provided 66% accuracy according to R-Studio Model. Chen et al. (2007) used SVM, NN, BM, DT, and LR to predict heart disease. They used a dataset collected from AnZhen Hospital in Beijing, China. The state-of-the-art SVM provided 91%, NN provided 89%, BM provided 82%, DT provided 78% and LR provided 74% accuracy.

Previous works on multiple datasets

Valarmathi & Sheela (2021) proposed hyperparameter optimization to predict heart disease in 2021. They used Grid Search (GS), Randomized Search (RS), and TPOT classifiers for optimizing the performance of RF and XGBoost classifiers. For this analysis, they used the Cleveland and Z-Alizadeh Sani datasets. Upon the Cleveland heart disease dataset, RF with GS provided 91% accuracy, RS provided 95% accuracy, and TPOT provided 98.5% accuracy. On the other hand, the XGBoost algorithm with GS provided 86%, RS provided 92% and TPOT provided 91% accuracy on the Cleveland heart disease dataset. However, on the Z-Alizadeh Sani dataset, RF with RS provides the highest 80%, 74%, and 77% accuracy for the three vessels. Rathi et al. (2021) in 2021 used the C4.5 algorithm to build a prediction model on the UCI heart problems dataset from Kaggle to predict the likelihood of heart disease. Their proposed model achieved 89% accuracy. Mienye, Sun & Wang (2020) used an ensemble technique for cardiac disease prediction. Both the Framingham and Cleveland datasets were used. Using CART, they created a homogenous ensemble AB-WAE. In addition, they employed several classifiers. Their employed LR presented 78%, LDA presented 78%, KNN presented 60%, SVM presented 79%, GB presented 81%, RF presented 83%, CART presented 68%, and the presented AB-WAE presented 93% accuracy using the Cleveland heart disease dataset. Ayon, Islam & Hossain (2020) investigated numerous ML approaches for heart disease prediction, utilizing the StatLog and Cleveland datasets separately. Also, two distinct K-values were used to split the data (5,10). On the StatLog dataset, SVM had a five-fold classification accuracy of 97% and a ten-fold classification accuracy of 95%. On top of that, SVM scored 97% for five-fold classification and 95% for ten-fold classification on the Cleveland dataset.

Khan (2020) built an IoT system to alert physicians to the possibility of a patient developing heart disease. The system is powered by an ML approach trained using the UCI, Framingham, and Public Health datasets. They chose the relevant features using the MCFA method. After that, the key features are used to train MDCNN and other models after they have been selected. The models, once trained, use sensor data to predict the likelihood that the subject will develop heart disease. As a result, MDCNN outperforms all other models in terms of overall performance. Gárate-Escamila, El Hassani & Andrès (2020) in 2020 applied different ML algorithms along PCA and Chi-square on the UCI heart disease dataset. They used Cleveland, Hungarian, and Cleveland-Hungarian combined datasets. As a consequence, they got a maximum accuracy of 99% for RF, LR, and Gradient boosting classifier (GBC), 98% for DT, 87% for MLP, and 69% for NB. In 2020, Spencer et al. (2020) proposed a feature selection and classification method for heart disease prediction. They created a mixed dataset for their analysis, combining data from Cleveland, Long Beach, VA, Hungary, and Switzerland. This combined dataset contains 14 features with 720 samples. Moreover, they applied Bayes Net (BN), LR, Stochastic Gradient Descent (SGD), Instance-Based Learner (IBK), AdaBoost, JRip, and RF. The BN algorithm provided a maximum of 85% accuracy on the original combined dataset. After that, they also applied PCA, Chi-square, ReliefF, and Symmetrical Uncertainty (SU) on the combined dataset. IBK algorithm provided a maximum 84% accuracy on the Heart-PCA dataset with 11 features. The BN algorithm provided a maximum of 85% accuracy on the Heart-Chi-Square dataset with ten features. On the other hand, the SGD algorithm provided a maximum of 85% accuracy on the Heart-RelifF dataset, and BN provided a maximum of 85% accuracy on the Heart-SU dataset. Reddy et al. (2019) employed KNN, SVM, RF, NB, and NN classifiers on the Cleveland and StatLog heart disease datasets for prediction. They chose six and eight features using methods for selecting features. Using six features, KNN 86%, SVM 83%, RF 91%, NB 87%, and NN 86% accuracy was displayed. Models with six features have a marginally higher average accuracy than those with eight. Masetic & Subasi (2016) in 2016 carried out the research by using Electrocardiogram reports to make a heart disease prediction model. The Autoregressive Burg method was used to extract features. A total of three datasets were used, collected from three different sources, two datasets contained data from people with congestive heart failure, and one dataset had data from people with normal conditions. Several classifiers were used to get the best-performing model. Among those used classifiers, RF had the best accuracy of 100% in both dataset combinations. A group of researchers employed two different ML techniques, DT and NB. The DT presented 91%, and the NB presented 87% accuracy. In 2021, Ali et al. (2021) developed a smart heart disease prediction approach utilizing feature fusion and ensemble deep learning. Both the sensor information and data from electronic medical records are used. They extracted significant features from those datasets and merged them. Moreover, they used LogitBoost as a base learner for the proposed ensemble mode. Their fusion dataset with specific feature weighting method provides accuracy 84% for SVM, 92% for LR, 85% for MLP, 87% for RF, 78% for DT, 83% for NB, and 99% for their proposed ensemble. A comparison of related literature is tabulated in Tables 1 and 2.

Dataset description

In this study, the well-known five heart disease datasets, namely Cleveland (Janosi et al., 1988), Hungarian (Janosi et al., 1988), Switzerland (Janosi et al., 1988), Long Beach VA (Janosi et al., 1988), StatLog (Statlog (Heart), 2024), and a combined dataset formed using these five datasets have been experimented with. The selection of these five datasets for the experiment is based on the similarity of features. Those five datasets are publicly available, and most of the research works in this domain are based upon the datasets, as the literature review sections justify. total observations include patients and controls, with percentages, as provided in Table 3. Notice that in our combined dataset, there are 918 observations after removing the 272 duplicates. We are interested in how often each patient and control group appear. In such a scenario, it is evident that the datasets are imbalanced, which is a crucial factor in diminishing the accuracy of classifiers. The study also analyzes how balancing the datasets might improve performance. Moreover, all the datasets, including the combined one, contain 11 common features (Age, Sex, ChestPainType, RestingBP, Cholesterol, FastingBS, RestingECG, MaxHR, ExerciseAngina, Oldpeak, and ST_Slope) and a target feature (heart disease). On top of that, the features are described in Table 4 with their data types.

Dataset recreation

It is a crucial stage to reform the datasets based on a threshold for working with different datasets (Stefenon et al., 2022). In this study, the other four datasets are recreated based on the Cleveland dataset’s sequence of attributes. After recreating the datasets, every dataset’s column is identical and contains the same attributes and data types. Figure 1 shows the overview of the recreation of the datasets. We find each dataset’s descriptive statistics (mean median, standard deviation, kurtosis, and skewness).

Combined dataset formation

The recreated datasets are merged one after another to form the combined one. However, the combined dataset contains some duplicate values, and after removing the duplicates, the final combined dataset is created.

Combination to check the inter-dataset performance

To check the inter-dataset discrepancy in heart disease prediction, we created 30 combinations using the five individual datasets with 1:1, 2:1, 3:1, 4:1, 2:2, and 3:2 ratios. In addition, Table 5 tabulates all the combinations of the datasets for analyzing the inter-dataset performance.

Method overview

The methodology is divided into four major segments: (i) data preprocessing, (ii) dimensionality reduction, (iii) model building using ML classifiers, and (iv) model evaluation. First, each of the five individual datasets is investigated to determine their statistical characteristics. Based on the characteristics, different preprocessing techniques are applied to convert the datasets into more ML-centric to fit the ML classifiers accurately. We recreate the datasets into the same form based on the columns of the Cleveland dataset. Following the basic preprocessing, we apply PCA and RF separately for single and combined datasets and various dataset combinations. The performance of each classifier for different phases is tabulated and visualized by different plots to find the inter-dataset discrepancy and how the proposed methodology solves the problem. The proposed methodology’s block diagram is depicted in Fig. 2.

Data preprocessing

For the analysis of the datasets, key preprocessing is mandatory. The total preprocessing techniques are divided into two different phases. The first phase contains missing value handling, log transformation, outlier detection, normalization, and imbalanced data handling. In the second phase, we reduce the features using two popular techniques, PCA as feature extraction and RF as feature selection.

Missing value handling: There is often a missing value in real-world datasets. It is the cause of the skewed outcome and has a major effect on how well ML algorithms perform (Thomas et al., 2020). Addressing the problem of missing data values in datasets is crucial for increasing prediction precision. Dropping missing data or filling it in with averages, medians, and point estimates are two methods for dealing with missing data (Zhang, 2016). The datasets used in this study contain many missing values that must be filled out. Because of the limited sample sizes of the datasets, it is infeasible to delete rows to process the values. In addition, most values have a property that allows us to replace the missing values gap by averaging the columns, which yields better performance than alternative handling methods. However, there are many missing values in some columns in Switzerland, Hungarian, and Long Beach VA datasets, and the traditional approach is inappropriate to handle these cases. So, we consider the missing value column as the dependent variable and other correlated columns as independent variables to solve this issue. As a result, we employ the RF regression model to impute missing values with suitable replacements, leveraging its robustness against overfitting and less sensitive nature to outliers. The pseudocode in Algorithm 1 illustrates the missing value-handling process. After that, the distribution of the data exhibits some properties of a normal distribution, and to transform it into a more normal distribution, we employ a log transformation.

Outlier detection: When building and deploying ML algorithms, outlier detection is crucial. However, outliers in a dataset reduce the performance of the algorithms (Ramaswamy, Rastogi & Shim, 2000), and it is detected using Turkey fences (Zhou et al., 2006). Since Turkey fences are suitable for datasets with a normal distribution, we employ them in our preprocessing pipeline to handle outliers. The dataset is split into three quartiles: Q1, Q2, and Q3. The first quartile, or Q1, is the value within the data set comprising 25% values below it. The third quartile, or Q3, is indeed the value that accounts for 25% of the values above it. Outliers are also valued below or above the lower or upper limits, as calculated below.

$$lower\mathrm{\_}limit=Q_1-1.5(Q_3-Q_1)$$

$$upper\mathrm{\_}limit=Q_1+1.5(Q_3-Q_1)$$

Outliers that fall below the lower limit are replaced with a lower limit, and outliers above the upper limit are replaced with an upper limit.

Normalization: Normalization converts the numerical column values of a dataset to a standard scale (García, Luengo & Herrera, 2015). It is important when an ML model employs Euclidean distance to interpret the inputs (Taunk et al., 2019). Since we transform our data to be normally distributed and remove outliers, both Min-max normalization and Standardization can work well. So, this work transforms the datasets into normalized ones using the Min-Max scaling method. Therefore, we employ the Min-max scaling method. It subtracts the smallest value from the maximum value of the column and divides it by the range. After normalization, each column’s value falls between 0 and 1.

Imbalance data handling: One drawback of imbalanced data is that insufficient data from the underrepresented group prevents the algorithm from learning an appropriate cutoff point (Norori et al., 2021). Under-sampling and over-sampling are the two most common approaches to solving this issue. Replicating random samples from the minority group is an example of oversampling while selecting random instances from the majority group and removing them is an example of undersampling. This investigation made use of SMOTETomek ensemble methods to cope with data imbalance (Hasan et al., 2022). SMOTETomek combines oversampling and undersampling strategies to boost the efficiency of the classifier model. The SMOTE method is used to oversample the minority group and then identify and eliminate samples from the majority group, as described in Tomek Links, to achieve a more equitable distribution of demographic characteristics. These methods outperform competing balancing methods when used to predict CVDs (Sahid et al., 2022).

Feature reduction

Feature extraction using PCA: PCA is the most widely utilized unsupervised algorithm for feature extraction (Uddin et al., 2021). In our study, we use PCA to mitigate the dimensionality of the dataset. PCA converts a series of potentially correlated observations together into a collection of Principle Component (PC) values that are either linearly statistically independent observations (Singh et al., 2017). PCA produces eigenvectors in decreasing order, such as PC1, PC2, PC3, etc. Using the PCs, the analysis of the dataset can be much better than the original dataset. Moreover, it significantly decreases the algorithm’s complexity while improving model performance in classification and regression.

Feature selection using RF: The feature selection process consists of choosing the most crucial features of a dataset and removing less important features that may help to improve the dataset’s quality. Sometimes, the performance of the model increases by eliminating less significant features from the dataset (Gayathri et al., 2022). RF is one of the many techniques for feature selection. Due to its ensemble nature, RF is less prone to overfitting than other algorithms. Therefore, we employ it for feature selection. An RF is constructed by combining several DTs (Oshiro, Perez & Baranauskas, 2012). Internal nodes and leaves are unique to each tree. The internal node performs what to do to segment the data set into two sets with strong responses according to the chosen feature. Internal node features are determined using some criterion. For regression tasks, variance reduction is used, while for classification, information gain and Gini impurity are used. When using information gain to train a DT, optimizing information gain yields the best split. Gini impurity indicates the impurity of a node, and its range is from 0 to 1. Moreover, lowering the value of the Gini impurity means a purer node, and we can calculate the average reduction in impurity for each feature. The feature gets more valuable as the nodes become purer. As well as, the feature significance is measured as the average of all trees in the forest. When the significance of all features is calculated, the much more significant features are chosen using a sequential backward elimination technique. Mathematically, $\mathrm{G}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{i}\mathrm{m}\mathrm{p}\mathrm{u}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{o}\mathrm{f}\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}\mathrm{a}\mathrm{f}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{a}\mathrm{c}\mathrm{h}\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}=1-{\sum }_{i=1}^c{p}_i^2$ with $Entropy={\sum }_{i=1}^c{p}_{i}lo{g}_2{p}_i$ and $\mathrm{I}\mathrm{n}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{G}\mathrm{a}\mathrm{i}\mathrm{n}=1-Entropy$, where $c$ = number of classes of a dataset and $p_i$ = the probability of selecting random an element from class $i$.

ML classifier

This study employs different supervised ML classifiers to analyze the heart disease prediction based on the properties of our preprocessed five different datasets in the inter-dataset discrepancy setup. A short description of each classifier is provided below.

SVM: SVM has become the most widespread supervised ML technique for classification and regression (Rabbi et al., 2022). In the SVM, perhaps every data item is depicted as just a point throughout $n$-dimensional space ( $n$ represents the number of features). Hyperplanes, or decision boundaries, are used to help classify data items. Support vectors are the number of observations or trajectories nearest to the hyperplane which influences its position. The distance between observations or trajectories and the hyperplane is the margin. SVM’s goal is to maximize this margin as much as feasible. The hyperplane with the biggest margin is the ideal hyperplane. To transform a low-dimensional input vector to a higher-dimensional vector, SVMs employ the kernel technique. The hyperplane is symbolized as $w.x+b=0$, where $w$ is the hyperplane’s normal vector and $b$ is indeed an offset. A decision rule must be defined to classify a point as negative or positive. The following is a definition of a decision rule:

$$\overrightarrow{X}.\overrightarrow{w}-c\ge 0$$

$$putting-casb,weget$$

$$\overrightarrow{X}.\overrightarrow{w}+b\ge 0hence$$

$$f(x)=\{\begin{array}{cc}+1,& \text{if}\overrightarrow{X}.\overrightarrow{w}+b\ge \text{0}\\ -1,& \text{if}\overrightarrow{X}.\overrightarrow{w}+b<0\end{array}$$

If $\overrightarrow{X}.\overrightarrow{w}-c\ge 0$ is active, then the result is positive; otherwise, it shows a negative point. $w$ and $b$ are responsible for maximizing the margin distance. There are some problems with classifying using SVM that it does not perform very well when the dataset is large, and the dataset has more noise (Maglogiannis et al., 2009). Furthermore, if the target cases overlap, or if the quantity of features for every piece of data surpasses the training data, SVM will perform poorly (Hasan et al., 2023b).

RF: RF is a DT-based bagging ensemble ML algorithm. It is used for classification and regression purposes (Breiman, 2001). A RF is an $m$-tree classifier composed of a set of tree-structured $h_1(x,Z_1),\dots ,h_n(x,Z_m),\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{t}\mathrm{h}\mathrm{e}{Z}_1,\dots ,Z_m$ are completely identical unbiased uniform random vectors. When given input $x$, each tree votes for the class with the highest support degree. A bootstrap sample of the data is used for its training set, and the data is then concatenated to generate a precise prediction. Each bootstrap sample is used as input for a DT, and the outputs from all trees are averaged to reach a consensus or mean. As a result, utilizing RF lessens the possibility of overfitting. Bootstrapping is training several separate DTs in parallel on different portions of the training dataset using different subsets of accessible characteristics (Hasan, Islam & Sohag, 2023). Bootstrapping minimizes the RF’s overall variance (Partopour, Paffenroth & Dixon, 2018). The RF, on the other hand, is unsteady, which implies that a comparatively tiny variation in the information can significantly alter the effective DT’s layout.

MLP: The MLP neural network consists of one input layer, several hidden layers, and an output layer (Cinar, 2020). An early step toward more complex NN was the representation of a single neuron called a perceptron. The depth represents the overall amount of layers, and the breadth represents the overall amount of a single layer. The backpropagation learning algorithm is used to train the neurons in the MLP. Neurons are arranged in layers. The amount of neurons present in the input layer for this pattern problem is proportional to the number of measurements. The amount of neurons within the output layer, on the other hand, is proportional to the number of classes. It is called feedforward since information is passed from $x$ to the function getting analyzed, then to the intermediate mathematical operations to define $f$, and consequently to the output $y$. Mathematically, $F(x)=y.$

LR: LR is an effective supervised ML technique for predicting the probability of a binary decision. Regularization is required in LR to avoid overfitting, especially if the number of training cases is limited or there are multiple parameters to understand (Cawley & Talbot, 2010). Furthermore, LR can classify multi-class classification. Binary data can be categorized using the logistic function as $\sigma (a)=\frac{1}{1-e^{-\alpha }}$. In this case, $e$ is Euler’s number, and $t$ is the input variable. When the value of $t$ is big enough, $(a)\to 1$ is returned, and when the value of $t$ is small, $(a)\to 0$ is returned. The logistic function is also known as the sigmoid function (Sultan et al., 2023). At the training stage, the logistic coefficients would be $a0,a1,a\mathrm{2\dots .}an$ for every instance $x1,x2,x\mathrm{3\dots .}xn$. The following formula updates the coefficients:

$$value=a_0{x}_0+a_1{x}_1+\cdots \dots +a_n{x}_n$$

$$\sigma (t)=\frac{1}{1-e^{-value}}$$

$$a=a+1\ast (y-\sigma (t))\ast (1-\sigma (t))\ast \sigma (t)\ast x$$

Each training instance’s output value is $y$, and the value of the coefficient is initially 0. Where $x$ represents the unfair input for $b0$, always being 1, and $l$ represents the learning rate. The coefficient parameters are adjusted since the training stage correctly predicts the outcome. LR creates linear frontiers, and when the amount of observations is lower than the number of features, overfitting may occur (Babyak, 2004). A major constraint of LR is the implication of linearity between independent and dependent variables.

GNB: The Bayes theorem serves as the basis for the supervised classification techniques in the naive Bayes family. The GNB variation is a naive Bayes variant depending upon that Gaussian normal distribution (Wan et al., 2019). The normal distribution is another name for the probability distribution. To predict future outcomes, GNB employs a probability distribution function. For every observation, the standard deviation and mean are first calculated. Probabilities are calculated depending on the mean and standard deviation of the test data, and a class level is assigned to each piece of data upon receipt. The probability of an event may change depending on the information available at the time (Alwateer et al., 2021), following Bayes’ theorem. Because the naive Bayes classification model is easy to implement and would not require sophisticated incremental parameter estimation, it is ideal for large datasets.

KNN: KNN is an ML algorithm that labels unknown data points by considering $k$-nearest neighbors of that unknown data point (Taneja et al., 2014). More accurately, for each test data point, KNN considers only the first $k$-nearest data points of unknown data points in the training space. To find the nearest neighbor, KNN uses Euclidean distance. Let us consider unknown data points as $A(x_1,x_2,....x_n)$ and others data point as $B(y_1,y_2,....y_n)$. The distance between unknown data points and other data points can be defined using the Euclidean distance equation as follows:

$$d(A,B)=\sqrt{{\sum }_{i=1}^n{(x_i-y_i)}^2}$$

KNN determines the distance between unknown data points and every other data point in this manner. Then, KNN selects $k$-nearest data points according to the minimum calculated distance. As KNN already finds $k$-nearest data points, now it figures out how many data points belong to each class label. Then voting is applied to identify the majority class label and assign this identified label to unknown data points. For a binary classification problem, choosing the value of $k$, an odd number, would be better to break the tie of two classes (Elkan, 2011). Roughly, if there are $m$ classes, then avoid the value of $k$ multiplied by $m$ to break the tie.

XGB or XGBoost: One well-liked boosting algorithm is gradient boosting. In gradient boosting, each predictor aims to fix the flaws of the one before it. Gradient Boosted DTs are implemented in XGBoost in sequential form. In the XGBoost algorithm, all independent variables are given weights, and also the DT that predicts outcomes utilizes these weights to predict the outcomes (Nabipour et al., 2020). Variables that the tree erroneously predicted are given a higher weight before being placed into the subsequent DT. Gradient-boosting DTs of this type are therefore assembled to develop a reliable and precise model.

DT: DT is a supervised ML classifier, $h:X\to Y$, which traverses from a tree’s root node to a leaf to determine the label associated with instance $x$. The leaf node of DT contains a specific label. Input space partitioning determines the successor child at every node along the root-to-leaf path. Typically, a preset set of splitting rules or one of $x$’s attributes is the basis for the splitting. DT uses the Gini index, which is a number that measures how accurately a split is between the groups that are categorized (Hasan et al., 2023a). A score between 0 and 1, where 1 represents a random distribution of the elements within classes, is evaluated using the Gini index.

Performance measure metrics

In this section, we discuss the performance measurement techniques employed in this study. We utilize accuracy, precision, recall, F1 score, specificity, Cohen Kappa, AUC, MCC, NPV, and PPV. The details of the metrics are outlined below:

Accuracy: Accuracy is defined as the ratio of correctly classified instances to the total number of instances. One of the fundamental performance assessment techniques is accuracy. The ratio of (True Positive (TP) + True Negative (TN)) and (True Positive (TP) + TN + False Positive (FP) + False Negative (FN)) is accuracy. In imbalanced data, sometimes accuracy leads to false illusions. Mathematically,

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

Precision: Precision indicates the accuracy of prediction of the positive instance by the model. In binary classification, the ratio between TP and the summation of TP and FP is the way to find the precision (Tharwat, 2021). Our goal with imbalanced data is to minimize FP. Precision also computes the accuracy of the minority class. As such, where FP is high, it is highly recommended to use precision as a performance measurement technique. Mathematically,

$$Precision=\frac{TP}{TP+FP}$$

Recall: Recall indicates the number of times positive events have been positively represented out of all positive events. The recall is also referred to as True Positive Rate (TPR) or sensitivity. It is the ratio of the TP and the sum of TP and FN. To reduce the FN imbalance data, recall is appropriate to measure performance. Mathematically,

$$Recall=\frac{TP}{TP+FN}$$

F1-score: The harmonic mean of precision and recall is defined as the F1-score. Only accuracy cannot judge whether a model is precise enough. When precision and recall are both high, a model makes more sense. To capture this important feature, the F1-score is calculated. It is frequently used to differentiate the effectiveness of two classifiers. Its range is [0,1], and the greater the F1-score value, the more reasonable the model is. Mathematically,

$$\text{F1score}=\frac{2\times Precision\times Recall}{Precision+Recall}$$

Specificity: The ratio between the TN and the sum of TN and FN is defined as specificity. Where negative classification is a high priority, specificity is used in that case. Mathematically,

$$\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}=\frac{TN}{TN+FP}$$

Area under the curve (AUC): A classifier’s ability to distinguish between positive and negative classifications is called AUC (Deepak & Ameer, 2019). The AUC measures how efficiently the model differentiates between negative and positive classes. The higher the AUC, the better. Its value ranges from 0 to 1. When AUC = 1, the classifier can discriminate between all Positive and Negative class occurrences. If the AUC is 0, the classifier predicts that all negatives are positives and all positives are negatives. When AUC = 0.5, the classifier is unable to distinguish between Positive and Negative class instances.

Cohen’s Kappa: Cohen’s Kappa (CK) is a statistic that measures inter-rater agreement for categorical variables. When classifying data, agreement between expected and actual labels is assessed, taking into account the possibility of coincidental agreement. The formula for Cohen’s Kappa is:

$$Cohe{n}^{,}sKappa=\frac{2\times (TP\times TN-FN\times FP)}{(TP+FP)\times (FP+TN)+(TP+FN)\times (FN+TN)}.$$

Matthews correlation coefficient (MCC): MCC in classification evaluates the correlation between expected and actual binary classifications, taking into account TP, TN, FP, and FN. The formula for MCC is:

$$\mathrm{M}\mathrm{C}\mathrm{C}=\frac{(TP\times TN-FP\times FN)}{(TP+FP)\times (FP+TN)\times (TP+FN)\times (FN+TN)}.$$

Negative predictive value (NPV): NPV in classification represents the proportion of true negative predictions among all negative predictions. The equation to calculate NPV is as follows:

$$\mathrm{N}\mathrm{P}\mathrm{V}=\frac{TN}{TN+FN}.$$

Positive predictive value (PPV): PPV in classification represents the proportion of true positive predictions among all positive predictions. The equation for PPV calculation is given below:

$$\mathrm{P}\mathrm{P}\mathrm{V}=\frac{TP}{TP+FP}.$$

Receiver operating characteristics (ROC): ROC was developed for military radar operations, specifically to describe an operator’s skill at determining whether or not an incoming radar signal was from a friendly or hostile aircraft. ROC curve helps to visualize, organize, and choose classifiers according to their performance. The TPR is depicted along the $y$-axis and the FPR along the $x$-axis, with each threshold represented by a separate point on the curve.

Experimental setting

The associated hyperparameters in the algorithms are determined with GS Cross Validation (CV) (Patil & Bhosale, 2021). Once the models have been fitted to the datasets, the algorithm’s output can be evaluated. Moreover, all the hyperparameters and their respective values are provided in Table 6.

Analysis without applying the proposed preprocessing pipeline for the inter-dataset performance discrepancy

Firstly, we analyze the five single and combined datasets created from the five different datasets without employing our proposed preprocessing pipeline. We train the ML models using an 80:20, 70:30, and 50:50 ratio of training and testing data. The performance of the classifiers is tabulated in Table 7. Moreover, among all the algorithms, SVM, KNN, RF, and MLP demonstrate a maximum accuracy of 95% with a recall of 95%. Additionally, the F1 score of SVM, KNN, RF, and MLP is 93%, which might indicate the high performance of the algorithms. However, considering the AUC score and the minimum scores of different performance measurement techniques achieved by the algorithms, the superiority of RF is more obvious than all. On the other hand, GNB shows a minimum accuracy of 32% and a maximum of 89%, but in some cases and combinations, it performs better than other classifiers. In terms of precision and AUC, XGB performs better than other algorithms. Nevertheless, the specificity (true negative) of KNN is lower than others. GNB also shows a high specificity as the accuracy is less than other algorithms in case of maximum accuracy. With these kinds of results, it is tough to find superior classifiers, and the performance of the algorithms is not stable in terms of precision and recall. In some cases, the classifiers show 0 specificity, which indicates that there were false positives, but no true negatives; all actual non-cases were incorrectly identified as positive. As such, having both of these suggests that everything was projected to be positive, whether it was an actual instance or not.

In Table 7, we only outline the minimum and maximum scores of different performance measurement techniques for individual algorithms. It is hard to find how the performance of the algorithms varies with the different ratios of training and testing the datasets. To show the variation in the performance of the algorithms, we plot the values and generate Fig. 3, where each subplot visualizes the performance of each algorithm. The scatter plot of RF is less scattered than other algorithms. More scatter indicates a lower overall performance of the classifier. SVM, KNN, XGB, and MLP are more scattered than other classifiers, where Table 7 shows the performance of those algorithms is better than others. It is clear that the performance of the algorithms is better in some cases, but they could be more stable in predicting heart disease for different datasets.

For the Switzerland and Long Beach VA datasets, most of the classifier shows poor performance because these two datasets have many missing values and many inconsistencies and outliers. When we go through the inter-dataset setup, we use the datasets in different combinations for training and testing, which shows a disaster in the results of the classifiers in Table 8. The performance goes down to 7% in minimum accuracy for SVM and KNN. In this setup, LR shows a maximum 91% accuracy with 93% precision, 91% recall, 97% specificity, and 87% AUC, score. MLP also shows 85% accuracy and recall, 93% precision, 96% specificity, 81% AUC, 89% CK, 87% MCC, 88% NPV, and 84% PPV score. DT, RF, XGB, and GNB show more than 80% accuracy and recall and more than 90% precision. Among these four classifiers, the RF and GNB have good specificities 96%, and 97% respectively. From this table, we cannot clearly define the classifier’s performance. To demonstrate the algorithm’s performance more clearly, we plot all of the performance measure technique values of different combinations in Fig. 4. This figure shows the classifiers’ performance individually in different subplots. We get a clear indication of the classifier’s performance. However, the values are more scattered, which is proof of an inter-dataset performance discrepancy in heart disease prediction.

The ROC curve also shows the performance of the classifiers for three different stages (without pipeline, PCA equipped, RF equipped) of different datasets individually. In Figs. 5–10 shows different ROC curves of different datasets in different states.

Analysis of applying PCA to mitigate the inter-dataset performance discrepancy

We use a preprocessing pipeline to extract features using PCA from the datasets to minimize the inter-dataset performance discrepancy. In terms of the individual and combined datasets, the performance of the classifiers improves compared to the previous analysis, where we did not use our proposed preprocessing pipeline. The results of the classifiers for individual datasets are in Table 9. In this table, GNB achieves a minimum accuracy of 60% compared to 32% in Table 7 for pipeline data without preprocessing. In Table 7 the maximum accuracy is 95%, but after using the preprocessing pipeline, we get 100% accuracy, precision, recall, and an F1 score for SVM, DT, RF, XGB, and GNB. In addition, KNN and LR show 99% accuracy, precision, recall, and F1-score, while MLP shows 98% accuracy, recall, F1 score, and 99% precision. All the classifiers show 100% specificity and an AUC score. In addition, the classifiers’ minimum score is higher than the result in Table 7. All the classifier’s results vary within the minimum and maximum ranges that are tabulated in Table 9. All the performance measure values are visualized in Fig. 11 by some scatter plots, where each subplot represents the scores for individual classifiers. The figure clearly shows that the results are not as dispersed as in Fig. 3. It indicates that our proposed methodology improves the classifier’s performance at a significant level.

As with the single and combined datasets, we use the same preprocessing pipeline in the different combinations of the dataset that is presented in Table 5. Table 8 and Fig. 4 show the performance of the classifiers and indicate that the inter-data discrepancy occurred here. However, after applying PCA, we mitigated the discrepancy issue and improved the accuracy significantly. In Table 10, GNB achieved 57% accuracy, the lowest among the algorithms, whereas, in Table 5, the minimum accuracy was only 7% by SVM and KNN. On the other hand, the maximum accuracy for the dataset that did not go through the proposed preprocessing pipeline was 91% by LR. After applying the PCA, we get 92% accuracy by RF. Though the highest range of accuracy change is insignificant, the data discrepancy removal is significant. In Fig. 12, the subplots are not too scattered like in the previous Fig. 4, which proves the removal of data discrepancy of the inter-dataset setup. If we compare Table 10 with the previous Table 5, we get a clear indication that PCA improves the overall performance of the classifiers. The scores are unstable in Table 5, and performances vary in different combinations. However, in Table 10, all the classifiers show stable performance with good accuracy. Here RF shows the superiority of prediction over other classifiers. All the second columns of each graph show the performance of the algorithms, and we get a clear indication that the performance of the PCA-based preprocessing pipeline helps to mitigate the data discrepancy.

Analysis of applying RF to mitigate the inter-dataset performance discrepancy

Like PCA, we employ RF to select the features in the preprocessing pipeline to mitigate the inter-data performance discrepancy problem in heart disease prediction. We use this process in the single and combined dataset analyses and the different combinations of the dataset analyses. “Analysis of Applying PCA to Mitigate the Inter-Dataset Performance Discrepancy” shows the analysis results on the dataset that does not undergo our proposed preprocessing pipeline. The results of the classifiers for the individual and combined datasets and different combinations of the datasets are not satisfactory in this medical domain. Also, we show the result of PCA enabling the pipeline in “Analysis of Applying PCA to Mitigate the Inter-Dataset Performance Discrepancy”, show the classifier’s good performance, and prove the inter-data performance discrepancy mitigation. In this section, the results obtained by the classifiers for the single and combined datasets are in Table 11, and the performance of the classifiers for the different ratios of training testing is visualized in Fig. 13.

Table 11 only displays the lowest and highest results of different performance measurement approaches for specific algorithms. Concerning all algorithms, SVM and RF show 98% maximum accuracy, precision, recall, and an F1 score; specificity and an AUC score are both 100%, which is the highest value among all algorithms. Regarding the minimum accuracy score for accuracy, precision, recall, and an F1 score metric, RF also outperforms all other algorithms. Compared to Table 9, it is clear that SVM and RF show the highest value of 100% maximum accuracy, precision, recall, and an F1 score using PCA-based FE. GNB algorithms perform poorly compared to other algorithms in terms of accuracy, precision, recall, F1 score, and specificity. On the other hand, GNB shows 100% maximum accuracy using PCA-based FE. XGB shows the second-highest performance, which is 95% maximum accuracy, precision, recall, and an F1 score; 97% specificity, and 99% AUC. Compared to Table 9, which is PCA-based FE, and Table 11, which is RF-based FS, we can conclude that all the algorithms show better performance in the case of PCA-based FE. The subplots in Fig. 13 are not as scattered as in Fig. 4, but they are a little more scattered than in Fig. 12, indicating a significant improvement in the classifier’s performance.

As with the single and combined datasets, we employ the same preprocessing pipeline in the numerous different dataset combinations shown in Table 12. Table 10 and Fig. 4 show the classifiers’ performance, which proves that the inter-data discrepancy has arisen. With the help of RF-based FS, the discrepancy issue is mitigated at a moderate level, improving the performance significantly higher than PCA-based FE for some classifiers. The RF and XGB algorithms show 96% maximum accuracy, precision, recall, and an F1 score; the AUC score is closed at 100%. KNN shows 94% maximum accuracy, precision, recall, and an F1 score. KNN also shows 97% maximum specificity, which is the highest among all other algorithms. DT achieves 95% accuracy, precision, recall, F1 score, specificity, and AUC score. These results show that RF, XGB, KNN, and DT algorithms improve performance significantly more than PCA-based FE using different combinations of datasets. In the case of SVM, the performance is nearly identical to that of the PCA-based FE result. GNB, LR, and MLP algorithms perform better in PCA-based FE than in RF-based FS. Figure 14 shows that classification performance for some algorithms improves more when using RF-based FS rather than PCA-based FE.

We use ROC to show the performance of the classifiers for checking the inter-dataset performance discrepancy among the five individual datasets. In Fig.15, we use one dataset for training and one for testing. The performance of the classifiers without preprocessing pipeline stages is unstable, and accuracy is below average in most cases. It indicates the inter-dataset performance discrepancy. After applying the PCA in the preprocessing pipeline, the performance of the classifiers is increased. It demonstrates that PCA feature extraction techniques can reduce the inter-dataset performance discrepancy. Also, the last columns show the ROC of the RF feature selection pipeline.

The results obtained by training the models using two datasets and testing only one dataset are visualized by the ROC curves in Fig. 16. The combination of 6 to 10 of Table 5 is here for the three phases. However, the performance of the classifiers is too low, and in most cases, it falls below the average line indicated in the first column subplots. After applying PCA and RF in the preprocessing phase, the performance of the classifiers improves dramatically, indicating that this proposed preprocessing pipeline minimizes the inter-dataset performance discrepancy problem. However, the performance of the classifiers is high in both PCA and RF, but compared to these two, RF feature selection shows better results in these cases. In conclusion, RF as a classifier is superior in predicting heart disease, and our proposed preprocessing pipeline mitigates the inter-dataset performance discrepancy problem in this domain.

The ROC curves of Fig. 17 are generated from the combinations of 11 to 15 of Table 5 where three datasets are in the training phase and 1 in the testing phase. Like the previous two phases, the classifiers’ performance is too low when we don’t apply the proposed preprocessing pipeline. Also, the performance is more scattered, and most cases are below 50% in accuracy. After applying the PCA and RF, we can recover these issues. The performance of the classifiers increases in a good position, and in most cases, it shows more than 85% accuracy. More specifically, the performance of the classifiers is better in the RF-integrated pipelines than in PCA-integrated pipelines.

When four datasets are in the training phases and one in the testing phases, the inter-dataset performance discrepancy occurs. Classifiers show a low performance that is not tolerable in this sensitive domain. We also apply our proposed preprocessing pipeline separately, where PCA and RF are mainly responsible for the dimensionality reduction of the datasets we already discussed. The performance of the classifiers increases after applying the preprocessing pipeline. The ROC is in Fig. 18.

In Table 5, the combinations of 21 to 25 hold two datasets as training and two as testing phases. Here, also the classifiers show a low performance and unstable classification report. After applying the proposed preprocessing pipeline, the performance of the classifiers increases, indicating the mitigation of the inter-dataset performance discrepancy to predict heart disease. The performance of the classifiers is visualized in Fig. 19.

The last five combinations of Table 5 contain three datasets as training and two as testing phases. The performance of the classifiers is also low when we do not apply the proposed preprocessing pipeline; however, after applying the pipelines with PCA and RF, all of the classifier‘s performances increase, and the inter-dataset performance discrepancy mitigates. Also, all the ROC curves are in Fig. 20.

Discussion

The main motivation of this study is to build a global ML model that can predict heart disease from any publicly available dataset. This model can also predict heart disease from multi-sensor data directly observed from the human body. After mitigating the inter-dataset performance discrepancy issue, the trained models are too flexible to other real-time data for prediction. To achieve this goal (inter-dataset performance discrepancy), we train different ML classifiers using the datasets and measure the performances of the classifiers. Besides this, we also train the classifiers using different combinations of the datasets, and in the testing phase, we also use different combinations of datasets. All the classifiers show unacceptable performances in the medical domain, where high accuracy is an urgent issue (Sumathi & Poorna, 2016). The problem behind the low performances of the classifiers is called the inter-dataset discrepancy problem. To mitigate this issue, we proposed a preprocessing pipeline where we apply both feature extraction and feature selection separately.

When we apply PCA as a feature extraction technique in the pipeline, the performance of the classifiers goes to a reasonable level. It sometimes rises to 95% accuracy, precision, and recall. Performances of the classifiers are stable, and the inter-dataset performance discrepancy problem solves most of the combinations. After applying RF in the preprocessing pipelines, the performances of the classifiers go high and stable. We get a good recall and specificity of the highly acceptable classifiers in this sensitive domain. Performances of the classifiers vary in different combinations. In some cases, it goes up; in others, it goes down. Among all the algorithms, RF is superior in predicting heart disease for individual datasets and inter-dataset setups. RF shows 96% accuracy with 96% precision, recall, and 99% AUC score. The results indicate that our proposed preprocessing pipeline solves the inter-datasets performance discrepancy, and the model is flexible to real-time observed data by sensors. We can use RF for preprocessing and classification purposes to solve this issue in heart disease prediction.