Analyzing patients satisfaction level for medical services using twitter data

Proposed methodology

This study aims at analyzing patients’ satisfaction with the healthcare industry and notifying physicians of the factors causing the lower satisfaction level. It allows physicians to understand what is more important to patients and how well they are interacting with them. Similarly, hospitals may quantify patients’ experiences and find areas for improvement by evaluating the positive and negative comments. As illustrated in Fig. 1, this study employs a machine learning approach to perform sentiment analysis regarding medical services. Our study approach involves six steps as follows:

Step (1): Data collection

In this step, we discuss the data collection process. We collected our dataset from Twitter using the Tweepy (https://www.tweepy.org/) library. To target different healthcare communities, we utilized three specific keywords, which are “health care”, “health care services”, and “medical facilities” to extract data from Twitter. We extract this data during Covid-19 in July 2021. As the initial dataset, we obtained 5,561 tweets for health care, 6,351 tweets for health care services, and 5,867 tweets for medical facilities. As the tweets for each keyword are not balanced, we randomly selected 5,000 from each keyword. Therefore, the final dataset of our study yielded a total of 15,000 tweets.

Step (2): Data preprocessing

After collecting the dataset, we need to clean the dataset by performing the complete preprocessing stages to increase the proposed models’ learning performance. The text is converted to lowercase during the preprocessing, followed by stop words and punctuation removal. Next, stemming is performed using the Porter Stemmer (https://tartarus.org/martin/PorterStemmer/) library to obtain the root form of each word. Stemming reduces the complexity of feature space and improves the learning process of the machine learning models (Kotsiantis, Kanellopoulos & Pintelas, 2006; Uysal & Gunal, 2014). Preprocessing steps for text cleaning are

• Remove number and punctuation: Number and punctuation are not important to evaluate the sentiment of the text, and they can create unnecessary complexity in the feature set. Thus, we removed them to reduce the complexity of the dataset. We used a regular expression in Python to remove numbers and punctuation.

• Convert to lowercase: Dataset can have both upper case and lowercase letters in the text, which increases redundancy in feature sets such as ‘Go’ and ‘go’. These words are both the same, but because of differences in case, they are different for learning models. Thus, converting to lower case technique solves this problem by converting all words later into lower case. We used the tolower() Python function for it.

• Remove stopwords: The text contains many meaningless words but makes the sentence more appropriate, such as the, a, an, etc. We remove these words to make the dataset clean without meaningless words. We have done this using python NLTK library (Loper & Bird, 2002).

• Stemming: It is also a very worthy technique in which we convert each word into its root form to reduce redundancy in the dataset as go and going are the same meaning words but the difference in the form they are separate features for machine learning model. Stemming will convert ‘going to go’, ‘goes to go’, ‘gone to go’, so complexity can be reduced. We have done this using the Porter stemmer library (Karaa & Gribâa, 2013). Table 2 shows the sample data before and after the processing steps.

Step (3): Sentiment analysis

In this step, we perform the annotation process to label our dataset. Figure 2 shows the number of samples corresponding to each sentiment in the dataset. The dataset is annotated using the TextBlob (https://textblob.readthedocs.io/en/dev/) library. TextBlob is a popular sentiment analysis lexicon-based Python library model that promises simplified text processing (Sarkar, 2019). The distribution of the most used words in the Tweets dataset is illustrated in Fig. 3 which indicates that ’healthcare’ is one of the most widely used words while giving views on medical services.

Step (4): Feature extraction

In this step, we used text data for sentiment classification using a supervised machine learning approach. Machine learning can not directly process the text data as they need numerical representation of text data (Rustam et al., 2020). For that, different feature extraction techniques are available in text mining domains. In this study, we used feature extraction techniques named term frequency-inverse document frequency (TF-IDF).

TF-IDF is used for feature extraction to train the machine learning models. TF-IDF is most widely used in text analysis and music information retrieval (Mujahid et al., 2021). TF-IDF assigns a weight to each term in a document based on its TF and IDF. The terms with higher weight scores are considered to be more important. TF-IDF is a product of TF and IDF, as we explain mathematically below (1)${\displaystyle tf=T{F}_{t,j}.}$

Here, tf is a term frequency of term t in document j. now IDF can be concluded as: (2)${\displaystyle idf=log\left(\frac{N}{d_t}\right).}$

Here, N is the number of documents and d_t is the number of documents containing term t. TF-IDF can be defined as (3)${\displaystyle tf-idf=T{F}_{t,j}\ast log\left(\frac{N}{d_t}\right).}$

This study proposed a hybrid approach using the LSTM-ETC model for analyzing patient’s satisfaction level through Twitter data. LSTM is suitable for large datasets to extract valuable features using various layers, whereas ETC is superior for text data. We utilized an embedding layer consisting of 7,000 input dimensions and 100 output dimensions. This embedding layer will receive text data as input and output a numeric representation for LSTM. We employed a five-layer LSTM model, with the second layer containing a 20% dropout rate and 200 LSTM units. The activation function of one dense layer is RELU, while that of the other is SOFTMAX. Then compile and fit the LSTM model with categorical loss to derive valuable features. The extracted features are subsequently utilized by the ETC model to produce extremely accurate predictions. Before making predictions, the ETC model must first be fine-tuned. The proposed architecture for analyzing the patient’s satisfaction level is shown in Fig. 4.

Step (5): Model selection

This study leverages eight machine learning models for the task at hand. We deployed state-of-the-art models with their best hyper-parameters setting which get using the model tuning. The hyper-parameter setting of these models is shown in Table 3.

Step (6): Model evaluation

We used four evaluation parameters for the machine learning model’s performance measure such as accuracy, precision, recall, and F1 score (Rehan et al., 2021). These evaluation parameters can be calculated using the confusion matrix terms which are true positive (TP), true negative (TN), false positive (FP), and false negative (FN). We define used evaluation parameters below:

Accuracy: It is used to measure the correctness of the learning model by dividing the number of correct predictions by the number of total predictions. The accuracy score range is between 0 and 1. Here 0 is the lowest score and 1 is the highest. We can define accuracy as: (8)${\displaystyle AccuracyScore=\frac{totalnumberofcorrectpredictions}{totalnumberofpredictions}.}$

or, (9)${\displaystyle AccuracyScore=\frac{TP+TN}{TP+TN+FP+FN}}$

here,

• TP: When the model prediction example is YES and the actual label of the example is also YES.

• TN: When the model prediction example is NO and the actual label of the example is also NO.

• FP: When the model prediction example is YES but the actual label of the example is NO.

• FN: When the model prediction example is NO the actual label of the example is YES.

Precision: It can be calculated as TP divided by the sum of TP and FP. The precision score range is between 0 and 1. Here 0 is the lowest score and 1 is the highest. We can define accuracy as: (10)${\displaystyle PrecisionScore=\frac{TP}{TP+FP}.}$

Recall: It can be calculated as TP divided by the sum of TP and FN. The recall score range is between 0 and 1. Here 0 is the lowest score and 1 is the highest. We can define accuracy as: (11)${\displaystyle RecallScore=\frac{TP}{TP+FN}.}$

F1 Score: It is also known as the F measure. It is a harmonic means of precision and recall. The F1 score range is between 0 and 1. Here 0 is the lowest score and 1 is the highest. We can define accuracy as: (12)${\displaystyle F1Score=2\ast \frac{PrecisionScore\ast RecallScore}{PrecisionScore+RecallScore}.}$

RQ 1: What are the sentiments of people for medical services worldwide?

This study evaluates the sentiment of people all around the world for the medical facilities provided by the health care departments. We first used lexicon-based techniques to measure the polarity in tweets related to health care. According to the results of TextBlob and Vader ratio of positive sentiment is more as compared to the negative sentiments as shown in Fig. 5. According to the TextBlob results the ratio of positive sentiments in the dataset is 41.62% and negative sentiment 9.4%. While we got some similar results with Vader also in comparison as it find the ratio of positive sentiments 54.44% and negative sentiments 14%. These results show that people are happier towards the facilities in healthcare domains.

TextBlob employs a simple algorithm that is unable to capture complicated emotional nuance. Vader, on the other hand, is designed specifically for social media tweets using a lexicon, grammatical strategies, and pre-trained lexicons. Vader comprehends the negation, intensity, and punctuation of both formal and informal languages. However, we combined TextBlob and Vader to take advantage of their respective strengths and capabilities to accurately label the raw tweets. Figure 5 shows the sentiment count using TextBlob, Vader, and a combination of TextBlob and Vader techniques. The annotated dataset with TextBlob-Vader is reliable, and almost the same numbers of positive, neutral, and negative tweets are obtained with this combination.

RQ 2:How effective is our proposed machine learning and deep learning approaches for the classification of sentiment on healthcare tweets?

The dataset contains three classes including positive, negative, and neutral. To train and test models for classification tasks the data is split into 80 to 20 ratios for training and testing, respectively. Experimental results are given in Table 4 where the evaluation is carried out in terms of precision, recall, and F1 score.

Results indicate that the ETC shows the best performance for sentiment classification with a 0.88 accuracy score which is the highest among all the models. It is followed by the RF which has an accuracy score of 0.87. SVM and SGDC have 0.86 accuracy scores. Results suggest that tree-based classifiers perform better than linear classifiers.

In addition to machine learning models, this study also deployed deep learning models for comparison with machine learning models such as long short term memory (LSTM) and convolutional neural networks (CNN). Both models are selected regarding their reported performance for the task at hand and deployed with state-of-the-art architecture. Each model takes the input through an embedding layer with a 5,000 vocabulary size and 100 output dimensions. In the LSTM model, the embedding layer is followed by a dropout layer with a 0.2 dropout rate and an LSTM with 100 units. The LSTM layer is followed by the dense layer with three neurons and a Softmax activation function. While in CNN, the embedding layer is followed by the 1D CNN layer with 128 filters and 4 ×4 kernel size with rectified linear unit (ReLU) activation function. The CNN layer is followed by the max-pooling layer with a 4 × 4 pool size to extract important features. After the max-pooling layer, we used flatten layer to convert the 3 dimension data to 1 dimension which is followed by the dense layer with three neurons and the Softmax activation function. Both models are compiled with the ‘Adam’ optimizer and categorical_crossentropy loss function. We fitted the model with 100 epochs and 16 batch sizes. The architecture of the used models is shown in Table 5.

Table 6 shows the results for deep learning models and indicates that LSTM performs better as compared to CNN because of its recurrent architecture. Recurrent architecture helps to perform well on sequential text inputs. Figures 6A & 6B show the per epochs results in terms of each evaluation parameter.

Besides the sentiment analysis using the machine learning and deep learning models, the sentiment frequency distribution indicates that the ratio of positive comments for the health services is substantially higher than the negative sentiments. It shows the higher level of satisfaction of the public regarding the provided services by health institutions.

Cross-validation to analyze efficacy of proposed model

We employed the K-fold cross-validation methodology in new experiments to verify the results and efficacy of the proposed method. Table 7 demonstrates that ML algorithms, particularly SGD, obtained a mean accuracy of 82% and a standard deviation of ±0.03, demonstrating their efficiency. RF and SVM achieved the same mean accuracy, but SVM obtained a better standard deviation rate. The mean accuracy of DL and CNN models was 84% and 91%, respectively, with ±0.05 and ±0.04 standard deviations. The ML or DL models are not performed excellently using a cross-validation dataset. Cross-validation results demonstrate that the performance of our proposed method is superior, as it attained a remarkable 93% accuracy with a standard deviation of ±0.02.

We performed more experiment analysis using correct and wrong predictions made by the models and validated the efficacy of the proposed model. Table 8 shows the result analysis through correct and wrong predictions. The GNB model attained the highest number of wrong predictions (53%), followed by KNN with 39%. These two models (KNN and GNB) are inefficient and do not contribute to good predictions. Result analysis demonstrates that ML models are incapable of making correct predictions, while DL-based CNN and LSTM models are better than ML in terms of making wrong predictions. CNN attained 15% of its predictions wrong. The proposed LSTM-ETC model attained minimum wrong and maximum correct predictions (5% and 95%). This analysis validates that the proposed model is better to understand and predict sentiments.

The effectiveness of DL models is assessed through ROC curves at different classification factors. The true positive rate and the false positive rate curves demonstrate the effectiveness of ROC-AUC for making the difference between positive and negative classes. The most popular area under the ROC curve (AUC-ROC) is employed to assess the model’s overall performance. They visualize classes very clearly. Figure 7 visualizes the AUC curves of the proposed model, which showed that positive and neutral classes attained superior 0.98 AUC and negative classes attained 0.93.

Figure 8 indicates a comparative analysis of top-performing models (ETC, CNN, LSTM, and a proposed hybrid model) using the confusion matrix. In the confusion matrix, 0 represents the positive class, 1 represents the neutral class, and 2 represents the neutral class. ETC achieved 2,142 true positives, CNN achieved 1,950, LSTM achieved 2,098, and the proposed model achieved 2,115 true positive values (TPV) for positive class. In addition, the confusion matrix analysis also showed better overall performance than the other three top-performing models with the fewest mistakes.

Implications of research

This study can be beneficial to the major stakeholders, including the government, health departments, organizations, policymakers, and patients. They can make policies and enhance or update their services and technological resources by analyzing the satisfaction level of these patients in this study. Also, stakeholders can capture changes and trends in healthcare through the study results. The collected dataset contains tweets worldwide and it’s a limitation of this study because our extracted tweets are from well-developed areas where facilities are good as shown in Fig. 9. There is a lack of tweets from under-developing countries such as India, Pakistan, Afghanistan, Bangladesh, etc. where healthcare facilities are not good. May the inclusion of tweets from these developing countries increase the ratio of negative sentiments.

Similarly, for the classification approach, the dataset is too small and imbalanced as the ratio of sentiments positive, negative, and neutral is not equal to train the learning models which can affect the validity of machine learning and deep learning models.

Figure 10 shows country-wise analysis of tweets as positive, negative, and neutral sentiments. In the United States, users have 48% positive opinions about medical services and 0.08% negative opinions about the services. England is on the second number, regarding the tweets posted by the user, and provides 60% positive sentiments and 16% negative sentiments. Between Mexico and Canada, there are limited health facilities that are 0% positive and 97% people dissatisfied with the health services and medical facilities. Figure 10 visualize country wise sentiments of peoples regarding the health services and facilities.

Performance comparison of proposed model with existing studies

Table 9 shows a comparison between the performance of the proposed model and that of other studies presented in the literature. The authors of the study (Ji, Chun & Geller, 2013) used Facebook tweets and Python NLTK tools to label the tweets. They use different ML models to figure out how people feel, and 87% of the time, they are right. Yang, Lee & Kuo (2016) used MedHelp to get text data for a study, and 13,535 tweets were used as a source. The authors used the AFINN sentiment analyzer and did topic modeling with latent Dirichlet allocation. The authors don’t do anything at all. Gopalakrishnan & Ramaswamy (2017) used a collection of drug reviews that only had about 3,600 reviews. They were 79% accurate in how they put the reviews into classification. The next two studies (Alayba et al., 2017; Rahim et al., 2021) used hand-annotated datasets to classify health tweets and got less accurate results. SuryaPrabha & Balakrishnan (2021), labeled the data with the sentiword-net dictionary, but they also got less accurate results. The suggested methodology produced the best results, utilizing the TextBlob approach on 18,000 custom tweets, Additionally, the proposed model was evaluated and trained using different ML models with Vader lexicons.

Discussion and limitations

In this research, we determine how to analyze patients’ satisfaction levels with medical services and sentiment using Twitter data. Sentiment analysis can be used to identify large data sets of free-text comments related to healthcare improvement. Twitter has established itself as an important element of the social media landscape, where people interact and share their views and concerns regarding public, governmental, and social services. These views are replete with ample information to analyze public opinion, formulate policies, comprehend the effect of events, and devise novel solutions to specific issues. This study provides an early attempt to monitor public attitudes toward healthcare services through the use of novel data sources as well as a technique for leveraging social media data. This study leverages many well-known machine learning and deep learning models to analyze sentiments regarding medical services. A comprehensive discussion is presented of the performances and behaviors of the conventional and deep learning approaches used.

Experiments are performed using several well-known machine learning models, including support vector machines, logistic regression, Gaussian naive Bayes, extra tree classifiers, k-nearest neighbor, random forests, decision trees, and AdaBoost. Results suggest that ETC achieves the best performance among machine learning models, with an accuracy score of 0.88. However, the proposed model surpasses it with an accuracy of 0.95 when predicting positive, negative, and neutral sentiments. The proposed model shows superior performance. The majority of people are satisfied with the services, and a very small number of people are dissatisfied with them.

Twitter can offer significant benefits to organizations and healthcare departments by serving as a method for evaluating healthcare services. In this era, healthcare departments and several other organizations possess the capability to actively monitor social media platforms such as Twitter. The organization faces obstacles in its ability to effectively monitor and analyze lengthy tweets or a large volume of tweets to improve decision-making and get insights into prevalent health-related issues. The analytical process is time-consuming, and the findings obtained are inaccurate in terms of the manual examination of tweets. The patients provide feedback about the health facilities and resources, as well as using online platforms to schedule appointments, among other activities. This study examines the levels of patient satisfaction with medical services. The utilization of Twitter data to extract the thoughts and attitudes conveyed by its users. Organizations have the potential to get valuable insights by analyzing tweets or doing sentiment analysis on articles about various health-related topics. The tweets provided data on the patients, who were routinely queried about their encounters with the medical personnel, hospital services, and other healthcare-related establishments. The analysis of Twitter’s outcomes can provide valuable insights for healthcare organizations seeking to improve the quality of their services. The prevailing allocation of resources is presently concentrated on improving the well-being of medical practitioners through the pursuit of machine learning research and development utilizing extensive quantities of health-related data. The organizations can make significant decisions, and hospital staff communicate with their patients more politely and provide correct information.

There are some limitations to this study, including the fact that the dataset was collected from different parts of the globe, the difficulty of analyzing every country due to their unique healthcare systems, and the potential bias of generalizing the results. Second, the collected healthcare services and facilities dataset is small for training transformer-based deep learning models without overfitting.