Persian sentiment analysis of an online store independent of pre-processing using convolutional neural network with fastText embeddings

Dataset

Having access to a large dataset with richness and content integrity is indispensable to train a deep model. Most available datasets to train a deep model and sentiment analysis are in English. To collect a rich dataset, web-mining methods were used and the reviews on the Digikala website were extracted which were in Persian. Posted reviews by buyers express their level of satisfaction with their purchase and product features. After submitting their reviews, buyers could choose between the “I suggest” and “I do not suggest” options. These two options were extracted and used in the model as labels for the problem of sentiment analysis. Our goal was to analyze the opinions of users of the Digikala website, so we extracted the data of the section related to digital goods using web-mining libraries such as the Beautiful Soup (Richardson, 2020). Beautiful Soup is a Python package to parse XML and HTML documents and it is useful for web scraping (Hajba, 2018).

Pre-processing

One of the first steps in natural language processing problems has always been data pre-processing. At this stage, the texts need to be cleaned and prepared to begin the analysis. In Persian, this stage is even more difficult and important because it has its complexities. This field has attracted many researchers in the last decade, therefore, libraries and algorithms in the field of pre-processing in Persian have been developed (Mohtaj et al., 2018; Nourian, 2013) which have become more complete and better over time. However, these algorithms cannot work as well as similar algorithms in English and need further development. We are seeking a way to achieve an accurate result by avoiding the complications of data pre-processing steps in Persian. Regular expressions are used for data pre-processing in all of the following steps using the “re” library (Rachum, 2020) in python. Pre-developed libraries for the Persian language have not been used to perform data pre-processing steps and we assume that the use of fastText and skip-gram in creating word embedding reduces the need for complex pre-processing.

Normalization

In Persian, some letters are not unique and may have different alternatives to other languages such as Arabic. For example, the letter “ی ” is Persian, but the letter “ي ” is Arabic, and these two letters will likely be used interchangeably. This causes the created words to be considered as two different words. In this way, they may be considered separately in the calculations and a vector can be drawn for each with its characteristics. To solve this issue, it is necessary to use the standard form for all available texts.

Tokenization

Tokenization is a stage in which attempts are made to divide sentences into meaningful words and phrases that can be considered as a suitable input for the next steps. The main challenge of the Persian language at this stage is that sometimes there are no clear boundaries between phrases as a result of three different modes of spacing in Persian. In other words, phrases in Persian can be without space, with half-space, or with space, which is often mistakenly used instead of each other. For instance, the words “نرم افزار” and “نرم افزار”, which both mean software, are written with both space and half-space forms. If the wrong form is used, the phrase border will be mistakenly recognized as two separate words “نرم ” and “افزار”. Vice versa, phrases that consist of several words can be considered as one word due to a mistake in using space. For example, the word “از مسیر دیگر”, which means “from another path”, maybe written as “ازمسیردیگر” without any spaces. These kinds of mistakes blur the line between phrases and words and make it difficult to pre-process.

Stemming

The stemming process seeks to remove part of the word in such a way that the root of the word is determined (Willett, 2006). The root of the word does not necessarily mean the dictionary root of the word and is acceptable in cases where it can improve the performance of the model. For example, we can refer to the phrase “رنگهایشان ”. In this phrase, “رنگ ” means color, and “ها” is used to represent plural and “یشان ” for determination of ownership. A significant number of stemming algorithms use the following rule (Mohtaj et al., 2018):

(possessive suffix)(plural suffix)(other suffixes)(stem)(prefixes)

Stemming is a rule-based process that is usually done by removing suffixes and prefixes. Consequently, it cannot be used in other languages and each language requires its algorithms. Stemming algorithms are being developed in Persian but due to the complexity of the Persian language, their performance needs to be improved.

Pseudo labeling

In classification problems, it is a common problem that a large number of samples do not have labels and therefore cannot be used in model training. Techniques such as pseudo-labeling can be used to overcome this problem and determine the labels of some samples (Lee, 2013). The first step in pseudo-labeling is to develop a model based on labeled samples in the dataset that is in charge of determining the label of unlabeled samples. Only labels that have been estimated with high confidence are accepted. In the next step, another model is developed using training data along with the new labeled data which can be used to predict test data with higher accuracy. In this way, 104.8 thousand Negative Feedback reviews and 30.5 thousand Positive Feedback reviews were labeled and could be used in the dataset for subsequent analysis. As will be shown in the results section, this method had a significant impact on improving accuracy.

Data balancing

Unequal distribution of data in different classes in a classification problem leads to data imbalance. The class with the most data is called the majority class, and the class with the least data is called the minority class. In these cases, the models tend to ignore the minority class and predict in favor of the majority class. Many machine learning models, such as Support Vector Machine, Naïve Bayes, Decision Tree and Artificial Neural Network cannot have good results in this situation (Díez-Pastor et al., 2015; Vorraboot et al., 2015). In general, data balancing solutions can be divided into two categories; over-sampling and under-sampling. The goal of both solutions is to approximate the number of data distributed in the minority and majority classes. In over-sampling, this is done by increasing the amount of data in the minority class, and in under-sampling by reducing the amount of data in the majority class. In the present problem, we used the random oversampling method to balance the dataset.

Feature extraction

Sentiment analysis model

Convolution neural network

Using CNN has shown high accurate results based on studies in English texts (Nedjah, Santos & De Macedo Mourelle, 2019). This model can receive and analyze word embedding as input instead of images, which are also effective in this area (Kim, 2014). Each row of the input matrix represents a word. Figure 1 shows the architecture of a CNN model used for the NLP classification problem (Zhang & Wallace, 2015). This figure shows how a CNN model treats a 6-word sentence. The matrix formed for this sentence is analyzed by six different convolution filters and converted to maps of attributes with dimensions of 1 × 4, 1 × 5 and 1 × 6. Finally, the pooling operation is performed on the maps and their outputs are merged to create a unique vector that can be used as input for the SoftMax layer to determine the class. The CNN model used in this article is based on the mentioned model and its architecture is shown in Table 1.

Table 1:

The CNN model structure.

Layer (type)	Output shape	Number of parameters
embedding_2 (Embedding)	(None, 400, 100)	11,147,700
dropout_8 (Dropout)	(None, 400, 100)	0
conv1d (Conv1D)	(None, 400, 128)	38,528
global_max_pooling1d (Global)	(None, 128)	0
dense_6 (Dense)	(None, 64)	8,256
dropout_9 (Dropout)	(None, 64)	0
dense_7 (Dense)	(None, 16)	1,040
dropout_10 (Dropout)	(None, 16)	0
dense_8 (Dense)	(None, 1)	17
Total parameters:		11,195,541

DOI: 10.7717/peerj-cs.422/table-1

Bidirectional long short-term memory

Another deep model used to solve the problem is BiLSTM. The LSTM model can decide which information is useful and should be preserved and which information can be deleted based on the dataset it has trained with. The LSTM has been widely used in NLP such as long document categorization and sentiment analysis (Rao et al., 2018). Figure 2 is a demonstration of an LSTM cell used in this article, which has an input layer, an output layer and a forget layer (Gers, 1999). Based on the figure, the LSTM cell mathematically expressed as follows:

(1) $$f_{\text{t}}=\text{σ}(W_{\text{fh}}{h}_{\text{t}-1}+W_{\text{fx}}{x}_{\text{t}}+b_{\text{f}})$$

(2) $$i_{\text{t}}=\text{σ}(W_{\text{ih}}{h}_{\text{t}-1}+W_{\text{ix}}{x}_{\text{t}}+b_{\text{i}}$$

(3) $${\tilde{c}}_t=tanh(W_{\tilde{c}h}{h}_{t-1}+W_{\tilde{c}x}{x}_t+b_{\tilde{c}})$$

(4) $$c_t=f_t.c_{t-1}+i_t.{\tilde{c}}_t$$

(5) $$o_t=\sigma (W_{oh}{h}_{t-1}+W_{ox}{x}_t+b_o)$$

(6) $$h_t=o_t.\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(c_t\right)$$where $x_t$ denotes the input; $h_{t-1}$ ,and $h_t$ denote the output of the last LSTM unit and current output; $c_{t-1}$, and $c_t$ denote memory from the last LSTM unit and cell state; $f_t$ denotes forget gate value; $W_i$, $W_{\tilde{c}}$, and $W_o$ are the weights; $b$ is the bias; the operator ‘·’ denotes the pointwise multiplication of two vectors. In LSTM, the input gate can decide what new information can be stored in the cell state, also the output gate can decide what information can be output based on the cell state. By combining the ideas of BRNN and LSTM it is possible to achieve Bidirectional LSTM (BiLSTM) which has better performance than LSTM in classification processes especially in speech processing tasks (Graves & Schmidhuber, 2005). Therefore, this article uses the BiLSTM structure, and Fig. 3 is shown a basic structure of the BiLSTM network (Yildirim, 2018). The BiLSTM model used in this article architecture is shown in Table 2.

Table 2:

The BiLSTM model structure.

Layer (type)	Output shape	Number of parameters
embedding_1 (Embedding)	(None, 400, 100)	11,147,700
dropout_4 (Dropout)	(None, 400, 100)	0
bidirectional_1 (Bidirection)	(None, 64)	34,048
dropout_5 (Dropout)	(None, 64)	0
dense_3 (Dense)	(None, 64)	4,160
dropout_6 (Dropout)	(None, 64)	0
dense_4 (Dense)	(None, 16)	1,040
dropout_7 (Dropout)	(None, 16)	0
dense_5 (Dense)	(None, 1)	17
Total parameters:		11,186,965

DOI: 10.7717/peerj-cs.422/table-2

Evaluation

Due to imbalanced data, indicators such as accuracy is not appropriate for this study. Because the developed model in the face of this type of data tends to ignore the minority class and can still be accurate. For this purpose, AUC and F-score indexes will be used, which are good choices for problems dealing with imbalanced data (Sokolova, Japkowicz & Szpakowicz, 2006). AUC indicates the area below the diagram in the ROC curve, and the ROC curve is a method for judging the performance of a two-class classifier (Luo et al., 2020). In the ROC curve, the vertical axis is the TPR (represents the true positive rate), Also, the horizontal axis is FPR (represents the false positive rate).

(7) $$FPR={\displaystyle \frac{fp}{tn+fp}}$$

(8) $$TPR={\displaystyle \frac{fp}{tp+fn}}$$

• – TP: positive samples are classified as positive

• – FN: positive samples are classified as negative

• – TN: negative samples are classified as negative

• – FP: negative samples are classified as positive

The F-score is the harmonic mean of precision and recall (Velupillai et al., 2009) and represents a weighted average of precision and recall (Gacesa, Barlow & Long, 2016). This index has wide applications in natural language processing (Derczynski, 2016), and like the AUC, it can be used in problems involved with imbalanced data.

(9) $$Precision={\displaystyle \frac{tp}{tp+fp}}$$

(10) $$Recall={\displaystyle \frac{tp}{tp+fn}}$$

(11) $$F-score=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

All the steps mentioned in the methodology section can be summarized in Fig. 4.

Dataset

The digital goods’ reviews of the Digikala website were extracted, which are a total of 2,932,747 reviews. Figure 5 shows the frequency of comments per category. Examining the comments of different product categories can increase the comprehensiveness of the model. To be specific, the words, phrases, and sentences are different in reviews of the products in the different categories, and considering various types of product categories will improve the generalization of the model in different situations. Table 3 shows the general structure of the collected dataset. In this table, the “Comment ID” column stores the unique values for each comment, the “Original Comment” column is the original of the comments written in Persian, the “Translated Comment” column is a translation of the “Original Comment” column into English. The “Translated Comment” column is used only to increase readability in the table and does not exist in the dataset. In the “Negative Feedback” column, if the value is 1, means that the user is not advised to buy the product, and in the “Positive Feedback” column, if the value is 1, it means the user is advised to buy the product, and the “Cat. Name” column represents the product category for which the comment was written.

The positive point of this website is that the buyers after submitting their comments can choose an option that states whether they generally recommend the product to others or not. Therefore, a significant number of extracted reviews are labeled. In other words, 308,122 of the reviews in the dataset do not recommend purchased items to others and the “Negative Feedback” column of these reviews in the dataset shows the number one. Likewise, 1,749,055 of the reviews in the dataset recommend the purchased items to others, and the “Positive Feedback” column of these comments in the dataset shows the number one. A significant part of the reviews is without labels and the reviews with labels are also imbalanced and these problems must be addressed in some ways.

Pre-processing

During the initial review of the comments, the first correction that had to be made was the removal of escape sequences. An escape sequence is a series of two or more characters starting with a backslash and when used in a string, are not displayed as they are written. In most reviews, there were some escape sequences such as “\n” and “\t” that needed to be removed. Also, sometimes users wrote some URLs to link to personal content that had to be removed. At this stage, all Persian numbers were converted to English, and letters that had different alternatives were standardized to normalize the text. Then all the phrases were tokenized by defining the word boundary and converting the half-space to space. In the stemming stage, prefixes and suffixes used were removed.

After the pre-processing steps, the number of words in the Negative Feedback class was 6.1 million and the number of words in the Positive Feedback class was 34.1 million. Using the word cloud diagram, the most repetitive words in each of the classes can be depicted. Figures 6 and 7 show the repetitive words in the Positive Feedback and Negative Feedback classes, respectively. Words like “I gave back”, “bad” and “I do not recommend” can be seen in the Negative Feedback figure, and words like “I’m satisfied”, “Appropriate” and “Speed” can be seen in the Positive Feedback figure.

Sentiment analysis

Data balancing is a crucial step that can increase accuracy. The random over-sampling method was used to balance the data. In other words, some data with the label of “Negative Feedback” were randomly selected and repeated. As a matter of fact, one of the common mistakes in this section is to apply the balancing method to the entire data which leads to errors in estimating the indicators. In these cases, the indicators are in a better position than the model capability and the results are reported incorrectly well. To avoid this, the balancing method was used only for the training data. After using the pseudo-labeling method, the number of positive feedbacks was about 1.8 million and the number of negative feedbacks was about 400 thousand. In this way, the negative feedbacks were repeated randomly about four times to balance the dataset.

The stratified K-fold cross-validation method is used to perform the evaluation. It is a method for model evaluation that determines how independent the results of statistical analysis on a dataset are from training data. In K-fold cross-validation, the dataset is subdivided into a K subset and each time one subset is used for validation and the other K-1 is used for training. This procedure is repeated K times and all data is used exactly once for validation. The average result of this K computing is selected as a final estimate. Stratified sampling is the process of dividing members of a dataset into similar subsets before sampling and this type of data sampling was selected due to imbalanced data. Using the stratified K-fold cross-validation method, we expect the values of the indicators to be more real. In all stages of measuring the accuracy of the model, K was considered equal to five.

As stated in the methodology, TF-IDF and fastText methods were used to extract the features. The BiLSTM and CNN models used the fastText output, and the Naïve Bayes and Logistics Regression models used the TF-IDF output, and their accuracy was finally compared with each other in Table 4. According to this table, the results of BiLSTM and CNN models are more accurate than others and CNN has given the best results.

As expected, due to the use of fastText and skip-gram methods, the need for data pre-processing has been reduced. In other words, stemming and normalization methods have not affected the final result. To examine this more closely, we chose the CNN model as the best model and we once performed the sentiment analysis process using the pre-processing steps and again without these steps. The AUC and F-score were 0.9943 and 0.9291 before pre-processing, and 0.9944 and 0.9288 after pre-processing. The results can be seen in Table 5. In the table, the meaning of the “before preprocessing” is just before the stemming and normalization steps. In other words, the methods used to create word embedding can depict the same words in the same range of spaces without the need to standardize letters and also without the need to identify the original root of words.

To implement pseudo-labeling, we developed a model that can estimate labels for unlabeled reviews using fastText and CNN models. After estimating all the labels, those with more than 90% probability for the Negative Feedback class and less than 1 × 10⁻⁷ for the Positive Feedback class were selected. Therefore, 104.8 thousand Negative Feedback reviews and 30.5 thousand Positive Feedback reviews were labeled and could be used in the dataset for subsequent analysis. In using the pseudo-labeling technique, most of our focus was on Negative Feedback as a minority class, which also leads to balance the dataset as much as possible. In this way, a significant amount of unlabeled data that had been excluded from the sentiment analysis process was re-entered into the process and helped to increase the accuracy and generalizability of the model.

Contrariwise of pre-processing, the use of the pseudo-labeling method significantly improved the results. After using pseudo-labeling, the values of AUC and F-score improved to 0.996 and 0.956. The values of the three mentioned states can be seen based on different folds in Table 5. Figure 8 also shows the ROC curve for all three states.

The suggested model has had better results than the previous models which have used pre-processing methods in Persian sentiment analysis. For instance, some researchers introduced pre-processing algorithms and succeed to enhance the results of machine learning algorithms (Saraee & Bagheri, 2013). In the research, the F-score of the proposed pre-processing algorithms employing Naïve Bayes as a classifier algorithm is 0.878. In another research, the various alternatives for pre-processing and classifier algorithms were examined and the best result was assisted with an SVM classifier by 0.915 F-score value (Asgarian, Kahani & Sharifi, 2018). Also, some researches were attempted to utilize state-of-the-art deep models in such a way to reduce dependency on pre-processing and avoiding complex steps (Roshanfekr, Khadivi & Rahmati, 2017). The F-score of the BiLSTM and CNN algorithms in the research is 0.532 and 0.534. All mentioned article's focus was on the digital goods reviews in Persian two-class sentiment analysis as same as this article. A comparison of the results in this paper with other researches and other common algorithms indicates that not only has the dependence on data pre-processing been eliminated but also the accuracy has increased significantly.

The result reveals that it is quite possible to create independent models from the pre-processing process using the method of fastText and skip-gram. Moreover, BiLSTM and CNN methods can have significant results. However, all of the mentioned methods need to have an immense dataset. To prove this, It is noteworthy that the use of the pseudo-labeling method because of increasing training data has a great impact on the result. Independency from pre-processing steps is not related to the Persian language. The results are reachable for other languages using sufficient labeled samples and the mentioned methods in this article.