Stock price movement prediction based on Stocktwits investor sentiment using FinBERT and ensemble SVM

Sentiment analysis techniques

Sentiment analysis is a subfield of natural language processing (NLP) that can deduce the sentiment or opinion of a phrase or text and classify it as positive, negative, neutral, etc., (Sidogi, Mbuvha & Marwala, 2021). One of the sources for obtaining comments from people is social media, and in recent years, there have been many studies that have used insights from social media comments to solve various problems. Shamoi et al. (2022) used Twitter posts to understand the public’s perception of plant-based diets and employed sentiment analysis techniques to determine if the sentiment in the text was positive or negative. These findings can help medical professionals develop appropriate health plans and encourage more people to maintain healthy plant-based diets. Ko & Chang (2021) conducted sentiment analysis on posts from PPT platform and used long short-term memory (LSTM) model to predict stock prices. However, due to the widespread nature of social media, there is the potential for personal attacks or discrimination based on race and gender to occur on social media. Raman et al. (2022) explained that although many companies have implemented various algorithms on their platforms, it is still difficult to regularly detect such comments, and they continue to appear on the platforms, causing negative impacts on target populations. Therefore, the study compared different hate and attack detection algorithms to detect speech with hateful sentiment. Therefore, the application of sentiment analysis is widely applicable and carries significant commercial value.

Sentiment-based stock price movement prediction using support vector machine

Support vector machine (SVM) is a technique for supervised learning that was introduced by Cortes & Vapnik (1995) and may be used to address problems related to both regression and classification. SVM may be utilized to achieve high performance in classification tasks, provided that it is equipped with the kernel function. When the data in the initial space cannot be separated by linear classifier, using the kernel function to perform nonlinear projection to separate the data in a higher dimensional space is what the objective of employing the kernel function is. Due to the fact that the stock market is a classic example of a non-linear, dynamic system, SVM is often used in predicting stock prices. Even though SVM does well in solving classification problems, it still has some drawbacks. According to Auria & Moro (2008), one drawback of SVM is the lack of transparency of results, which makes it difficult to get the classified score and probability. This limitation may hinder our ability to make flexible decisions, particularly when buying or selling based on the probability of a classification result.

According to the findings of Hu, Zhu & Tse (2013), SVM has good generalization capacity as well as strong resistance to overfitting difficulties. In contrast, training a SVM model is analogous to solving a linearly constrained quadratic programming problem, and SVM classification always produces a unique global optimal solution. This is in contrast to training a neural network, which requires nonlinear optimization and runs the risk of becoming mired in a local minimum solution. Additionally, the study forecasts stock prices using a combination of annual reports and macroeconomic factors. The final findings demonstrate that SVM is a useful predictive technique for use in stock forecasting.

In a study proposed by Ren, Wu & Liu (2019), it was believed that sentiment features may contain valuable information about the basic value of assets and can be regarded as one of the leading indicators of the stock market. They also mentioned that SVM can solve nonlinear problems by transforming them into quadratic programming and has a unique and globally optimal solution, making it a useful tool for predicting the price movement of the stock market. The researchers implemented fivefold cross validation using SVM, but found that doing so caused look-ahead bias. To eliminate this bias, they integrated SVM with the realistic rolling window approach. In order to avoid overfitting in our research, the rolling window method will be used to verify its effectiveness.

Using Stocktwits comments to make sentiment-based stock price movement predictions

In a study conducted by Koukaras, Nousi & Tjortjis (2022), comments on stocks were collected from people on Twitter and Stocktwits, and stock price movements were predicted through sentiment analysis and machine learning. The study used VADER as a sentiment analyzer and predicted the movement of Microsoft’s (MSFT) stock price. Various machine learning models were tried for predicting stock price movements, and the final results showed that SVM outperformed the other classification models. This study used sentiment analysis in social media for stock forecasting in the US market, achieving good results and inspiring the researchers to use SVM to solve the problem of sentiment-based stock price movement prediction in their own study.

Gupta & Chen (2020) collected comment data from Stocktwits between January 1st, 2019 and September 30th, 2019. They developed several sentiment analyzers for different stocks using techniques such as logistic regression and TF-IDF. Then, they experimented with several US stocks for stock price movement prediction and found that adding more sentiment data helped improve the accuracy of predicting stock price movement.

Furthermore, in recent years, many studies have developed two-class sentiment analysis models using comments on Stocktwits. This is beneficial for our research and comparison. Lee, Gao & Tsai (2020) fine-tuned the pre-trained BERT on a labeled sentiment dataset and achieved an accuracy of over 87.3% in recognizing the sentiment of investors. Bozanta et al. (2021) conducted multiple experiments and ultimately used RoBERTa (Liu et al., 2019) to achieve an accuracy of 83–88%. Cao et al. (2022) fine-tuned the roberta-base model on 3.2 million comments from Stocktwits, with user-labeled tags of bullish or bearish, and achieved a validation accuracy of 93.43%. Therefore, our study uses the model proposed by Cao et al. (2022) for two-class sentiment analysis to predict stock price movements and compares it with the method proposed in this article.

FinBERT: financial sentiment analysis with BERT

FinBERT (Araci, 2019) is the first application of BERT (Devlin et al., 2018) in the financial domain, which is a pre-trained sentiment analysis model specialized in analyzing financial texts. FinBERT first pre-trains the BERT language model using a large financial dataset and then fine-tunes it on a specific financial dataset (https://huggingface.co/ProsusAI/finbert). The SOTA (state-of-the-art) level is achieved on the Financial Phrasebank dataset. The dataset consists of 4,840 sentences from English language financial news categorized by sentiment (Malo et al., 2013), and the accuracy is 15% higher than the current state-of-the-art model. The model can produce three classifications of sentiment, including positive, negative, and neutral, enabling us to detect neutral sentiment and compute the sentiment index more precisely, thereby enhancing the accuracy of the stock price movement prediction. FinBERT for sentiment analysis often analyzes financial news and articles. FinBERT performs sentiment classification by adding a dense layer after the last hidden state of the [CLS] token, which is the recommended method for using BERT for any classification task. The classifier network is then trained on a labeled sentiment dataset. The procedure is summarized in Fig. 2.

Fazlija & Harder (2022), used the FinBERT model to classify news, forecast the trend of stock prices with the random forest classifier, and measure the investment performance of the models. The performance of the proposed method in the study outperformed the S&P 500 index. However, although this article uses K-fold cross-validation to find the best model parameters, splits the training and test data sets into multiple copies, and trains multiple models for validation, the authors did not mention the use of the rolling window approach in their study. Therefore, in our research, we intend to apply FinBERT to analyze comments on social media, allowing us to gather more opinions about the target asset. Then, by making predictions using the rolling window approach, we can not only avoid the problem of look-ahead bias, but also update the training data over time, which can lead to more accurate predictions.

In this study, we propose using FinBERT to perform sentiment analysis on investors’ comments on Stocktwits in order to predict stock price movements. Few studies have used FinBERT for sentiment analysis on social media. However, we believe that applying it specifically to analyze Stocktwits comments, rather than using a two-class sentiment analysis model, will improve the macro-level sentiment index and enhance the accuracy of forecasting stock price movements.

Data collection

On Stocktwits, users may share a message and label it as “bullish” or “bearish”, or they might ignore these sentiments. There are three data columns: message, date/time, and sentiment (optional). The Stocktwits tweets are shown in Fig. 3.

Accessing https://api.Stocktwits.com/ allows us to examine JSON-formatted investor comments on a certain stock from Stocktwits. To get data from Stocktwits, we developed a Node.js app that automatically downloads and saves CSV files with all comments on a specific stock over a certain period. Then, we brought the data into the Python environment to perform more analysis.

The historical stock data is collected from cnYES (https://www.cnyes.com/), and the data columns include stock opening price, closing price, high price, low price, VWAP (volume-weighted average price), volume, trade value, change, and percentage change, as shown in Table 1.

We used the AlphaVantage API (https://www.alphavantage.co/) to obtain historical real-time stock prices. This enables us to acquire a minute-by-minute view of stock prices, allowing us to make more flexible forecasts. However, the data is restricted to the last two years. Table 2 shows historical stock price data in 15-minute increments.

Sentiment data preprocessing

We refer to Cao et al. (2022). Before conducting sentiment analysis, the following text preprocessing steps should be performed:

1. Remove all web links from the text, including those that start with “http://”, “https://”, or “www.”.

2. Replace the HTML character for single quotation marks with standard single quotation marks.

3. Identify and replace hashtags (#) and cashtags ($) in the text. Find all words that start with ”#” or ”$”, such as $AAPL, and substitute them with terms like “cashtag_AAPL”. This change is made to preserve the original symbol information while making it more suitable for text processing.

4. Replace Twitter usernames with a term that begins with “mention_”, followed by the username. This modification is also intended to preserve the original information while rendering it more suitable for text processing.

5. Convert all emojis into their corresponding text descriptions to maintain the emotional or semantic information they convey, but in a format that is easier for the model to understand.

Stock price movement prediction

The architecture depicted in Fig. 4 was used to predict stock movements. We collected the necessary data, which was divided into two parts: the first part consists of comments about specific stocks on Stocktwits, some of which are not labeled as bullish or bearish, while the second part consists of historical data on stock prices. We can choose to use a two-class sentiment analysis model to label the unlabeled comments or use a three-class sentiment analysis model like FinBERT or VADER to label all comments. Finally, we applied the sentiment index formula to calculate the entity index of comments from multiple periods. To predict stock price movements, we combined the sentiment indexes with historical stock data as features and input them into a prediction model designed for stock price movements. We used the rolling window approach to predict and analyze the accuracy of stock movement predictions.

Sentiment index calculation

The two-class sentiment index calculation used in this article is related to Antweiler & Frank (2004) methods for calculating sentiment indexes. ${\displaystyle S_t=ln\left[\frac{1+M_t^{pos}}{1+M_t^{neg}}\right].}$

This index enables us to compute the sentiment for a period, where ${\mathit{M}}_{\mathit{t}}^{\mathit{pos}}$, ${\mathit{M}}_{\mathit{t}}^{\mathit{neg}}$ are the number of positive and negative messages during that period. If the sentiment index is positive, market sentiment is optimistic. If the sentiment index is negative, market sentiment is pessimistic. If the sentiment index is zero, market sentiment is neutral. This index helps us to determine investor sentiment over a certain time period and also adjusts for overly optimistic or pessimistic signals during calculation.

For three-class sentiment analysis models (including positive, negative and neutral sentiments), refer to the equation used to calculate the sentiment index in Hiew et al. (2019). ${\displaystyle S_t=\frac{M_t^{pos}-M_t^{neg}}{M_t^{pos}+M_t^{neu}+M_t^{neg}}}$

where ${\mathit{M}}_{\mathit{t}}^{\mathit{pos}}$, ${\mathit{M}}_{\mathit{t}}^{\mathit{neg}}$, ${\mathit{M}}_{\mathit{t}}^{\mathit{neu}}$ are the number of positive, negative and neutral messages within the period.

Support vector machine

Support Vector Machine (SVM) is a supervised machine learning algorithm developed by Cortes & Vapnik (1995), which can be used to solve classification and regression problems. The concept is simple: find a decision boundary that maximizes the margin between two classes, as shown in Fig. 5.

The equation for the distance from x, y to Ax + By + C = 0 is given by ${\displaystyle \frac{Ax+By+C}{\sqrt{A^2+B^2}}.}$

Extending it to the n-dimensional space, we express it as w^Tx + b = 0, and the distance as ${\displaystyle \frac{\left(w^{T}x+b\right)}{\left|\left|w\right|\right|}}$

where w denotes the weight of each feature, b denotes intercept.

If we disregard the issue of mixed data, i.e., if all data can be perfectly classified, we call this hard-margin SVM and divide the two classes according to the equation below. ${\displaystyle \left\{\begin{array}{c}{w}^T{x}^{\left(i\right)}+b\ge 1,\forall y^{\left(i\right)}=1\hfill \\ w^T{x}^{\left(i\right)}+b\le -1,\forall y^{\left(i\right)}=-1\hfill \end{array}.\right.}$

And simplify the formula as follows ${\displaystyle y^{\left(i\right)}\left(w^T{x}^{\left(i\right)}+b\right)\ge 1,\forall i=1,\dots ,n.}$

The solution to SVM is a simple optimization problem. Under some conditions, the larger the margin distance between the two classes, the better. The formula is as follows ${\displaystyle min\frac{1}{2}\left|\right|w|{|}^2}$ ${\displaystyle s.t.y_i\left(w^T{x}_i+b\right)\ge 0,\forall i=1,\dots ,n.}$

However, it is rare that real-world data would include examples that can be precisely identified. Therefore, we allow certain points to transcend the initial decision limit and enable some unusual samples to be wrongly classified. Although this may result in incorrect categorization, the generalization capacity may be superior in practical applications. This kind of SVM is known as soft-margin SVM, using the following formula

$s.t.y_i\left(w^T{x}_i+b\right)\ge 1-{\xi }_i,{\xi }_i\ge 0,\forall$i =1 , …, n

where ξi is a tolerable training error, and C determines how much penalty weight is to be given.

From the above equation, we can see that if C is larger, the space for error tolerance is smaller, and if C is positive infinity, it will become a hard-margin SVM. Conversely, if C is smaller, the space for error tolerance is larger.

To solve the optimal solution of the primal problem, we can transform it into its dual problem, as shown in the following equation ${\displaystyle \underset{\alpha }{max}\sum _{i=1}^n{\alpha }_i-\frac{1}{2}\sum _{i=1}^n\sum _{j=1}^n{\alpha }_i{\alpha }_j{y}_i{y}_{j}K\left(x_i,x_j\right)}$

s.t. ${\sum }_{i=1}^n{\alpha }_i{y}_i=0$ ${\displaystyle 0\le {\alpha }_i\le C,\forall i=1,\dots ,n}$

where $\alpha ={\left({\alpha }_1,\dots ,{\alpha }_{\mathit{n}}\right)}^{\mathit{T}}$ is a Lagrange multiplier, and $\mathit{K}\left({\mathit{x}}_{\mathit{i}},{\mathit{x}}_{\mathit{j}}\right)$ signifies that we employ a kernel function to redefine the inner product of x_i and x_j. There are several common kernel functions shown in Table 3.

Finally, the decision function $\mathit{D}\left(\mathit{x}\right)=\mathit{sgn}\left({\mathit{w}}^{\ast }\cdot \mathit{x}+{\mathit{b}}^{\ast }\right)$ is given by

${\displaystyle D\left(x\right)=sgn\left(\sum _{i=1}^n{y}_i{\alpha }_i\ast K\left(x_i,x_j\right)+b^{\ast }\right)}$ ${\displaystyle b^{\ast }=y_j-\sum _{i=1}^n{y}_i{\alpha }_i\ast K\left(x_i,x_j\right).}$

Ensemble-based technique

According to Mohareb et al. (2016), the ensemble-based technique is based on the development of several classifiers, with the final prediction output being a composite of all classification results in the integration. Suppose we use n classifiers ɛ  = { C₁, …, C_n} for predicting a sample, if the result of C₁ is wrong, but the results of C₂, …, Cn are correct, then the result can be classified correctly by majority voting.

Individual classifiers must have good accuracy for this method to be applicable. If there are a total of T classifiers for a c-class problem, the ensemble choice will be correct if at least [T/c + 1] classifiers select the correct class. Assume that each classifier of ɛ has a probability p of selecting the correct class. The ensemble’s chance of picking the right class has a binomial distribution, and the probability of picking k > T/c + 1 correct classifiers out of T is calculated as follows: (Mohareb et al., 2016). ${\displaystyle {\mathit{p}}_{\varepsilon }=\sum _{\mathit{k}=\left(\frac{\mathit{T}}{\mathit{C}}\right)+1}^{\mathit{T}}\left(\frac{\mathit{T}}{\mathit{k}}\right){\mathit{p}}^{\mathit{k}}{\left(1-\mathit{p}\right)}^{\left(\mathit{T}-\mathit{k}\right)}.}$

SVM ensemble with bagging

According to Kim et al. (2002) in bagging, several SVMs are individually trained using the bootstrap approach and then aggregated using a suitable combination technique. The training set needs to be split into L subsets to construct the SVM ensemble with L independent SVMs. Based on statistical evidence, it is necessary to make the training sets as diverse as feasible to get a greater improvement in the aggregation result. We employ the bootstrap method to do this.

The bootstrap method was proposed by Efron (1979). In simple terms, each model is trained by randomly picking n sample data from the training set, and each sample is placed back after picking, as shown in Fig. 6. We will utilize models with trained subsets to forecast the test data, and the final prediction will be generated using these models through a majority vote, as explained below.

Majority vote is the simplest method for combining classifiers. The results of many classifiers are mixed by majority vote. The output with the most votes is then selected as the final classification decision (Kim & Upneja, 2021).

Data preparation

The SPDR S&P 500 Index ETF (SPY) was used in this study because it is currently the most popular S&P 500 ETF (Exchange Traded Fund) and one of the stocks with the most Stocktwits discussions. As shown in Fig. 7, this is the most active rank at Stocktwits on September 7, 2022. As we can see, SPY has significantly more discussions than any other stock, and based on our observations, SPY has been in the top 3 for an extended period of time.

We collect all SPY comments and historical stock data on Stocktwits from April 1, 2020, to February 28, 2022, a total of 482 trading days, of which approximately 1.33 million Stocktwits messages were labeled and 1.73 million messages were not labeled by users, for a grand total of 3.06 million messages. The training set starts on April 1, 2020, and the period from April 05, 2021, to February 28, 2022 is used as test data.

Feature extraction

We train the model using a total of 10 features, as shown in Table 4, including three sentiment features and seven derived from the historical stock data. We have found that considering both the pre-market sentiment and the after-market sentiment from the day before may significantly increase the accuracy of our stock price movement forecast. This is due to the fact that the stock market places a significant emphasis on these two time periods.

The calculation formulas for features S1 to S3 are shown in the subsection named ‘Sentiment Index Calculation’ of previous section. To obtain feature values, all comments about a certain stock on Stocktwits during the specified period must be obtained and processed through a sentiment analysis model. Then, the formulas for calculating the sentiment index can be used to compute these feature values. Features S4 to S10 are historical stock data and do not require calculation.

The features we have selected are accessible prior to the opening of the forecast day, which allows us to obtain sentiment and make forecasts without overlap. This means that the forecast results will be available to investors at the start of the market, enabling informed trading decisions.

Model building

Since we are forecasting a moving price trend (either upward or downward) rather than the actual price of the SPDR S&P 500 Index ETF, we have to convert a regression problem into a categorical one. This is because we are focusing on moving price trends rather than actual prices. It is necessary for us to label the data based on the equation that is shown below. ${\displaystyle y=\left\{\begin{array}{c}1,Priceat12:00EST\ge OpeningPrice\hfill \\ 0,Priceat12:00EST<OpeningPrice\hfill \end{array}.\right.}$

We used a support vector machine (SVM) ensemble with bagging to build the model. This required setting the parameters for both the support vector classifier (SVC) and the bagging classifier. The SVC has several important hyperparameters, such as the kernel function, C (the regularization parameter), and gamma (the kernel coefficient). These parameters are shown in detail in Table 5. We also used the BaggingClassifier from the scikit-learn python library (Pedregosa et al., 2011). The bagging classifier has several important hyperparameters that must be set, such as the base estimator, the number of estimators (n_estimators), and whether samples are drawn with replacement (bootstrap). These parameters are shown in detail in Table 6.

Evaluation metrics

The equation for the classification accuracy is as follows, which can represent the percentage of the total number of correct predicted results. ${\displaystyle Accuracy=\frac{TP+TN}{TP+FP+FN+TN}}$

where TP denotes that the predicted value is 1 and the true value is also 1; TN denotes that the predicted value is 0 and the true value is also 0; FP denotes that the predicted value is 1 but the true value is 0; and FN denotes that the predicted value is 0 but the true value is 1. ${\displaystyle Precision=\frac{TP}{TP+FP}}$

where precision is the percentage of times out of a sample of predicted stock price increases that the actual price went up. ${\displaystyle Recall=\frac{TP}{TP+FN}}$

where recall is the percentage of times out of a sample of actual stock price increases that the predicted price went up.

In predicting whether a stock price will rise, precision is considered more important than accuracy. In a bull market, both precision and recall are equally important for decision-making. However, in a bear market, recall becomes more crucial than precision as it has a greater impact on potential financial loss (http://www.bituzi.com/2019/10/confusionmatrix.html). ${\displaystyle F1Score=\frac{2Precision\ast Recall}{Precision+Recall}.}$

The F1-score above is the harmonic mean of precision and recall. A score of 1 represents the highest performance, and a score of 0 represents the worst. Both precision and recall contribute equally to the F1-score; hence, the closer the score is to 1, the better the model (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html). Therefore, for this topic, we evaluate the methods using the F1-score.