Multi-modal affine fusion network for social media rumor detection

Introduction

As Internet technology gradually matures, online social networking (OSN) has become the habitat in which social relationships are formed. Since OSN information is open and easily accessible, social networking software such as Weibo, Twitter, and Facebook have become the primary sources for millions of global users to receive news and information. They serve as essential approaches for Internet users to express their opinions. However, the authenticity of published information cannot be detected without supervision. Such social networking software has become the source of public opinion in hot events and news media.

For example, during the tenure of Barack Obama as the US President, a tweet from the “so-called” Associated Press said, “Two explosions occurred in the White House, and US President Barack Obama was injured.” Three minutes after the tweet was sent, the US stock index plunged like a “roller coaster,” and the market value of the US stock market evaporated by 200 billion US dollars within a short period, which had a tremendous effect on both the stock and bond futures. Soon after, the Associated Press issued a statement saying that its Twitter account had been hacked, and that tweet proved to be false news. Therefore, it is of great necessity to automatically detect social media rumors in the early stage, and this technology will be extensively applied with the rapid development of social networks.

Nowadays, online rumors are no longer in the single form of texts. Instead, they are often in multiple modalities that combine images and texts. Figure 1 shows the cases of rumors in the Twitter dataset, displaying the texts and images of each tweet. In Fig. 1A, the news is fake based on the images and texts; it is hard to identify whether the news in Fig. 1B is true or not, but the images are fake; we cannot determine the authenticity of the news in Fig. 1C based on the images, but we can confirm that the information is not true according to the texts.

Currently, most methods used to detect social media rumors automatically are based on traditional machine learning (Tacchini et al., 2017; Dongo et al., 2020; Choi et al., 2020; Chou, Liu & Lee, 2021) and deep learning (Song et al., 2021; Jinshuo et al., 2020; Rani, Das & Bhardwaj, 2021; Gokhale et al., 2020). The neural network (Rauf, Bangyal & Lali, 2021), and other learning mechanisms such as federated learning (Gao et al., 2021) can learn the constantly changing high-dimensional feature representation of posts in the training process with the superior ability to extract features. The currently available research on rumor detection primarily focus on single modality (Jin, Wu & Guo, 2020; Abdulrahman & Baykara, 2020; Luo et al., 2021; Balpande et al., 2021), while multi-modal researches are still in infancy, and only a few recent researches have tried to explore the multiple modalities (Jin et al., 2017; Wang et al., 2018; Khattar et al., 2019; Jinshuo et al., 2020; Huang et al., 2019).

In current studies, the features of images and texts are mostly fused through feature concentration and averaging results. Nevertheless, this single fusion method fails to represent the posts fully. First, it cannot solve the problem caused by the difference in semantic correlation between texts and images in rumors and non-rumors; second, the semantic gap cannot be overcome. Moreover, unlike paragraphs or documents, the texts in posts that are usually short fail to provide enough context information, making our classification fuzzier and more random.

This paper introduces a new end-to-end framework to solve the above problems. This framework is known as the multi-modal affine fusion network (MAFN). In the proposed model, employing affine fusion, we fused the features of images and texts to reduce the semantic gap and better capture the semantic correlation between images and texts. Entity recognition was introduced to improve the semantic understanding of texts and enhance the ability of rumor detection models. MAFN can gain multi-modal knowledge representation by processing posts on social media to detect rumors effectively. This paper makes the following three contributions:

• We proposed the multi-modal affine fusion network (MAFN) combined with entity recognition for the first time better to capture the semantic correlation between images and texts.

• The proposed MAFN model enriched the semantic information of text with entity recognition, and entity recognition was fused with the extracted textual features to improve the semantic comprehension of text.

• Experiments show that the MAFN model proposed in this paper can effectively identify rumors on Weibo and Twitter datasets and is superior to currently available multi-modal rumor detection models.

Related work

In early research on rumor detection (Castillo, Mendoza & Poblete, 2011; Kwon et al., 2013), the rumor detection model was mainly established based on the differences between the features of rumors and factual information. Castillo, Mendoza & Poblete (2011) designed a simple model to evaluate the authenticity of information on Twitter by counting the frequency of words, punctuation marks, expressions, and hyperlinks in texts. On this basis, Kwon et al. (2013) used the communication structure to build rumors into a communication network and put forward 15 structural features, including the mid-values of network depth and width. Yang et al. (2012) introduced other client-based and location-based functions to identify rumors on Sina Weibo. However, it is time and energy-consuming to design these features manually, and the language patterns are highly dependent on specific time and knowledge in corresponding fields. Therefore, these features cannot be correctly understood.

Rumors on social media have gradually transformed from text-based to multi-modal rumors that combine both texts and images. Data in different modalities can complement each other. An increasing number of researchers have tried to integrate visual information into rumor detection. Singh, Ghosh & Sonagara (2021) manually designed textual, and image features in four dimensions, i.e., content, organization, emotions, and manipulation, and eventually fused multiple features to detect rumors. Jin et al. (2017) detected rumors by fusing the image and textural features of posts using the RNN combined with the attention mechanism. However, multi-modal features still depend highly on specific events in the dataset, which will weaken the model’s generalization ability. Therefore, Wang et al. (2018) put forward the EANN model that connected the visual features and textual features of posts in series and applied the event discriminator to remove specific features of events and learn the shared features of rumor events. Experiments show that this method can detect many events that are difficult to distinguish in a single modality.

Ma et al. (2016) introduced recurrent neural networks (RNN) to learn hidden representations from the texts of related posts and used LSTM, GRU, and 2-layer GRU to model text sequences, respectively. It was the first attempt to introduce a deep neural network into post-based rumor detection and achieve considerable performance on real datasets, verifying the effectiveness of deep learning-based rumor detection. Yu et al. (2017) used a convolutional neural network (CNN) to obtain critical features and their advanced interactions from the text content of related posts. Nonetheless, CNN is unable to capture long-distance features. Hence, Chen et al. (2019) applied an attention mechanism to the detection of network rumor and proposed a neural network model with deep attention. This model extracts adequate information and essential features from highly repeated texts, which solves the problems of excessive redundancy of texts in the data to be tested and weak information links between remote sites.

According to Khattar et al. (2019), a single fusion method cannot effectively represent the posts. So, they used the encoder and decoder to extract the features of images and texts and learned across modalities with the help of Gaussian distribution. Jinshuo et al. (2020) put the text vector, the text vector in the image, and the image vector together, and then processed them using Gaussian distribution to get a new fusion vector to discover the association between the two modalities of hidden representation. Besides learning the text representation of posts, Zhang et al. (2019) retrieved external knowledge to supplement the semantic representation of short posts and used conceptual knowledge as additional evidence to improve the performance of the rumor detection model.

Methodology

This paper introduced the four modules of the proposed MAFN model in this Section, i.e., the entity recognition enhanced textual feature extractor, the visual feature extractor, the multi-modal affine fuser, and the rumor classifier. Furthermore, we described the integration of the proposed modules to represent and detect rumors.

We instantiated tweets on Weibo and Twitter. The total tweets were expressed as $S=\{t_1,t_2,\dots ,t_n\}$, and each tweet was expressed as t = {T,E,V}, where T denotes the text content of the tweets, E represents the entity content extracted from the tweets, and V stands for the visual content matched with the tweets. $L=\{L_1,L_2,\dots ,L_m\}$ denotes the corresponding rumor and non-rumor tags of tweets. This paper aims to learn a multi-modal fusion classification model F by using the total tweets S and the corresponding tag sets L. F can predict rumors on unmarked social media. Figure 2 shows the framework of the proposed model.

The entity recognition enhanced textual feature extractor and obtained the joint representation R_u of text using Bert pre-training and self-attention mechanism. The visual feature extractor used the pre-trained model VGG19 to capture visual semantic feature R_v. The multi-modal affine fuser fused the joint representation and visual representation to obtain R_s, and the rumor classifier was utilized in the end to detect rumors.

Entity recognition enhanced textual feature extractor

Extraction of text representation

Text representation is a short text representation generated from tweets. Our model extracted the feature vector of tweets through the Bert model to better capture the context’s possible meaning and semantic meaning. Bert is a natural language processing model with the transformer bidirectional encoder representation as to the core, which can better extract the text context representation bidirectionally. By inputting the sequential vocabulary of the words in the tweets, the words were first embedded into the vector. The dimension of the ith word in the sentence is denoted by m, which is expressed as W_i∈ R^m, and by inputting it into the sentence, S, it can be expressed as:

(1) $$S=[W0,W1,W2,\dots ,Wp]$$

where, S∈ R^m*p, p denotes the total number of words, W0 denotes [CLS], and Wp represents [SEP]. By inputting the complete texts of tweets into the Bert model, we obtained the feature vector of the given sentence as

$$Sf=[Wf0,Wf1,Wf2,\dots ,Wfp]$$

Then the sentence feature vectors Sfn were given to the two fully connected layers. The above steps can be defined as follows:

(2) $$R{t}^{\mathrm{\prime }}=\sigma (Wft2\cdot \sigma (Wft1\cdot Sf+bt1)+bt2)$$

where W ft1 denotes the weight matrix of the first fully connected layer with activation function, W ft2 represents the weight matrix of the second fully connected layer with activation function, and bt1 and bt2 are the bias terms.

The attention-based neural network can better obtain relatively long dependencies in sentences. The self-attention mechanism is a kind of attention mechanism that associates different positions of a single sequence to calculate the representation of the same sequence. To enable the model to learn the correlation between the current word and the other parts of the sentence, we added the self-attention mechanism after the fully connected layer, the process of which was expressed as follows:

(3) $$Attself=softmax[QT\cdot K{T}^{\mathrm{\top }}/\sqrt{}m]\cdot VT$$

where, Q_T = R_t′ × W_QT, K_T = R_t′ × W_KT, V_T = R_t′ × W_VT · W_QT, W_KT, W_VT denote the three different matrices learned by Q, K, and V, respectively. To make the model automatically recognize the importance of each word, degrade unimportant features to their original features, and process essential features using the self-attention mechanism, we used the residual connection to extract the features better. Figure 3 shows the architecture of a residual self-attention. A building block was defined as:

(4) $$Rt=Attself+R{t}^{\mathrm{\prime }}$$

where, R_t denotes the eventually extracted text representation, R_t ∈ R^k.

Extraction of entity representation

Named entity recognition identifies person names, place names, and organization names in a corpus. It was assumed that the combination of entity tagging and text coding in a post could supplement the semantic representation of the short text of the post in a certain way so that the model could identify rumors and non-rumors more accurately. Explosion AI developed spacy, a team of computer scientists and computational linguists in Berlin, and its named entity recognition model was pre-trained on OntoNotes 5, a sizeable authoritative corpus. In this paper, Spacy was applied to train the two datasets and extract the entities of posts. There were 18 kinds of identifiable entities.

First of all, we identified the recognizable word W_i as the entity e ∈ E_s in every sentence $S=[W0,W1,W2,\dots ,Wp]$ of the tweet, and then obtained the tag $L\in \{L_1,L_2,\dots ,L_n\}$ corresponding to this entity, where L_i is one of the tags {PERSON, LANGUAGE,…,LOC}. For instance, to instantiate a piece of text, we instantiated the entities in the text, as shown in Fig. 4. The extracted entity L_European = {NORP}, NORP means nationalities or religions or political groups; L_Google = {ORG}, OPG represents companies, agencies, institutions etc.

Based on the obtained L_i, the corresponding entity tags were connected in series to capture semantic features by Bert. E_f ∈ R^k, where R^k denotes the embedding dimension of tags. By inputting E_f into the residual attention mechanism, we gained Re ∈ R^k.

In the end, we combined the extracted text representation with the entity representation to obtain the joint representation Ru, Ru ∈ R^k, which was defined as follows:

(5) $$Ru=addRe,Rt$$

Visual feature extractor

Images in tweets form the input into the visual feature extractor. This proposed framework used the pre-trained model VGG-19 and added two fully connected layers in the last layer to more comprehensively extract the visual features matched with the rumors in the tweet. According to the parameters unchanged after pre-training, VGG-19 adjusted the representation dimension of final visual features to k through two fully connected layers. We added the batch normalization layer and drop-out layer between the two fully connected layers and the activation function to prevent overfitting during the extraction of image representation. The eventually obtained feature of visual representation was expressed as Rv, where Rv ∈ R^k. The equation for extracting image features was defined as follows:

(6) $$R{v}^{\mathrm{\prime }}=Wfv2\cdot \sigma (BN(Wfv1\cdot Rvgg+bv1))+bv2$$

(7) $$Rv=Dropout(\sigma (BN(R{v}^{\mathrm{\prime }})))$$

where, Rvgg represents the visual features extracted from the network in the pre-trained model VGG19, σ is the activation function, W fv1 denotes the weight matrix of the first fully connected layer with the activation function, and bv1 and bv2 are the bias terms.

Multi-modal affine fuser

Affine transformation transforms into another vector space via linear transformation and translation. Through affine transformation, the multi-modal affine fuser fuses the multi-modal features extracted by the entity recognition enhanced textual feature extractor and the visual feature extractor, the joint representation and visual features of text and entity. It was assumed that the data of the two modalities could be fused more closely and the high-level semantic correlation could be better extracted. The corresponding equation was defined as follows:

(8) $$R_c=\mathcal{F}{R}_v\cdot R_u+\mathcal{H}(R_v)$$

where, Rc is the feature R_c ∈ R^k gained after the fusion of all features, and $\mathcal{F}\cdot$ and $\mathcal{H}\cdot$ were fitted by the neural network. After extracting the fused features, in order to get more robust features, we reconnected the fused features with the textual features to obtain the total feature R_s. The equation was expressed as:

(9) $$R_s=R_c\oplus R_t$$

where, ⊕ denotes concatenation.

Rumor detector

The rumor detector, based on the multi-modal affine fuser, sent the finally obtained multi-modal feature R_s to the multilayer perceptron for classification to judge whether the message was a rumor or not. The rumor detector consists of multiple completely connected layers with softmax. The rumor detector was expressed as G(Rⁱ_s, θ), where θ represents all the parameters in the rumor detector, and Rⁱ_s denotes the multi-modal representation of the case of the ith tweet. The rumor detector was defined as follows:

(10) $$p_i=G(R_s^i,\theta )$$

where p_i denotes the probability that the ith post input by the detector is a rumor, in the process of model training, we selected the cross-entropy function as the loss function, which was expressed as follows:

(11) $$Loss=\sum _{i=1}^N-[L_i\times log\left(p_i\right)+(1-L_i)\times log(1-p_i)]$$

where, L_i denotes the tag of the tweet in the i-th group, and N refers to the total number of training samples.

Experiments

This section first described the datasets used in the experiment, namely two social media datasets extracted from the real world. Secondly, we briefly compared the results obtained by the most advanced rumor detection method and those gained by the model proposed in this paper. Through the MAFN ablation experiment, we compared the performances of different models.

Case study performance visualization

A qualitative analysis was performed on MAFN. After analyzing and ranking the examples of rumors successfully classified by MAFN, we selected the best two examples on Twitter and Weibo and showed them in Figs. 5 and 6, respectively. Without the support of affine fusion and entity recognition, the examples in Twitter could not be detected. Since the model failed to effectively capture the relationship between texts and images, these examples were misjudged as non-rumors. Insufficient text information and the absence of close connections between information and images are the reasons why the examples in Weibo could not be detected using “w/o entity+ affine fusion.” However, we can identify rumors with affine fusion by judging the image features.

Conclusion

This paper proposed an affine fusion network combined with entity recognition. This network accurately identifies rumors using the affine fusion between the entity recognition joint representation of images and texts. When extracting text representation, we used Bert to generate sentence vector features and learn semantics by extracting knowledge from the outside through entity recognition. Moreover, affine fusion was used for multi-modal fusion to better summarize the invariant features of new events. The Twitter and Weibo datasets experiments show that the proposed model is robust and performs better than the most advanced baselines. In the future, we plan to capture and identify rumor propagation in the field of rumor text and short videos to strengthen the generalization ability of the multi-modal fusion model.