Investigating the occurrence of selfie-based emotional contagion over social network

Abstract

Happiness is obviously one of the most fundamental essence that affects many aspects of our lives. Past research found that happiness of one person affects that of other people. What occurs under this propagation of emotion is called “emotional contagion,” a phenomenon wherein through perception, people experience the same emotion expressed by someone when communicating with them. Although online communication is increasing due to growth of mobile computing, emotional contagion on online communication is not well studied yet. Particularly, it is not yet clear if emotional contagion among people occurs through selfie photographs posted on the social network media. We implemented “SmileWave,” the social networking system for investigating selfie-based emotional contagion. The key feature of SmileWave is detecting “smile degree” in user’s posting selfies and in reactive facial expressions when the user is viewing the posted photographs from others. Our in-the-wild user studies with 38 participants for 2 weeks revealed the occurrence of selfie-based emotional contagion over the social network, based on the results that the users’ smile degree improved (15% on average) when the user looked at posted selfie photographs.

1 Introduction

Fig. 1

Two-stage process of emotional contagion

Emotion plays an important role in our lives, and research on emotions has been actively conducted. Inspired by positive psychology (Seligman 2002), an increasing number of studies have focused on the role of positive emotion (Seligman and Csikszentmihalyi 2014). Particularly, happiness is known to affect various phenomena, such as user’s income (Easterlin 2003), disease (Ubel et al. 2003), divorce (Easterlin 2003), and voting (Wilson et al. 2003). Furthermore, happiness of one person is found to affect that of other people (Hatfield et al. 1993). For example, we actually have experiences such that we become happier when we see people smiling happily and that we feel dark and gloom when we hear the story of people undergoing the hardships. A specific mechanism occurring in such emotional propagation is called “emotional contagion.”

Emotional contagion is defined as “the tendency to automatically mimic and synchronize expressions, vocalizations, postures, and movements with those of another person and, consequently, to converge emotional” (Hatfield et al. 1993). One widely known argument on emotional contagion is that emotional contagion among people is composed of a two-stage process. The first stage is described as “people automatically and continuously mimic and synchronize their movements with the facial expression,” and the second stage is described as “emotional experience is affected, moment to moment, by the activation of and feedback from facial mimicry” (Hatfield et al. 1993). Figure 1 illustrates the example situation of a two-stage process of emotional contagion. Alice and Bob are in conversation and Alice with smiley face talks to Bob about the good things that happened today. The first stage of the process is interlocking of expression between people. For example, looking at Alice’s smiling expression, Bob’s facial expression changes into a smile. The second stage is that this change in facial expression contributes to the person’s emotional state. The physical change in Bob’s facial expression (to smile) affects his internal emotional state, such as his degree of happiness.

Meanwhile, on observing our communication with others when the first stage of emotional contagion occurs, online communication such as social networks and messaging is increasing compared to offline communication. This is driven by the popularity of smartphones among all generations, now that more than half our access to the online media is from mobile devices. In spite of such recent distinctive trends in communication, however, emotional contagion through our online communication has not been well investigated yet. More specifically, the occurrence of the “first stage” on online communication is not well studied, while various studies already confirmed the existence of the second stage (Tsujita and Rekimoto 2011; Kleinke et al. 1998).

In this research, we intend to reveal which emotional contagion occurs on mobile online social media, particularly focusing on selfie photographs in which a user is smiling as communication media to propagate happiness among people. Since the second stage of emotional contagion does not depend on the difference in communication media, confirming the occurrence of the first stage is our key focus in this research toward the confirmation of overall emotional contagion on the selfie-based social network.

Our extensive in-the-wild user study, 38 participants for 2 weeks, w conducted by building the original social network system “SmileWave” in which users share selfie photographs among friends. It can analyze the changes in a user’s smile degree on looking at posted selfies. The smile degree is a value that represents how much the user smiles. In the user study, user’s smile degree improved by average 15% when the user looked at posted selfie photographs. We also found that there is a higher effect on selfie-based emotional contagion when the user looks at selfie photographs with a higher smile degree. Further, presenting a series of selfies ordered from lower degree to higher degree resulted in a stronger effect than other types of orderings.

The contribution of our paper is the following fourfold.

• 1. We present “SmileWave,” the first system for revealing the occurrence of selfie-based emotional contagion on social network and conduct in-the-wild user study with 38 participants for 2 weeks.

• 2. We confirm that the first stage of selfie-based emotional contagion occurs in social network since the users’ smile degree improved 15% on average when the user browses posted selfie photographs.

• 3. We find a slight tendency that there is a higher effect on selfie-based emotional contagion when the users view selfie photographs with higher smile degree.

• 4. We show that there is an aspect that leads to a stronger effect of emotional contagion regarding the order of multiple photographs (in the order from lower smile degree to higher degree).

The rest of the paper is organized as follows: Sect. 2 describes what is emotional contagion and the focusing point in our research. Section 3 overviews SmileWave including its design, system architecture, and preliminary experiment to decide the parameters inside the system. Section 4 provides the details of our in-the-wild evaluation of SmileWave. We discuss the evaluation results in Sect. 5. Finally, Sect. 6 concludes this paper.

2 Emotional contagion

In this section, we firstly overview the concept of emotional contagion and review related study in several different research areas.

2.1 Offline emotional contagion

Emotional contagion is a phenomenon that people experience the same emotion by “catching” other’s expressed emotion during their communication (Hatfield et al. 1993) as described in Sect. 1. It has been studied from various perspectives, such as sociology (Fowler and Christakis 2008) and physiology (Hatfield et al. 1993). In the early research, emotional contagion was considered to occur only through limited types of human behavior, such as gesture, pose, and facial expression (Condon and Ogston 1966). However, it is now concluded that emotional contagion is a more primitive and universal nature of human beings and it can occur even through one’s existence at a venue (Hatfield et al. 1993; Hoffman 1994).

Emotional contagion is composed of a two-stage process (Hatfield et al. 1993). In the first stage, expressions and behaviors among multiple persons synchronize mutually over time. Next, in the second stage, feedback of the own movement mimicry contributes to the expressed emotion of the affected user. This second stage is supported by facial feedback theory (Buck 1980). It explains a direct link between a facial part associated with a facial expression and particular emotion. For example, a linkage between pulling up a muscle at the edge of the mouth and feeling happy is found (Buck 1980). Then, while making a specific facial expression, feedback from the associated muscle flows into the cerebral cortex, and this makes us experience a particular emotion.

Fig. 2

Overview of the SmileWave system

Emotional contagion is the process which is relatively automatic, unintentional, uncontrollable, and largely inaccessible to conversant awareness (Hatfield et al. 2014). Additionally, the mimicry of facial expressions and the subsequent emotional contagion appear to be culturally universal. Facial mimicry and emotional contagion have been observed in Americans, Finns, Dutch, Western Europeans, Japanese participants, and participants in many other countries (Stel and Van Knippenberg 2008; Surakka and Hietanen 1998; Tamura and Kameda 2006). Also, many researchers have found that there is no gender difference in susceptibility to mimicry and feedback (La France and Ickes 1981; Lundqvist and Dimberg 1995).

Emotional contagion is confirmed to occur in real-world social network. Based on the survey data collected over 20 years, Flow suggested that longer-lasting moods can be transferred through social networks (Fowler and Christakis 2008). Worline mentioned that organizations have emotional ability and suggested that emotional contagion occurs in organizations (Schrock et al. 2004). Further, a study revealed that a leader in the organization is more strongly contagious on emotion than other person (Sy et al. 2005). McColl-Kennedy argued that a high degree of emotional contagion is observed in the interaction between the server and customer (McColl-Kennedy and Smith 2006). Therefore, emotional contagion is known to occur ubiquitously in face-to-face offline communication.

In this research, we focus on emotional contagion over online social network. In the next chapter, we describe the gap of related research and summarize the novelty of this research.

2.2 Online emotional contagion

Recently, since Picard advocated the concept of affective computing (Picard 1997), research works in which emotional recognition and emotional state affect user’s cognition and behavior are active not only in the field of psychology but also in the field of computer science (Costa et al. 2016; Mottelson and Hornbak 2016). Along with that, research works on emotional contagion of online social network have also appeared.

Kramer showed that emotional contagion occurs via text-based computer-mediated communication, without face-to-face interaction through social network (Kramer et al. 2014). In an experiment with Facebook users, this research was used to test whether emotional contagion occurs outside of face-to-face interaction by reducing the amount of emotional content in Facebook News Feed. When positive expressions were reduced, the users produced fewer positive posts and more negative posts and vice versa. In addition, Emilio investigated emotional contagion on Twitter (Ferrara and Yang 2015). The research evaluated emotional contagion by measuring the emotional value of the content on the Twitter timeline (without manipulating the timeline content) before posting their own tweets and comparing it against the value in their posts. Emilio concluded that there is a correlation between the emotional values of the users’ viewing timeline tweets and posting tweets. These two literature works focus on the emotional contagion on the text-based social network.

Comparing with text, image is more attractive content for users. A Facebook report shows that image makes event engagement 100 times faster than text (Wang et al. 2015). There is a research regarding emotional contagion of an image social network. Yang et al. investigated how users’ emotional state influences each other in the social network by employing a widely used image social network (Flickr) data (Yang et al. 2016). Our work is at the same viewpoint in terms of focusing on the emotional contagion over the image social network. However, the biggest novelty in our work is our particular focus on the selfie-based emotional contagion. To the best of our knowledge, so far there is no other work addressing this area and our work is the first for investigating facial mimicry over the selfie-based social network.

3 SmileWave

Fig. 3

Photograph-taking screen

Fig. 4

Browsing screen

This section introduces SmileWave, our original social networking system for this research. Figure 2 shows the overview of SmileWave.

3.1 System requirements

We aim to observe the first stage of selfie-based emotional contagion in social network. Therefore, there are some requirements for measuring the occurrence of the first stage of emotional contagion. First, we have to acquire the users’ facial image when they view the others’ selfie photographs in the social network system. It also naturally needs to be conducted on social network. Thus, our options are either utilizing existing social networks, such as Facebook and Twitter, or creating our original social network particularly for this research purpose. On Facebook or Twitter, mainly because of vertical system integration of such services with official client applications, it is difficult for us to insert a third-party system that can detect and collect data on the user’s facial expression when they see the posted selfie photographs. Additionally, on such existing services, there is too much miscellaneous information that affects users’ emotional states, such as various types of media contents including text, photographs, videos, and even games. Since we cannot easily and accurately evaluate the effect of emotional contagion solely by smile selfie images on such environments, we decided to implement our original social network “SmileWave” in order to enable accurate evaluation with the real users. Furthermore, to get users use such an original social network in reality, it is important to design the service attractive enough for the users to enjoy it actively, spontaneously, and continuously. In the following section, we describe detailed functions of SmileWave followed by an explanation on the system architecture.

Fig. 5

Architecture of SmileWave system

3.2 Functions of SmileWave

We have implemented the following four distinctive functions:

• (1) Social networking system that can be used to share selfie photographs: SmileWave, as shown in Fig. 2, is implemented as a smartphone application to post and exchange selfies among users. The user takes a selfie photograph on the screen shown in Fig. 3 and taps the “Next” to go to the screen for browsing selfie photographs posted by other users, as shown in Fig. 4. When the user continues to tap the “Next” button and browses five other user’s selfie photographs, the photograph-taking screen shown in Fig. 3 is displayed again. In this way, the cycle that displaying the five browsing screen and photograph-taking screen is repeated. This purpose is to encourage the user to naturally take and post selfie photographs as they browse the others’ selfie photographs, without user’s feeling of uncomfortable usability.

• (2) Continuous analysis of reactive facial expression: In the browsing screen shown in Fig. 4, SmileWave displays other users’ selfies one by one and detects the viewing user’s facial expression continuously with the phone’s front camera. During this detection period, we will present our preliminary experiment for the design consideration in Sect. 3.3.

• (3) Smile degree of taken photographs and reactive facial expression for analysis: Taken selfie photographs and reactive facial expressions are uploaded to the server and calculated the smile degree on the server. The smile degree is a numerical value in the range of 0–100%. On the server of the system, the smile degree is calculated by using the Face++ API provided by Megbii (Megvii Inc 2013). The evaluation on the performance analysis of Face++ API will be verified in the preliminary experiment in the next chapter.

• (4) Real-time feedback to the users on smiling: For each selfie photograph in the browsing window, reacted user list who have viewed the posted photograph will be displayed below the image. We illustrated it in Fig. 4 as “Mikio, Shin and 1 other user.” In this manner, the users can receive real-time feedback that can motivate and encourage the users to use the system continuously.

Figure 5 shows the architecture of SmileWave.

3.3 Preliminary experiment

In order to finalize the parameters inside the system, we conducted two preliminary experiments. Firstly, we need to make sure that our face analysis engine appropriately extracts smile degree of human faces in a reasonably similar way as we human do. Next, we need to evaluate the necessary duration for capturing images of the user’s face in order to detect changes in their facial expressions accurately. The change in facial expression typically occurs in a moment. Continuously capturing user’s image and detecting the smile degree while the user is browsing photographs results in high load and power consumption in the system, particularly in the mobile device side. Thus, SmileWave should enable such function only for a minimum duration to realize a certain detection performance with minimized resource consumption.

3.3.1 Validity of smile degree

Smile degree is essential when evaluating selfie-based emotional contagion. However, it is difficult to grade a smile in an image. This is because each person evaluates the degree subjectively when he/she looks at other people’s eyes or mouth. Although there is some range within the degree of a smile, the judgment toward magnitude of that smile should match to certain extent among people. Therefore, we evaluate the difference between the smile degree calculated by our system and that indicated by users.

We first asked 10 participants to take 11 photographs each with different smile degrees starting from 0% and increasing by 10% until they reached 100%, all of which is based on the user’s subjectivity. We collected 110 images in total and conducted two different measurements. First, we calculated the smile degrees of these images using the analysis component in our system. Next, we asked 36 university students and faculty members (male 20, female 16) apart from the 10 users mentioned above to evaluate the smile degree of these photographs, based on their own personal sense, on a Web application shown in Fig. 6. The Web application displays three photographs: two showing ground truth smile images of a target person with 0% and 100% smile degrees and the third, the middle, showing the target photograph that the participant is asked to evaluate. Referring to the ground truth photographs, the participants decide the smile degree of the target photograph subjectively. Thirty-six participants were requested to evaluate photographs of as many users as possible, with a minimum quota of at least two people per person. The order of photographs and the target users were randomized in the system. The results of the two measurements are shown in Figs. 7 and 8. We found that the smile degree determined by Face++ and by the 36 participants was approximately in agreement.

In order to test whether the performance of smile degree measurement by Face++ and the user’s subjective evaluation was equivalent, we used the two one-sided tests (TOST) procedure. It is essentially a statistical method which can be used to validate the hypothesis that the difference between two means is within a given interval. This interval is specified in terms of a lower ($-\varDelta _{\mathrm{L}}$) and an upper ($\varDelta _{\mathrm{U}}$) equivalence bound which is set the range acceptable for equivalence by user’s definition. In this study, the two patterns of the equivalence bounds ($-\varDelta _{\mathrm{L}}$, $\varDelta _{\mathrm{U}}$) were chosen as ($-10, 10$) and ($-20, 20$). With these two patterns of the equivalence bounds, we conducted the TOST procedure on evaluating the equivalence of the observed smile degree of user’s subjective evaluation and the observed smile degree of Face++ from 10% to 90% of the smile degree of ground truth. We calculated the p value of the TOST procedure for each equivalence bound ($-\varDelta _{\mathrm{L}}$, $\varDelta _{\mathrm{U}}$) such as ($-10, 10$) and ($-20, 20$). Table 1 shows the greatest p values of ($-\varDelta _{\mathrm{L}}$, $\varDelta _{\mathrm{U}}$) toward each smile degree of the ground truth (from 10% to 90%). When equivalence bound ($-\varDelta _{\mathrm{L}}$, $\varDelta _{\mathrm{U}}$) is ($-10, 10$), 10%, 20%, 30%, and 90% of smile degree of the ground truth were found to be within the equivalence bounds at $\alpha$ = 0.05. On the other hand, when equivalence bound ($-\varDelta _{\mathrm{L}}$, $\varDelta _{\mathrm{U}}$) is (-20, 20), results indicated that most data are to be within the equivalence bounds at $\alpha$ = 0.05 except for 70% and 80% of smile degree of ground truth. Although 70% and 80% of smile degree of ground truth were not found to equal, these had approached significance. From these results, we concluded that human subjective evaluation and performance of Face++ API are roughly matched, and we decided to use the value of smile degree by Face++ API.

Table 1 Results of TOST procedure

Fig. 6

Web application for evaluating the smile degree according to personal sense

Fig. 7

Human user’s subjective evaluation results of smile degree

Fig. 8

Smile degree values detected by Face++ API

3.3.2 Validity of detection time of facial expression and system performance

It is known that a change in the facial expression occurs immediately and continuously in emotional contagion. In the case of a smile, the smile degree starts to increase right after the user views the smile selfie, reaches up to a certain degree, and decreases back to a lower degree. Therefore, a fast response from the system is required. Since it is already known to validate that the change duration of facial expression is immediately (Ekman 1993), we conducted the small-scale experiment using SmileWave to test how quickly the smile degree reached the maximum and whether the system can detect every frame of user’s smile degree. We asked five university students (male 3, female 2) to create a smile using SmileWave in our laboratory. SmileWave was configured to capture images at 15 frames per second. For each experiment, we selected 1 second as the maximum duration (maximum number of images = 15) after which the user started changing his/her facial expression; we also analyzed how long it takes for the smile degree to reach a maximum. The smile degree of each image was analyzed by Face++ API.

The result of each user’s trace of smile degree over time is illustrated in Fig. 9. According to the results, the average number of frames for the users’ smiles that reach up to the highest smile degree was 11.3. Furthermore, in all cases the smile degree reached the top within 13 frames from the beginning (i.e., the change is generally completed within 13/15 seconds). Moreover, the system could detect the user’s smile degree in all 15 frames. From these results, we conclude that 15 frames (=1 second) of detection time is sufficient and appropriate as our design parameter.

Fig. 9

Change of smile degree in users’ reactive smiles

4 In-the-wild evaluation

In order to evaluate the selfie-based emotional contagion, we conducted extensive in-the-wild user on 38 participants for the duration of 2 weeks. The objectives of this study are as follows:

• 1. OBJ1: We want to confirm that the first stage of selfie-based emotional contagion occurs in a selfie-based social network.

• 2. OBJ2: We want to investigate if there is any difference in viewing user’s reactive smile degree according to the differences in smile degrees of a series of posted photographs that the users view for a certain time span.

• 3. OBJ3: We want to evaluate if there is any difference in viewing user’s reactive smile degree according to multiple different sets of a series of posted photographs ordered differently based on the smile degrees.

• 4. OBJ4: We want to confirm if there is any difference in viewing user’s reactive smile degree between a configuration that shows smile photographs of only those who had reactively smiled to the viewing user’s smile selfie in the past, and the contrary configuration.

4.1 Participants

For this study, we recruited 38 university students (22 males and 16 females) between the ages of 18 and 22, who used their smartphones on a daily basis. We installed SmileWave into the participants’ own iPhones. Upon the completion of this study, the subjects were eligible to win four latest gadgets via a lottery.

4.2 Photograph presentation strategies

In this study, SmileWave was configured to display the photograph-taking screen after the user scrolled through five posts. Thus, from the viewpoint of the user, he/she was prompted to take a photograph every time after viewing five photographs. This flow was repeated until the user either quit the application or switched to another application. By controlling the content of five photographs to be presented, we can evaluate several different configurations described above. Concretely, we prepared the following eight different strategies in such photograph displaying.

• 1. “Latest”: Show five latest posted photographs.

• 2. “Random”: Show five posted photographs from any arbitrary posted photographs.

• 3. “HighDegree”: Show five posted photographs with high smile degree (80 and above).

• 4. “LowDegree”: Show five posted photographs with low smile degree (below 30).

• 5. “HighToLow”: Show five posted photographs in the order of their smile degree from high to low.

• 6. “LowToHigh”: Show five posted photographs in the order of their smile degree from low to high.

• 7. “WithHistory”: Show five posted photographs of other users who have reactively smiled to the viewing user’s smile selfie in the past.

• 8. “NoHistory”: Show five posted photographs of other users who have not reactively smiled to the viewing user’s smile selfie in the past.

Regarding more precise system configuration, the system at first displays five latest photographs (which is equal to Latest strategy) to the user. This is to present the latest situation (posted photographs) on the SmileWave social network to each user, with the nature of “timeline.” Then, after the user experiences the first photograph-taking screen, following photograph stream on the screen will be displayed based on a randomly selected strategy (except for the Latest). The information on the strategy selection is hidden from the users; thus, the users do not know that strategy he/she is currently experiencing. As long as the user keeps using the application, the photograph-taking screen and five photographs chosen based on a randomly selected strategy come one after the other.

4.3 Experiment setup

There is a risk that the evaluation results of the experiment may be affected if the participants know that we focus on the change of facial image when participants browse the posted others’ selfie image. Therefore, in this experiment, we did not tell the intention of this experiment. Instead, we allowed the users to use SmileWave freely and explained that it was a social networking application that shared selfies. Therefore, participants did not just upload smiley selfies, but also shared selfies that were not smiling. At this point, we informed the subjects that the system captured the face image of the participants with carefully considering privacy when browsing it in the background. We also explained that the user list displayed at the lower part of the viewing screen is just a list of users who have already viewed the image in order to entertain the system for real-time feedback. In addition, we hid the application name to install it.

4.4 Experiment procedure

On the first day of the user study, we conducted an instruction meeting for the participants. We explained overview of the experiment and installed our software into their own smartphones. After all the subjects completed the installation, we started the actual experiment period. On the final day of this experimentation period, we sent out a post-study questionnaire to the subjects. During the experiment, we did not limit any usage of SmileWave, other iPhone features and applications to get data in the users’ natural daily smartphone lives.

4.5 Evaluation results

During this user study, we were able to collect 512 selfie photographs and 9798 reactive facial images from 38 participants.

4.5.1 OBJ1: Confirmation of the first stage of emotional contagion

We compared the smile degree values of the following two images, (1) the first captured reactive facial image and (2) the reactive facial image with the maximum smile degree. Figure 10 shows the smile degree of the first captured reactive image (mean 52.1%, std 25.9) and the reactive image with maximum smile degree (mean 67.3%, std 20.1). We conducted the t test and revealed the significant difference ($p < 0.05$). The analysis also revealed that user’s smile degree increased by 15% on average in the reactive facial expressions when viewing other user’s posted photograph. Although it might be natural that smile degree is increased by comparing the first capture image and maximum smile degree’s images, it notes that the average smile degree of first captured reactive facial image was not low, 52%, and the smile degree is calculated between 0 and 100%. Therefore, we concluded that the first stage of selfie-based emotional contagion was occurred in this in-the-wild experiment.

Figure 11 shows the increase in the smile degree of the person viewing the posted image, in respect to the smile degree of the posted image. We conducted the correlation analysis between smile degree of posted selfie photographs and the increase in viewers’ smile degree. The correlation coefficient was 0.26 that indicated a weak positive correlation. Therefore, we found a slight tendency that higher the smile degree value of the posted photograph, the more the increase in reactive smile degree of the viewing users. It means that photographs with a higher smile degree strengthened the influence of selfie-based emotional contagion.

Fig. 10

Comparisons of smile degree of the first captured reactive image and reactive image with maximum smile degree

Fig. 11

Smile degree of posted photographs and increase in viewers’ smile degree

Fig. 12

Men’s data in Fig. 10

Fig. 13

Women’s data in Fig. 10

Figure 12 shows the smile degree of the first captured reactive image (mean 45.2%, std 26.2) and the reactive image with maximum smile degree (mean 63.4%, std 21.2) using only men’s collected reactive image. On the other hand, Fig. 13 shows the smile degree of first captured reactive image (mean 60.3%, std 24.1) and the reactive image with maximum smile degree (mean 72.1%, std 17.0) using only women’s collected reactive image. We found that there was a significant difference between the men’s and women’s data ($p < 0.05$). From the t test result for comparing men’s reactions with women’s reactions, women tended to have a higher smile degree on both the first captured reactive image and the reactive image with maximum smile degree ($p < 0.05$). We have confirmed in advance that there was no difference between men’s and women’s performance in Face++ smile degree measurements.

4.5.2 OBJ2: Difference in smile degree after viewing photographs with different smile degrees

Figure 14 shows the average smile degree of users’ posted selfies after viewing five photographs with three different presentation strategies, namely Random (left), HighDegree (center), and LowDegree (right). After viewing five photographs with the Random strategy, users’ average smile degree in posted selfies was 31.4% (standard deviation 20.0). In the case of HighDegree, the average was 44.8% (standard deviation 26.7). In the case of LowDegree, the average was 15.7% (standard deviation 13.8). Compared with Random, HighDegree resulted in 43% increase in users’ smile degree. On the other hand, LowDegree resulted in 50% reduction in users’ smile degree. Two-way repeated ANOVA revealed an effect of strategy ($F_{(2,111)}$ = 6.9, $p=0.0015$). A multiple comparison test with Ryan’s method also revealed the significant difference between Random and HighDegree ($p = 0.025$) and between HighDegree and LowDegree ($p < 0.0001$). No significant difference was found between Random and LowDegree ($p = 0.15$). From these results, we discuss that, in the case of photographs with higher smile degree, the rate of user’s selfie-based emotional contagion increases when the time to some extent has passed.

The number of posted selfies with Random strategy was 29, HighDegree was 37, and LowDegree was 22. We observed that the users took more pictures when photograph-taking screen was displayed in the case of HighDegree and took less in the case of LowDegree.

Fig. 14

Comparisons of smile degree in posted selfies after viewing photographs with different smile degrees

4.5.3 OBJ3: Difference in smile degree after viewing photographs ordered differently based on the smile degrees

Figure 15 shows the average smile degree of users’ posted selfies after viewing five photographs with three different presentation strategies in terms of ordering of photographs, namely Random (left), HighToLow (center), and LowToHigh (right). After viewing five photographs with the HighToLow strategy, users’ average smile degree in their posted selfies was 21.6% (standard deviation 16.2). Compared with Random (31.4%), smile degree in HighToLow was decreased by 31%. On the other hand, in the case of LowToHigh, the users’ smile degree was 34.2% (standard deviation 22.1). These results indicate that the smile degree in LowToHigh was increased by 9% from Random. Again, two-way repeated ANOVA revealed an effect of strategy ($F_{(2,111)}$ = 5.85, $p=0.004$). Moreover, a multiple comparison test with Ryan’s method revealed the significant difference between Random and HighToLow ($p = 0.03$) and between HighToLow and LowToHigh ($p = 0.001$). No significant difference was found between Random and LowToHigh ($p = 0.20$).

Figure 16 shows the number of posted selfies with Random strategy (left) (number, 29), HighToLow (center) (number, 24), and LowToHigh (right) (number, 37). Comparing three strategies, we can observe that the users took more pictures after he/she viewed five photographs ordered from the one with the lower smile degree to the one with the higher degree, and vice versa.

Fig. 15

Comparisons of smile degree in posted selfies after viewing photographs with different smile degree-based ordering

Fig. 16

Comparisons of the number of posted photographs between strategies (with different smile degree-based ordering)

Table 2 Summary of the answers for questions 1–8 (1: strongly disagree, 2: disagree, 3: neutral, 4: agree, 5: strongly agree)

4.5.4 OBJ4: Difference in smile degree after viewing photographs from those who had reactively smiled in the past

Figure 17 shows the average smile degree of users’ posted selfies after viewing five photographs with three different presentation strategies in terms of relationship between users, namely Random (left), WithHistory (center), and NoHistory (right). After viewing five photographs with the WithHistory strategy, users’ average smile degree in their posted selfies was 31.8% (standard deviation 22.4). This result is speculated by 1% compared with that in Random (31.4%). On the other hand, in the case of NoHistory, the users’ smile degree went down to 22.9% (standard deviation 9.4). The value was decreased by 27% in a comparison. Our two-way repeated ANOVA did not reveal an effect of strategy ($F_{(2,111)}$ = 1.94, $p=0.15$) in this case. Figure 18 shows the number of posted selfies with Random strategy (left) (number, 29 ), WithHistory (center) (number, 27), and NoHistory (right) (number 24).

Fig. 17

Comparisons of smile degree in posted selfies after viewing photographs from different types of users

Fig. 18

Comparisons of the number of posted photographs between strategies (with different types of users)

4.6 Post-study questionnaire

After 2--week user study period has completed, we conducted a post-study questionnaire to 38 participants. We asked each user 10 questions that are composed of 8 questions of 5-level Likert scale (1—strongly disagree, 2—disagree, 3—neutral, 4—agree, and 5—strongly agree) and 2 questions of free form text. Table 2 summarizes 8 question contents and participants’ answers. We asked each user “Did you have any findings through this experiment of viewing each other’s selfie?” for Question 9 and “Were there a difference in your smile, expression, and emotion due to using SmileWave comparing to present and before experiment starts? If yes, what kind of change is it?” for Question 10.

On Question 1, approximately half (48.4%) of the users agreed (41.9%) or strongly agreed (6.5%). It is an evidence that we confirmed the occurrence of the first stage of selfie-based emotional contagion.

On Question 2, the answer with the biggest number of the users was “Agree” (41.9%). Using our system is considered to have made the users happy. This result strongly supports the occurrence of not only the first stage of the selfie-based emotional contagion, but also the second stage. Therefore, we become more confident that selfie-based emotional contagion in a photograph-based social network is occurred.

On the results on Questions 5 and 7, we believe that SmileWave successfully fulfilled all the functional requirements on (1) real-time feedback to the user, (2) easy to use anytime and anywhere, and (3) simple user interface and high usability.

On Question 8, surprisingly, few users (agree 12.9%, strongly agree 3.2%) were conscious of the in-camera being activated. Therefore, we conclude that the potential bias by consciousness that the user has taken the photographs with the in-camera had not been affected to our in-the-wild experiment.

On Question 9, an answer from one user includes “I think SmileWave have strengthen our friendship bonds since watching smiles of each other made me feel like we are really facing each other every day,” and another user answered “At first, I felt the resistance to share my face, but it gradually disappeared.”

On Question 10, one user answered “By taking a smile selfie through using SmileWave, I tried to keep smiling more than usual” and another user answered “I feel that time spent smiling on everyday life has speculated somehow.”.

5 Discussion

In this user study, the results confirmed that the smile degree was increased by viewing the photographs posted from other users described as Fig. 10 in Sect. 4.5.1. We thus concluded that the first stage of smile emotional contagion does occur in a selfie-based social network. Since the second stage is independent of the difference of communication media (as we described in Sects. 1, 2), we concluded that selfie-based emotional contagion over the social network is confirmed throughout our entire evaluation studies. Interestingly, we also found that there was a weak correlation between smile degree of posted selfie photograph and increase in smile degree when users view other users’ posted photographs in Fig. 11. There is possibly new research perspective that the effect on emotional contagion may differ depending on the degree of facial expression. On the other hand, we also found that viewing five photographs with highly smile degree has a possibly significant impact on the smile degree of user’s selfie when taking photographs. Although a previous literature discussed that emotional contagion is instantaneous (Hatfield et al. 1993), there is a possibility that the momentary effect of emotional contagion may accumulate continuously and causes longer-term emotional contagion as observed in our evaluation. This required more careful investigation because we did not confirm in this experiment that all five smiley selfie photographs have an influence and it might be the influence of only the photograph just before the photograph was taken. This investigation is one of our future works.

Furthermore, we observed that there is a stronger effect on emotional contagion after he/she viewed the photographs ordered from the one with the lower smile degree to the one with the higher degree in Fig. 15. From this result, we hypothesize that the latest influence is more powerful for the effect of emotional contagion, reflecting the nature of human’s short-term memory. The result of Fig. 16 shows that the content of order from the one with the lower smile degree to the one with the higher degree may improve the user’s engagement.

Finally, our post-experiment survey described in Sect. 4.6 revealed positive outcome both from qualitative and quantitative questions in terms of the effect of selfie-based emotional contagion. Regarding Questions 1 and 2, approximately half of the users are conscious of change in facial expression and feeling happy. Emotional contagion is known to occur at both the conscious and unconscious levels (Kelly and Barsade 2001; Druckman and Bjork 1994) and our results support it. The result collected by Question 9 possibly supports the positively relationship friendship among group members and emotional contagion (Barsade 2002; Kramer et al. 2014).

Although we successfully confirmed the occurrence of facial mimicry which the user’s facial expression mimic (to smile) by browsing posted others’ selfie photographs, as the first stage of online emotional contagion described as shown in Fig. 1, it is noted that this study is the first step to investigate emotional contagion in online communication over social network. In this study, we mainly focus on the change of smile degree for “individual” user in order to confirm the occurrence of selfie-based emotional contagion over online communication. Therefore, it can be seen that selfie-based emotional contagion occurred regardless of the scale of this experiment since it was investigated from an individual point of view. When we further investigate the online emotional contagion over social network such as how to spread the effect of emotional contagion among people and how users’ positions in social network affect their influential strength, we need to conduct longer-term and larger-scale user study in order to generalize the results. This is one of future works.

One of our limitations is that our experimental participants were targeted only to college students between the ages of 18 and 22. Existing studies in the past have mentioned emotional contagion as a property of all human beings, regardless of culture, age, and gender (Hatfield et al. 1993). However, when it comes to online emotional contagion, we could not investigate whether it was a shared property of all human beings due the different time and familiarity with online communication. Therefore, as mentioned above, larger-scale user study is one of future works.

Collecting user’s emotional states through various types of methodology (such as ESM or psycho-physiological sensors) is another future work of us. We need to carefully design the system so that such data collection does not interfere and/or discourage the use of SmileWave in the longer-term and large-scale evaluation phase. SmileWave is now released in Apple’s AppStore, so that it helps to enable to get more in-the-wild data.

In addition, building our original model on smile degree estimation is our immediate future research item. The Face++ API is a private API with their mechanism not opened to the public. Thus, in this paper, we analyzed the current performance and classification characteristics of the API in our preliminary experiment in order to support the validity of our user study results. Meanwhile, we are starting to build our original smile degree estimation model so that we can have full control on the smile degree analysis. Deployment of such model is among our future work.

6 Conclusion

In this paper, we proposed SmileWave, a novel social network that shares selfie photographs among users and detects the change of smile degree while the users browse others’ selfie photographs, in order to evaluate the occurrence of facial mimicry. Our evaluation with 38 participants for 2 weeks has clarified the following key things. Firstly, the users’ smile degree improved 15% on average when the user looked at posted selfie photographs. Therefore, we concluded that selfie-based emotional contagion occurs over the social network. We also found a slight tendency that there is an effect on selfie-based emotional contagion by showing smile selfies with higher smile degree. Finally, we observed that there is a stronger effect on selfie-based emotional contagion after he/she viewed the posted selfie photographs ordered from the one with the lower smile degree to the one with the higher degree. To the best of our knowledge, our extensive evaluation is the first work that deeply investigated and revealed the effects of selfie-based emotional contagion over the social network.