Engagement and Effectiveness of a Healthy-Coping Intervention via Chatbot for University Students During the COVID-19 Pandemic: Mixed Methods Proof-of-Concept Study

Introduction

Increased numbers of adults, particularly university students, are experiencing symptoms of stress, anxiety, and depression [1,2], which are exacerbated by the recent restrictions introduced because of the COVID-19 pandemic [3,4]. In addition, up to 74% of adults experience their first onset of a mental health diagnosis before the age of 24 years [5]. However, about three-quarters of college students who are in need of clinical services do not access them [6]; this is because they have low mental health literacy and do not recognize a need for treatment [7], but also because of the high cost of treatment, low availability, or attitudinal barriers, such as perceived stigma [8,9]. In recent years, wider access to digital technology and mobile phones has presented new opportunities for overcoming these barriers, by offering the possibility of delivering digital mental health interventions in a more scalable and convenient way [10,11].

Empirical studies on evidence-based digital interventions for mental health, including internet-based cognitive behavioral therapy (CBT), have shown that these interventions are effective, feasible, and acceptable to users [12-14], although some limitations have been found, mainly regarding low engagement by users and low completion rates [15,16]. The integration of human coaching and support in digital mental health interventions can help improve adherence and behavior change outcomes [17,18], although this may reduce the scalability of such solutions. The design and deployment of conversational agents, such as chatbots, as virtual coaching solutions to deliver psychoeducational interventions for mental health and well-being have so far been shown to be ideal in maintaining the intuitiveness and naturalness of dialogue-based interaction, while exploiting the benefits of full automation [19,20]. These solutions seem to be particularly interesting to deploy at the time of the COVID-19 pandemic, when restrictions to face-to-face social encounters and interactions make it even more difficult to access human psychological support.

The use of chatbots for digital mental health interventions has attracted interest in the design community. A growing number of studies are reporting their acceptability and feasibility for users [20-22], as well as their effectiveness in reducing perceived stress [16-25] and abnormal eating behavior [26], thereby improving symptoms of anxiety [10,23-25], depression [10,24,25], and insomnia [27]. Like previous work and meta reviews have found on digital mental health interventions for the general adult population and for university students, a main limitation of these studies is their exclusive focus on randomized controlled trials, which prevents a full understanding of the challenges regarding user engagement, uptake, and adoption of these solutions [15-28]. More research is needed to understand the user experience (UX) and engagement with digital mental health interventions, to not only prove their clinical efficacy but also to facilitate their successful implementation in real-world settings [25-29].

The objective of this study was to assess the levels of engagement by university students and effectiveness of their interactions with a psychoeducational intervention delivered by the Atena chatbot over 1 month, in order to improve their coping and resilience skills during the COVID-19 pandemic. The study design was based on a mixed methods approach and encompassed two phases of the ORBIT (Obesity-Related Behavioral Intervention Trials) framework [30] for intervention design (Phase I) and preliminary testing (Phase IIa). In Phase I, the intervention, targets, and components were defined in order to specify their clinically relevant effect on users and to refine the intervention components. In Phase IIa, a proof-of-concept implementation of the digital intervention and chatbot was realized and preliminary testing was done for engagement and effectiveness with a convenience sample of university students. We hypothesized that use of the Atena chatbot over a 1-month period would lead to a reduction in symptoms of stress and anxiety and would prove to be engaging and acceptable to use by students.

To our knowledge, this is the first study to investigate the potential effect of a healthy-coping chatbot intervention during the COVID-19 pandemic, a time when the empowerment gained from stress management skills is much needed by the adult population, particularly university students.

Methods

The Atena Chatbot Design

Atena is a chatbot that delivers psychoeducational content in order to coach users in using coping strategies and improving their mental well-being by means of conversational dialogues with the coach Atena and audiovisual educational materials. It is accessible for free on the Telegram messaging app and is available on mobile or desktop devices. The chatbot was built using JavaScript and was developed by the Digital Health Lab at Fondazione Bruno Kessler (FBK) research center.

The digital mental health intervention delivered by Atena is aimed at improving users’ well-being by raising self-awareness about one’s thoughts and emotions; it does this by suggesting effective coping strategies that can be adopted when facing typical stressful situations, thus promoting mental well-being and preventing mental distress. The full program consists of eight short sessions, each lasting about 10 minutes, delivered twice a week for 4 weeks. Each session is initiated by the chatbot on a scheduled plan decided by the user during the first session. Users are invited by the chatbot to fill in web-based versions of psychological symptom questionnaires at baseline and postintervention, as well as a user engagement scale at the end of week 2.

The conversations between Atena and the user are informed by evidence-based approaches and intervention strategies from positive psychology and CBT, including psychoeducation on self-awareness and self-efficacy, conflict resolution, assertive communication, and practical exercises on mindfulness delivered at the end of each session [31,32]. These intervention strategies based on positive psychology, CBT, and mindfulness practices have been recently deployed by fully automated conversational agents targeting anxiety, stress, and depression, with promising results in terms of efficacy and acceptability by users [10,16,24,25]. The aim of the conversations is to trigger and support the user in self-reflecting on personal thoughts and emotions experienced in daily stressful settings, in learning how to best deploy more functional strategies to overcome difficulties, and in better managing stress and anxiety. The intervention program, including the behavioral and clinical targets, as well as the audiovisual content, were originally developed by a team of three clinical psychologists to fit the needs of the general adult population facing stress and anxiety challenges posed by the COVID-19 pandemic (Figure 1). A refinement of the conversations and video materials was then performed by the same psychologists in collaboration with two UX and behavior change experts in the design team, in order to adapt the language, videos, and chatbot interaction to the needs of the students in the target group.

The Atena chatbot always starts the conversation on the scheduled date and time for that session, and the user replies by choosing among a predefined set of answer options. In this way, the conversation flow can be customized to sound more relevant and empathic to the user’s answers. Each session starts with a short psychoeducational video cartoon, representing typical challenging situations experienced by young characters and the corresponding strategies adopted to cope with them; they mimic relevant situations experienced by the target users, fostering their identification with those situations and their learning. In the final part of the session, the chatbot invites the user to perform a mindfulness exercise to focus her sense of presence and attention by following a coaching voice provided through an audio track (Figure 2).

Upon enrollment, users were made aware that the Atena chatbot was not intended to replace professional mental health treatment, but that it was a prototyped digital tool designed to support psychoeducational interventions and was going through preliminary testing in this study. In the first session, an introductory video cartoon was presented to the users by the Atena chatbot to explain the main features, applications, and limitations of chatbot technology, in order to facilitate the creation of appropriate expectations toward the digital tool and intervention tested.

‎View this figure

‎View this figure

Participants

The Atena chatbot was voluntarily accessed and used by a recruited convenience sample of 71 university students; students ranged in age from 18 to 34 years (mean 20.6, SD 2.4) and the group consisted of 68% females (48/71). Students attended a human-computer interaction course in the first year of a bachelor’s degree program at the University of Trento, Italy; they were recruited and invited by SG and RM to access the chatbot via the messaging app Telegram. Participation was on a voluntary basis and the inclusion criteria were as follows: (1) being a university student in their first academic year and (2) owning a smartphone with a Telegram account. Atena was designed to offer coaching conversations, audiovisual materials, and mindfulness meditation in order to improve coping skills and well-being. Students used Atena between mid-October and November 2020, a period affected by the second wave of the COVID-19 pandemic in Italy, with restrictions to citizens’ mobility, social distancing, and blended learning recommended at the university. All users were Italian speakers located in the northeast of Italy.

Measures

Perceived Stress Scale

The 10-item Perceived Stress Scale (PSS-10) is a brief self-report instrument containing 10 items rated on a 5-point Likert scale, ranging from 0 (never) to 4 (very often). The PSS-10 measures the perception of stress (ie, the degree to which situations are appraised as stressful) by asking respondents to rate the frequency of their thoughts and feelings related to situations that occurred recently [33] (eg, “In the last month, how often have you felt nervous and stressed?”). Levels of stress are determined based on scores as follows: low (0-13), moderate (14-26), and high (27-40). The PSS-10 is one of the most widely used psychological instruments, reporting good psychometric properties. In this study, the Cronbach α was .84.

Generalized Anxiety Disorder Scale

The 7-item Generalized Anxiety Disorder scale (GAD-7) [34] is a 7-item self-report scale based on a 4-point Likert scale, ranging from 0 (not at all) to 3 (nearly every day). The GAD-7 is used to assess anxiety symptoms over the past 2 weeks (eg, “How often have you been bothered by feeling afraid something awful might happen?”). The total scores are separated into the following four categories to determine the anxiety symptom levels: none (0-4), mild (5-9), moderate (10-14), and severe (≥15) [23,24]. In this study, the Cronbach α was .86.

Five-Facet Mindfulness Questionnaire

The Five-Facet Mindfulness Questionnaire (FFMQ) [35] is a 39-item self-report measure, evaluated on a 5-point Likert scale, ranging from 1 (never or very rarely true) to 4 (very often or always true); the questionnaire assesses the tendency to be mindful in daily life. The five facets are as follows: (1) observing (8 items; eg, “When I’m walking, I deliberately notice the sensations of my body moving”), (2) describing (8 items; eg, “I’m good at finding words to describe my feelings”), (3) acting with awareness (8 items; eg, “When I do things, my mind wanders off and I’m easily distracted”), (4) nonjudging (8 items; eg, “I criticize myself for having irrational or inappropriate emotions”), and (5) nonreactivity (7 items; eg, “I perceive my feelings and emotions without having to react to them”). Individual facet scores range from 8 to 40—except for the nonreactivity facet, which ranges from 7 to 35—with higher scores indicating more mindfulness. The sum of the direct- and reverse-scored items gives a total score that ranges from 39 to 195. In this study, the Cronbach α values were .74 for observing, .92 for describing, .85 for acting with awareness, .91 for nonjudging, .79 for nonreactivity, and .84 for the total score.

User Engagement Scale–Short Form

The User Engagement Scale–Short Form (UES-SF) [36] is a self-report measure comprised of 12 items rated on a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). The UES-SF measures the main determinants of adherence and, in particular, it assesses four factors: (1) perceived usability (eg, “Using Atena was frustrating”), (2) aesthetic appeal (eg, “Atena appealed to my senses”), (3) focused attention (eg, “I lost myself in this experience”), and (4) reward (eg, “This experience was rewarding”). The reward factor is a summary of three factors from the original User Engagement Scale (UES): endurability, a measure of how successful the interaction was and the likelihood of recommending the app to others; novelty, a measure of curiosity and interest; and felt involvement, a measure of the feeling of being “drawn in” and having fun [36]. A total score is calculated and scores for each of the four subscales are calculated by adding scores for all the items related to their factor and dividing them by the total items.

A back-translation was conducted; thus, all scales were translated into Italian and validated for use within this population. In this study, the Cronbach α values were .66 for perceived usability, .66 for focused attention, .52 for aesthetic appeal, .87 for reward, and .83 for the total scale.

Procedures

After signing and submitting their digital consent form, users were instructed on how to fill in the web-based versions of the PSS-10, the GAD-7, and the FFMQ questionnaires, by using an alphanumeric pseudonymization code decided by them, and how to access the Atena chatbot on the Telegram app. In the first session with the chatbot, users were welcomed and provided with the introductory video on chatbot technology. They were also asked to set a desired day and time for their two weekly sessions with Atena, according to their preferences. The chatbot prompted the user to start a session at the scheduled day and time, but the user was free to pause, continue, or discontinue the session at any time.

At the end of week 2 (session 4), the users were invited by the chatbot to fill in the UES-SF questionnaire to assess their early engagement with the intervention and submit any free-text comments about their UX with the chatbot during the first 2 weeks of interaction.

At the end of week 4 (session 8), the users were invited by the chatbot to again fill in the PSS-10, GAD-7, and FFMQ questionnaires. The chatbot thanked them for their participation in the study and recommended that they continue engaging with the psychoeducational content that had been delivered, in order to improve their coping skills. A total of 4 weeks after the end of the study, participants were also asked to fill in a brief online survey to report on what they most liked and disliked about their experience with Atena, and whether they had continued to practice any of the exercises provided during the intervention with or without the support of Atena in the last 4 weeks. No monetary incentive was provided for participating in this study.

Ethics and Informed Consent

The study was reviewed and approved by the FBK Institutional Ethics Board since it involved a nonclinical population. Participants indicated their consent for their pseudonymized data to be used for research purposes after reading an information sheet. All study data were collected by the Digital Health Lab of FBK. Because of deidentification of all data transmitted between the Atena chatbot and the user, usage data were not linked to specific research participants and were, therefore, reported in an aggregated format.

Statistical Analyses

Statistical analyses were performed using R, version 4.0.0 (The R Foundation) [37], and SPSS Statistics, version 24.0 (IBM Corp) [38]. The Shapiro test was carried out to evaluate the normal distribution of the variables included in this study, such as the GAD-7 and the PSS-10 questionnaire scores (ie, anxiety and stress symptoms, respectively), the total score and the five subscale scores of the FFMQ, and the UES-SF questionnaire score.

The main descriptive analyses (ie, mean, standard deviation, and frequencies) were performed in order to assess the demographic characteristics of the overall sample (ie, age and gender), as well as the GAD-7 and the PSS-10 scores (ie, anxiety and stress symptoms, respectively), the total score and the five subscale scores of the FFMQ, and the UES-SF questionnaire score.

A paired-samples t test was conducted to evaluate differences between pre- and postintervention scores concerning the GAD-7 and the PSS-10 (ie, anxiety and stress symptoms, respectively) as well as the total score and the five subscale scores of the FFMQ. A P value equal to or less than .05 was considered statistically significant.

An independent-samples t test was conducted in order to evaluate differences between pre- and postintervention findings considering two clusters of users’ symptoms, extreme versus moderate ranges, as follows: (1) minimal and severe and (2) mild and moderate referring to the GAD-7 and the PSS-10 questionnaire scores. A P value equal to or less than .05 was considered statistically significant.

Participants’ responses to open-ended questions from the online final survey were analyzed by SR and SC using thematic analysis and were reported as frequencies. Data were analyzed thematically using an inductive, data-driven approach guided by the procedure outlined in Braun and Clarke [39]. Data codes were generated systematically, then collated into themes and applied to the entire data set to generate frequencies.

Results

Participants’ Questionnaire Scores by Gender

All the variables included in the analyses were normally distributed. As displayed in Table 1, at baseline, the overall sample (N=71) showed a mean that was close to the moderate range regarding the GAD-7 and the PSS-10 scores. More specifically, regarding the level of anxiety measured with the GAD-7, 35% (25/71) of students were in the mild range (ie, score of 5-9), 31% (22/71) were in the moderate range (ie, score of 10-14), and 20% (14/71) were in the severe range (ie, score of ≥15). Only 14% (10/71) of students were in the minimal range (ie, score of 0-4). Regarding anxiety symptoms for each gender, the analyses show that males experienced mild anxiety symptoms, while females experienced moderate anxiety symptoms.

Regarding stress symptoms evaluated with the PSS-10, 65% (46/71) of the sample were in the moderate range (ie, score of 14-26), 7% (5/71) were in the low range (ie, score of 0-13), and 28% (20/71) were in the high range (ie, score of 27-40). Both males and females displayed moderate stress symptoms.

Regarding the FFMQ scores, participants had an average total mindfulness score of 119.96 (SD 16.99); males scored higher (mean 126.26, SD 19.22) than females (mean 116.93, SD 15.10).

Attrition

A total of 86% (61/71) of participants provided data on the user engagement questionnaire at the end of week 2, which represented an overall attrition rate of 14% (10/71). A total of 58% (41/71) of participants completed the postintervention questionnaire, which represented an overall attrition rate of 42% (30/71).

Dropout was higher among participants in the minimal and mild anxiety ranges of the GAD-7 questionnaire—50% (5/10) and 44% (11/25), respectively—and was lower in the moderate and severe anxiety ranges: 40% (9/22) and 35% (5/14), respectively. Moreover, dropout was higher among participants in the moderate and low stress ranges of the PSS-10 questionnaire—45% (21/46) and 40% (2/5), respectively—and lower in the high stress range: 35% (7/20).

User Engagement With the Atena Chatbot

Table 2 shows the results of user engagement with the Atena chatbot as measured by the UES-SF questionnaire at week 2. A total of 61 out of 71 (86%) participants were in agreement with the perceived usability factor, which measured the affective (ie, frustration) and cognitive (ie, effortful) aspects as a result of the interaction. Participants answered in neutral ways regarding (1) the total UES-SF score; (2) the aesthetic appeal factor, which measured the sensory and visual appearance of the interface; and (3) the reward factor, which measured the hedonic aspects of experience, the felt involvement, the overall success of the interaction, and the willingness to engage with the chatbot in the future. Lastly, regarding the focused attention factor, which evaluated the focused concentration, absorption, and temporal dissociation, participants were in disagreement. Since participants were students attending a human-computer interaction course, their expectations regarding the UX and the quality of the user engagement with the chatbot might have been higher compared with students attending other higher education subjects. However, it should also be considered that participants were attending the first semester of their bachelor’s degree program, so their expertise in the field of technology design was still quite limited and comparable to that of other students in their age group.

Preliminary Efficacy From Completer Analysis

There was a reduction of participants in the severe anxiety range of the GAD-7 at postintervention, from 20% (14/71) to 10% (4/41) (Table 3). Also, 7 participants who were above the clinical cutoff score for the GAD-7 of 8 or higher at baseline moved below this cutoff point at postintervention (7/41, 17%).

The independent-samples t test between pre- and postintervention between the two clusters of students—cluster 1 students had extreme symptoms and cluster 2 students had moderate symptoms—showed a significant difference (t₃₉=0.94; P=.009) among anxiety ranges (ie, GAD-7 scores) in cluster 1, with a decrease of symptoms between preintervention (mean score 12.14, SD 6.88) and postintervention (mean score 10.07, SD 4.58). No other significant difference between pre- and postintervention GAD-7 scores was found.

In line with the GAD-7 results, the PSS-10 scores also showed an increase in the low stress range and a decrease in the high stress range (Table 4). This might indicate a positive effect of the intervention on participants who were in the more extreme stress level ranges compared to those in the intermediate stress ranges. Table 5 shows that the levels of stress symptoms (ie, PSS-10 scores) exhibited significant decreases (t₃₉=2.00; P=.05) between pre- and postintervention. Moreover, the mean scores of the subscales describing and nonjudging, as well as the mean total score of the FFMQ, showed significant increases (P<.05) between pre- and postintervention.

Use of the Chatbot

Participants interacted with the Atena chatbot an average of 78 times (SD 24.8; median 81; range 5-158) over the 4-week period. The average number of uncompleted sessions was 3.1 (SD 2.3) out of 8 overall sessions. Figure 3 shows a graph with the percentage of completed sessions over the 4-week period, showing that engagement and willingness to complete a session was higher during the first and the last weeks of the study.

‎View this figure

Qualitative Results

Overview

A total 17 out of 71 (24%) students—65% (11/17) females and 35% males (6/17)—provided qualitative data via the follow-up survey. Three main themes with related subthemes were identified (Table 6): (1) content, with the subthemes learning, reflection, multimedia, routine, mindfulness, motivation, originality, and repetitiveness; (2) UX, with the subthemes sense of reality, interaction, and flexibility; and (3) tasks, with the subthemes notification and availability.

Content

In terms of content, what students liked most was the availability of videos versus only having text-based dialogues in the interaction with the chatbot:

It was nice to have the possibility of accessing videos and not just using text, that might be boring.

However, one participant suggested that videos should be improved from a graphic design point of view:

What I liked less were the videos. They were interesting, but I think they should be improved in terms of graphics and visuals in order to be more engaging for users.

Some students appreciated the opportunity to learn new things—“I appreciated tips provided during the course, which were very interesting and also useful for learning new skills”—and to approach the mindfulness practice:

I find the bot an excellent first opportunity to start mindfulness practices, especially for those who - like me - can never find time to stop and breathe and have never tried anything like this before...

Some students also appreciated the originality of the exercises:

The originality of the exercises, in my opinion, is very important to foster change in people and help them.

One student suggested that the chatbot questions should be more varied in format so as not to be too repetitive:

In my opinion, questions should change over time: “how are you” should be asked in a more nuanced way, otherwise it sounds repetitive. I suggest, if possible, to vary the dialogues with the user, especially the welcoming messages.

User Experience

Regarding the UX, one student reported that what they liked most was the feeling of real-life interaction with the chatbot:

First of all, I appreciated the continuity of the short course with the chatbot, the interactions were well thought out and articulated, it felt like chatting with a real person.

Some students, however, reported some criticism regarding the user-chatbot interaction and gave some suggestions for future improvements:

The interaction with Atena should be more personalized based on the user’s answers, sometimes the answering options did not take into account the different nuances of mood.

Tasks

In the task theme, participants’ remarks mainly concerned the chatbot notifications and lack of reminders to resume a session when the user was interrupted by some other task:

I often didn’t have time to watch the videos at the scheduled time, then I forgot to resume them because there was no reminder.

The possibility of having the materials provided by Atena available in the chat was much appreciated:

I can watch videos again whenever I like.

A total of 8 out of the 17 (47%) students who provided qualitative data (ie, 4 females and 4 males) reported that they accessed the Atena videos and materials again after the 1-month study duration.

Discussion

Principal Findings

Results from this preliminary evaluation of the Atena chatbot intervention indicate that healthy-coping psychoeducation can be effectively deployed to university students and can have positive effects, especially on those who are more in need of psychological support to cope with stress and anxiety symptoms. Our results are in line with recent studies targeting the same population, showing that online stress management interventions are more effective for students with higher levels of stress, anxiety, and depression [40,41]. Higher engagement and lower attrition rates were also observed in those students in our sample who had more severe levels of anxiety at baseline. This is a very promising result for implementing future anxiety prevention and management solutions to be delivered during the COVID-19 pandemic and beyond. Results also showed a significant improvement in the capacity of participants to describe and accept their emotions, which can be an effect of the mindfulness practice and self-reflection elicited by the conversations with the chatbot. Training in these kinds of skills may be particularly needed by the university student population and could have positive effects on students’ mental well-being.

The baseline levels of stress and anxiety among our participants were significantly higher than those found in the same population of university students by previous research [42-44], as well as in the general population [45]. Previous studies analyzing the mental health of university students found lower levels of anxiety and stress symptoms, evaluated through the GAD-7 and the PSS-10 questionnaires, compared to those of our sample of university students [42-44]. This is not surprising, since the COVID-19 pandemic had a worsening effect on the general population and on university students in particular: a research study conducted on Italian university students to identify psychological consequences of the living conditions during the COVID-19 lockdown reported high levels of anxiety and stress, concentration disorders, psychosomatization, and, in several cases, reactivation of trauma and worsened sleep quality [46]. Moreover, from the time of our baseline assessment until the end of our study, the COVID-19 pandemic in Italy reached higher levels of infection, which required more severe restrictions to be introduced in schools and universities (ie, full online teaching) and in citizens’ everyday lives. It is likely that students experiencing more severe symptoms of anxiety felt more motivated to engage with the Atena chatbot and found it a more convenient solution to access psychoeducational support, by avoiding, at the same time, stigma and possible difficulties in accessing mental health services.

Overall, our quantitative and qualitative findings are aligned with recent chatbot evaluation studies [10,23-25] and will inform our design decisions for future developments. Results regarding attrition rates during the study, user engagement, as well as our qualitative findings suggest that our proof-of-concept intervention needs to be further refined to fully meet the requirements and preferences of the target users before being ready for randomized controlled trial evaluations. Our analysis indicates that the engagement and attractiveness of a chatbot-based mental health intervention for university students might wear off or reduce significantly after 2 weeks of interaction, requiring deeper levels of engagement through conversation and rewarding feedback from the chatbot in order to maintain users’ interest and commitment during the intervention. This might be particularly useful for supporting adherence of less motivated users, such as the ones with mild to moderate levels of stress and anxiety. Our study also helps shed light on what might be the ideal length, frequency, or intensity of digital mental health interventions for nonclinical populations. The duration and intensity of our intervention was sufficient to provide psychoeducational support to students without interfering too much with their daily life commitments, and it was also effective in triggering more self-reflection and mindfulness practice in the follow-up period. This might be interpreted as a signal of user empowerment and desired behavior change, although more research is needed to confirm this interpretation.

Limitations

This study presents some limitations that affect the generalizability of the findings. It reports data from the preliminary evaluation of a proof-of-concept chatbot intervention targeting a homogeneous population of university students without a control group. Although the findings on user engagement and preliminary effectiveness of the intervention are promising and aligned with previous research, further testing by means of controlled trials should be conducted to confirm any conclusion about efficacy and to verify its maintenance at follow-up. However, the evidence presented from students’ responses and feedback to the intervention confirm the importance of deploying user-centered methodology in the iterative design and refinement of these interventions before investing additional resources in conducting more rigorous efficacy testing.

Another limitation is related to our method of collecting objective data on users’ engagement and interaction with the chatbot intervention during the study. Since our log data were deidentified, it was more difficult to assess any difference among users in how deeply they focused attention and self-reflected upon the psychoeducational videos’ contents and chatbot suggestions during each session. Although we could derive some information on users’ satisfaction with these contents from participants’ qualitative comments, a more complete and objective monitoring of users’ behavioral interactions with the intervention’s components would be preferable to deploy in future studies.

Finally, the study was conducted during the second wave of the COVID-19 pandemic in Italy, characterized by the introduction of increasingly more rigid restrictions to social behavior and educational practices that might have strongly impacted the mental well-being of our participants and reduced the positive effect of our intervention. However, the challenging contextual setting in which our intervention was deployed can also be considered a point of strength of the contribution provided by this study, offering interesting insights for the future wider deployment of digital mental health interventions in challenging conditions.

Conclusions

This study further extends previous research on the use of chatbot-based interventions for healthy coping with stress, confirming their effectiveness in supporting university students experiencing higher levels of distress. Although the generalizability of the reported findings should be viewed with caution, since no control group was involved and the intervention was deployed during the COVID-19 pandemic, these preliminary findings are interesting for inspiring the future design of digital mental health interventions for university students and public health.