Most Influential Interventions
Each participant was asked to select which received intervention component(s) were most influential for behavior change. In order from most to least helpful, participants selected personalized feedback/tailored coaching (i.e., three-fourths of CIAN participants), the website (i.e., over one-third of all participants), diaries (i.e., almost one-third of CIAN and A + SM participants), and biosensors (i.e., less than one-fifth of all participants).
Suggestions for Future Interventions
During exit interviews, participants also offered suggestions for current and future intervention improvement. Many participants were interested in measuring additional sleep-related factors, most commonly that they Want diet and exercise-related feedback: “[I’d like to know] what time when you exercise, how it affects your sleep…what that does to…your whole body chemistry, everything.” Others mentioned they Want caffeine-related feedback, Want environment-related feedback (e.g., impacts of light, noise, and temperature on sleep), or Want other potential feedback (e.g., cognitive alertness, tobacco, and stress): “Maybe you can see how stressed you are during… the week. Just get those numbers, and then I think that would be pretty interesting to figure [out].” To receive additional feedback, most responded that they would be Willing to wear more devices and Willing to complete new diary questions. Fewer stated they were Unwilling to add a question and/or device, usually disinterest in another ankle biosensor. Several volunteered suggestions for received interventions in the current study. These included Improvements for feedback (e.g., combined, summary report for both intervention weeks, Improvements for diaries (e.g., higher pay per diary), Improvements for biosensors (e.g., eliminating ankle biosensors, measuring GPS location), and Improvements for website (e.g., wider distribution, alternative summaries).
Nonmetric Multidimensional Scaling of Interview Themes
We used multivariate analysis, nonmetric multidimensional scaling (NMDS)24, to visualize the interrelationships among qualitative themes and participants. NMDS is an ordination method that condenses variation in matrices, such as participant-by-theme frequency, to a small number of orthogonal dimensions25. We used five orthogonal dimensions based on stress-level testing, and the first two dimensions are plotted in vector space in Fig. 2. Also, multivariate correlational analysis allowed us to fit participant characteristics as factors to the NMDS ordination of themes. Trial group (R2 = .29, p = .001) and lifetime history of any MHD/SUD (R2 = .03, p = .04) were significantly correlated with NMDS scores. That is, participants’ statements about study interventions in their exit interviews were associated with trial group and diagnostic history. Other participant characteristics (e.g., age, gender, race, ethnicity, student status) were not associated with NMDS scores. In the NMDS shown in Fig. 3, CIAN and A + SM participants were closer to themes of feedback and diary helpfulness than website or biosensor helpfulness in vector space, which suggests their preference for feedback and diaries. Conversely, control A and A + SM participants were closer to themes of helpful website aspects in vector space than CIAN participants, and control participants were also significantly more likely to state these themes during interviews (X2 = 27.34, p < .001).
Natural Language Processing of Exit Interviews ( n = 112)
We also used quantitative Latent Dirichlet Allocation (LDA), a topic modeling analysis that identifies the likelihood of terms occurring in topics and topics occurring in documents26, with an expanded dataset of 112 participants’ exit interviews and coaching transcripts and identified nine topics across participants’ statements (see Fig. 3; see Supplemental Material for all topic definitions and example quotes). The goals of NLP analyses were to help qualitative thematic analysis be more targeted and assess convergence of findings with thematic analysis and exit survey analyses. These topics did not vary significantly by trial condition (X2 = 16.72, p = .40). We found that Awareness with monitoring was the most likely topic in the largest number of interviews (n = 18, 16%). This included participants’ perceptions that both active diary self-monitoring and passive biosensor monitoring increased their awareness and mindfulness of their behaviors. As one participant stated, “Knowing that I had to do the sleep diary the next day and…people were watching what I was doing…I was just more aware of my habits.”
Other common topics focused on web-based advice, especially sleep strategies. The topic, Website improves sleep (n = 16, 14%), included experiences of beginning to improve sleep using sleep strategies, such as consistent sleep schedules. A participant described, “My sleeping habits were definitely awful before now, and I think they've improved a lot, since starting this [study]…I'm starting to go to bed earlier and…reduce my activities before bed.” Almost as many interviews were most likely to include the topic, Strategy barriers (n = 15, 13%), or descriptions of challenges implementing sleep strategies, such as situational factors (e.g., work and school schedules, dormitory environment) or personal factors (e.g., memory, motivation). The topic Changing poor sleep (n = 15, 13%), included descriptions of poor sleep quality or nonrestorative sleep, how this motivated study participation, and a desire to learn about different sleep factors.
Other topics focused on participants’ interest in gaining more wellness-related knowledge to change sleep or alcohol use. Thirteen interviews (12%) were most likely to include the topic Learning and reduced drinking, which focused on gaining new information from digital interventions and reducing drinking given its impacts on sleep. The topic Using feedback and website (n = 11, 10%) included participants’ accounts of how they integrated personalized feedback and coaching with web-based advice, including information and strategies, to alter their sleep and alcohol use. The topic Multiple strategies and factors (n = 10, 9%) focused on participants’ curiosity and interest in the impacts of varied sleep-related factors, such as situations, environments, substances other than alcohol, and diet and exercise.
The two least frequent topics included the burden of wearable biosensor devices. The topic Strategies, not devices (n = 8, 7%) focused on helpful website aspects, attempts to implement sleep strategies, and challenges with wearable devices, especially the transdermal ankle monitor. Similarly, the topic Feedback, not devices (n = 6, 5%) focused on the benefits of personalized feedback and coaching while also discussing difficulties with wearable devices.
The sentiment scores of exit interviews and transcripts (n = 112) were generally positive (M = 15.07, SD = 10.54) and ranged from − 16 to 47 (see Fig. 4). Positive vs. negative sentiment scores reflect the aggregate valence of participants’ word usage in exit interviews, so “0” reflects a neutral net score in an interview. Sentiment scores did not vary as a function of trial condition (CIAN, A + SM, and A; F(2, 109) = 0.78, p = .46). Positive sentiment scores also did not vary by demographic characteristics, including gender (F(1, 110) = 0.62, p = .43), age (b = 0.03, t = 0.06, p = .95), race (F(4, 107) = 1.23, p = .30), ethnicity (F(1, 110) = 0.20, p = .66), student status (F(3, 108) = 0.75, p = .53), and history of any MHD/SUD (F(1, 110) = 0.23, p = .63). However, the word length of each document was positively correlated with sentiment scores (r = 0.21, p = .03), indicating a tendency for participants who spoke at longer length to speak more positively about their experiences. To boost rigor and account for this correlation, we derived a valence-per-word score for each document and reran sentiment comparisons across condition and demographic groups with the valence-per-word score as the outcome. There were still no between-group differences (p > 0.05).
Exit Survey ( n = 118)
On the exit survey, participants across conditions (CIAN, A + SM, and A) reported generally positive user experiences (n = 118; see Table 3). On a 5-point scale, participants reported high overall program satisfaction, website helpfulness, diary helpfulness, and feedback helpfulness. Mean feasibility ratings were generally high for the overall program, website, watch biosensor, and diaries. However, mean feasibility ratings for the ankle biosensor were lower based on visual inspection, especially due to interference with clothing, feeling uncomfortable, and being noticeable. Perceived effectiveness was higher among CIAN (Δ = 0.48, p = .008) and A + SM participants (Δ = 0.55, p < .001) than A participants (F(2, 115) = 8.45, p < .001). Further, young adults with a lifetime history of any MHD/SUD rated their intervention as more effective on average than those without diagnoses (F(1, 116) = 4.64, p < .001; Δ = 0.32, p = .03).
Table 3
CIAN Exit Survey Outcomes (N = 118)
Outcome (Agreement 1–5)
|
A
|
A + SM
|
CIAN
|
Total
|
|
M (SD)
|
M (SD)
|
M (SD)
|
M (SD)
|
General program outcomes
|
|
|
|
|
Overall satisfaction
|
4.45 (0.74)
|
4.48 (0.57)
|
4.60 (0.56)
|
4.53 (0.61)
|
Understandability
|
4.52 (0.87)
|
4.59 (0.83)
|
4.65 (0.66)
|
4.60 (0.75)
|
Schedule workability
|
4.66 (0.94)
|
4.62 (0.86)
|
4.67 (0.82)
|
4.65 (0.85)
|
Appropriate visit length
|
4.31 (1.11)
|
4.10 (1.14)
|
4.27 (0.92)
|
4.24 (1.02)
|
Lifestyle change promotion
|
4.03 (0.87)
|
4.10 (0.77)
|
4.23 (0.96)
|
4.15 (0.89)
|
Comfortability
|
4.76 (0.79)
|
4.66 (0.81)
|
4.57 (0.81)
|
4.64 (0.80)
|
Hope for change promotion
|
4.00 (0.89)
|
3.97 (0.91)
|
4.10 (1.02)
|
4.04 (0.96)
|
Importance of target habits
|
4.48 (0.87)
|
4.52 (0.69)
|
4.57 (0.81)
|
4.53 (0.79)
|
Overall effectiveness
|
4.00 (0.76)
|
4.48 (0.51)
|
4.55 (0.57)
|
4.40 (0.64)
|
Advice (A) outcomes
|
|
|
|
|
Module 1 helpfulness
|
4.24 (0.74)
|
4.24 (0.64)
|
4.43 (0.65)
|
4.34 (0.67)
|
Module 2 helpfulness
|
4.07 (0.96)
|
4.31 (0.66)
|
4.45 (0.65)
|
4.32 (0.75)
|
User-friendliness
|
4.52 (0.51)
|
4.41 (0.50)
|
4.40 (0.94)
|
4.43 (0.76)
|
Comparability to other
websites
|
4.76 (0.44)
|
4.52 (0.57)
|
4.73 (0.58)
|
4.68 (0.55)
|
Enjoyment
|
3.97 (0.91)
|
3.97 (0.73)
|
4.00 (0.77)
|
3.98 (0.79)
|
Passive-monitoring
(biosensor) outcomes
|
|
|
|
|
Actiwatch comfortable
|
3.93 (1.13)
|
4.14 (0.69)
|
3.92 (0.94)
|
3.97 (0.94)
|
Actiwatch not embarrassing
|
4.00 (1.16)
|
4.24 (0.64)
|
4.23 (0.79)
|
4.18 (0.86)
|
Actiwatch did not interfere
with work
|
4.34 (0.97)
|
4.55 (0.51)
|
4.52 (0.62)
|
4.48 (0.70)
|
Actiwatch did not interfere
with exercise
|
4.41 (0.78)
|
4.31 (0.66)
|
4.32 (0.91)
|
4.34 (0.82)
|
Actiwatch did not interfere
with sleep
|
4.41 (0.68)
|
4.34 (0.67)
|
4.30 (0.89)
|
4.34 (0.79)
|
Actiwatch did not interfere
with concentration
|
4.59 (0.57)
|
4.55 (0.57)
|
4.52 (0.60)
|
4.54 (0.58)
|
Actiwatch did not interfere
with clothing
|
4.41 (0.87)
|
4.21 (0.98)
|
4.18 (1.02)
|
4.25 (0.97)
|
Actiwatch not noticeable
|
4.07 (1.00)
|
4.17 (0.93)
|
4.17 (0.96)
|
4.14 (0.95)
|
Actiwatch did not change
routine
|
4.59 (0.57)
|
4.38 (0.82)
|
4.28 (1.01)
|
4.38 (0.88)
|
Actiwatch wear more
|
4.24 (0.87)
|
4.34 (0.81)
|
4.27 (0.92)
|
4.28 (0.88)
|
Ankle monitor comfortable
|
2.15 (0.95)
|
2.66 (1.11)
|
2.60 (1.08)
|
2.51 (1.07)
|
Ankle monitor not
embarrassing
|
2.59 (1.19)
|
2.66 (1.42)
|
2.98 (1.38)
|
2.81 (1.35)
|
Ankle monitor did not
interfere with work
|
3.37 (1.08)
|
3.79 (1.05)
|
4.05 (1.00)
|
3.83 (1.06)
|
Ankle monitor did not
interfere with exercise
|
2.48 (1.16)
|
3.03 (1.30)
|
3.02 (1.32)
|
2.90 (1.29)
|
Ankle monitor did not
interfere with sleep
|
3.04 (1.19)
|
3.34 (1.14)
|
3.25 (1.16)
|
3.22 (1.16)
|
Ankle mon. did not
Interfere with
concentration
|
3.56 (1.25)
|
3.97 (0.91)
|
4.05 (1.02)
|
3.91 (1.06)
|
Ankle monitor did not
interfere with clothing
|
2.33 (1.47)
|
2.17 (1.28)
|
2.55 (1.43)
|
2.41 (1.40)
|
Ankle monitor not
noticeable
|
2.85 (1.22)
|
2.66 (1.37)
|
2.60 (1.24)
|
2.67 (1.26)
|
Ankle monitor did not
change routine
|
3.11 (1.25)
|
3.21 (1.18)
|
3.10 (1.36)
|
3.12 (1.28)
|
Ankle monitor wear more
|
3.26 (1.53)
|
2.93 (1.25)
|
3.42 (1.37)
|
3.26 (1.38)
|
Self-monitoring (SM)
outcomes
|
|
|
|
|
Diary helpfulness
|
-
|
4.07 (0.72)
|
4.19 (0.79)
|
4.15 (0.76)
|
Diary easiness
|
-
|
4.34 (0.61)
|
4.58 (0.56)
|
4.51 (0.59)
|
Diary not burdensome
|
-
|
3.97 (1.09)
|
4.17 (0.91)
|
4.10 (0.97)
|
Diary did not change
schedule
|
-
|
4.24 (0.83)
|
4.27 (1.06)
|
4.26 (0.98)
|
Diary easy to remember
|
-
|
3.72 (1.16)
|
3.92 (1.00)
|
3.85 (1.05)
|
Diary enjoyable
|
-
|
3.52 (0.91)
|
3.83 (0.91)
|
3.73 (0.91)
|
Diary would continue
|
-
|
3.52 (1.15)
|
3.83 (0.92)
|
3.73 (1.01)
|
Feedback (F) outcomes
|
|
|
|
|
Sleep feedback helpfulness
|
-
|
-
|
4.70 (0.53)
|
4.70 (0.53)
|
Alcohol feedback
helpfulness
|
-
|
-
|
4.65 (0.55)
|
4.65 (0.55)
|
Note. “CIAN” stands for Call it a Night, the full digital sleep intervention, including personalized feedback and coaching, active diary self-monitoring, passive biosensor monitoring, and web-based advice. “A + SM” stands for advice plus self-monitoring control condition, which also includes passive monitoring. “A” stands for advice control condition, which also includes passive monitoring. |
Total means and standard deviations are based on 118 participants who completed the exit interview, including 60 CIAN, 29 A + SM, and 29 A participants. Some items had minor levels of missingness (< 5 participants). |
Adherence to interventions was also high across study phases. Almost all participants (98%) completed the two-week intervention phase, and 96% completed the 12-week follow-up appointment. Regarding monitoring activities, 98% of participants wore the sleep watch biosensor for 14 days, and 95% wore the alcohol ankle biosensor for 14 days. Further, 95% of participants in the CIAN and A + SM groups completed their assigned 14 days of active diary self-monitoring of sleep and alcohol use.