https://orcid.org/0000-0002-1949-3138
Figures
Abstract
Ecological Momentary Assessment (EMA) is an important methodology to understand risky behaviour and holds promise for HIV research. EMA is still novel in sub-Saharan Africa. We describe challenges and lessons learned on a novel study implementing mobile phone EMAs with adolescent boys in South Africa. The Tsamaisano study was a longitudinal study from 2020–2023 to recruit adolescent boys aged 15–19 years; including those without HIV and those perinatally infected and living with HIV. Participants were prompted to complete 52 weekly mobile phone survey on emotional state, exposure to and perpetration of violence, and sexual risk behaviour. Surveys were delivered using a random algorithm to choose the day. We incorporated mechanisms to assess challenges and optimize survey completion: weekly team meetings with youth representation and real-time data monitoring. Additionally, 20 frequent vs infrequent survey submitters participated in qualitative interviews about barriers and recommendations. Real-time monitoring indicated low (defined as <50%) survey completion in the first months of study implementation. To ensure that both the adolescent participant and their caregiver understood the commitment required for successful EMA, we created and implemented a guided discussion around mobile phone access during the enrolment visit. We identified a need for increased and ongoing technical support; addressed by creating technical guides, implementing a standard two-week check-in call after enrolment, adding an automated request button for call-back assistance, creating a WhatsApp messaging stream, and reaching out to all participants failing to submit two sequential surveys. Entry-level smartphones, including those initially distributed by the study, did not have capacity for certain updates and had to be replaced with more expensive models. Participants struggled with randomly allocated survey days; completion improved with set completion days and targeted reminder messages. Together, these steps improved survey completion from 40% in December 2020 to 65% in April 2022. We describe key lessons learned to inform future study designs with mobile phone EMAs, drawing on our experience implementing such among adolescent boys, including persons living with HIV, in a low-and-middle income setting. The key lessons learned through the Tsamaisano study are important to inform future study designs with EMA utilizing mobile phone, electronic data collection among adolescent boys in low-and-middle-income settings.
Author summary
In this study, participants completed 52 weekly mobile phone surveys on emotional state, exposure to and perpetration of violence, and sexual risk behaviour. Although our data are restricted to adolescent boys, lessons can be applied to youth in low-income South African settings. Our study started with randomly selected survey submission days and then switched to set weekly survey days. We incorporated youth engagement strategies through working closely with the Adolescent Community Advisory Board (ACAB) of the PHRU throughout the study implementation, including youth representation as of the study operations meetings.
Citation: Dietrich JJ, Hornschuh S, Madi P, Ramsammy CW, Tsotetsi L, Tshabalala G, et al. (2024) Implementing ecological momentary assessments to measure violence and adolescent HIV transmission risk: Lessons from Johannesburg, South Africa. PLOS Digit Health 3(2): e0000283. https://doi.org/10.1371/journal.pdig.0000283
Editor: Amelia Fiske, Technical University Munich, GERMANY
Received: May 26, 2023; Accepted: December 4, 2023; Published: February 2, 2024
Copyright: © 2024 Dietrich et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The qualitative datasets generated and/or analysed during the current study are not publicly available to maintain participant privacy and confidentiality, but are available through email request by contacting info@phru.co.za.
Funding: Research reported in this publication was supported by the National Institute of Mental Health under Award R01MH119878. The work reported herein for JJD was also made also made possible through funding by the South African Medical Research Council through its Division of Research Capacity Development under the Early Investigators Programme through funding received from the South African National Treasury (Award number: N/A). SH was supported by the Consortium for Advanced Research Training in Africa (CARTA). CARTA is jointly led by the African Population and Health Research Center and the University of the Witwatersrand and funded by the Carnegie Corporation of New York (Grant No. G-19-57145), Sida (Grant No:54100113), Uppsala Monitoring Center, Norwegian Agency for Development Cooperation (Norad), and by the Wellcome Trust [reference no. 107768/Z/15/Z] and the UK Foreign, Commonwealth & Development Office, with support from the Developing Excellence in Leadership, Training and Science in Africa (DELTAS Africa) programme. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Ecological Momentary Assessment (EMA) is an important tool to capture how risk behavior develops in real-time. EMA is typically characterized by repeat, self-report assessments that take place under natural conditions [1,2]. EMA thus limits recall bias and provides results that are more accurate to the subject’s environment [3]. In its original form, EMA methods involved participants recording their experiences in a paper-based diary format [1,4]. Increasingly, data capturing is moving towards technology-enabled platforms, a process that was accelerated during the COVID-19 pandemic [5]. Most people now have access to a mobile phone [6], which further provides unique opportunities for mobile data collection, such as updating the survey and allowing for monitoring of large volumes of incoming data in real time, [7–9].
EMA has been successfully implemented to study physical activity, mental health (mood disorders and substance abuse), eating behaviours, and chronic pain, most frequently among adult populations [10]. Among young people specifically, EMA studies have been conducted in high income settings to assess physical activity, substance abuse (particularly smoking) and mental health [10,11]. Risk behaviour assessments remain a challenge in HIV research. However, its use within the HIV field is growing. In the US, a recent literature review on EMA in HIV research identified EMA methods using smartphone apps for HIV primary prevention, HIV treatment adherence, and substance use among young adults and adult populations [12]. Notably, previous South African research utilising daily mobile phone EMA measures was conducted among young women 18–24 years to assess HIV risk and showed more accurate reporting via the mobile phone when compared to the in-clinic survey measure [13–16].
EMA designs hold great promise, but researchers are still learning how best to implement them to collect quality data [17,18]. For example, a recent review noted challenges, such as incompatible smartphones and poor network connectivity, that reduced compliance [10,18]. A systematic review evaluating the feasibility of using EMA for youth psychopathology found that the main barrier to success was overburdening participants with submissions. The authors recommended adapting the methods to the specific population group to encourage compliance [11]. For example, the survey questions should be clear and easily understandable to avoid misinterpretation and take the literacy levels of participants into account. Furthermore, suggestions to increase compliance included the use of phones that are compatible to the data collection program used [19].
EMA could yield important insights in the relationships between HIV and violence. Both are endemic in South Africa [20]: intimate partner violence (IPV) and other types of victimization are widely recognized as key drivers of HIV transmission [21–24]. Adolescent boys living with HIV and who are perinatally infected may be more vulnerable to violence. This is because many have lost a parent to AIDS, which increases the likelihood of childhood sexual abuse by approximately 50% [25] and also HIV disclosure can trigger violent reactions from partners or peers [26]. Engagement in violence can trigger risky sexual behaviours among young men including multiple partners, condom use, drugs and alcohol abuse and transactional sex [27].
We designed the Tsamaisano study to better understand how violence shapes risk behaviors among adolescent boys in South Africa, utilizing a mobile EMA approach. As we were one of the first studies to use EMA to measuring sensitive topics among adolescents in this setting, we reviewed the literature for best practices. However, EMA is relatively novel in sub-Saharan African settings, and there was limited guidance on how to develop and implement EMA methods in these unique contexts. Within the main study, we thus embedded processes that would allow for continuous quality improvement. The aim of this paper is to describe the lessons learned with recommendations for smoother EMA data collection. This might benefit other researchers who intend to employ similar strategies with populations, such as adolescent boys, that are usually hard to engage in clinic and health settings [28].
Methods
Tsamaisano study
The Tsamaisano study was a prospective longitudinal study (2020–2024) to understand how violence influences HIV transmission risk. Participants included adolescent boys aged 15–19 years; those without HIV and persons living with HIV through perinatal infection. The study took place in Soweto, South Africa, a densely populated and large urban settlement or township of about 1.7 million inhabitants [29]. Eligible participants are: male; 15–19 years at baseline; living in Soweto; and either in a current dating relationship with someone they see face-to-face or have had sex in the last month. Inclusion criteria for persons living with HIV included perinatal HIV infection (defined as having had a history of HIV infection before age 10 years) and awareness of their HIV diagnosis. Participant exclusion criteria included acute psychiatric illness or cognitive impairment rendering them unable to perform the weekly surveys.
Following recruitment (in the community and in HIV clinics), prospective participants were invited to an in-person enrolment visit. The visit was conducted at the Perinatal HIV Research Unit (PHRU) [30], located at the Chris Hani Baragwanath Academic Hospital. At this visit participants completed an interactive survey training session. Participants were told they would get a text request to complete a survey on their mobile phones once a week for 52 weeks. The mobile phone survey, hereafter referred to as the survey, was designed within the SurveyCTO application (app) [31] an online survey platform, and data were managed by ikapadata [32], a local South African digital health company.
Initial literature review
Prior to study initiation, a brief literature search was conducted to review strategies shown to help retain youth in mobile phone-based research, including but not limited to EMA. Previous research among youth in Soweto showed more than 90% of young people aged 14–19 years owned a mobile phone, and 65% had access to the internet [33,34]. Therefore, we planned to provide entry level android smartphones to 20% of participants enrolled in the study. Previous qualitative research on mobile phone-based data collection conducted among youth in Soweto indicated a preference for short surveys that were easy to answer, with mobile phone data provided to participants to access the applications for survey completion, followed by an airtime incentive, a voucher that could be redeemed immediately for mobile phone airtime, after survey submission [13–15]. Additionally, reminder messages were recommended for repeated EMA mobile phone based surveys [14].
Patient and Public Involvement: Youth Engagement before the study start
To ensure a youth engaged strategy, the study team worked closely with the Adolescent Community Advisory Board (ACAB) of the PHRU throughout the study implementation. The ACAB is an important bridge back to the community. We hosted a workshop at the beginning of the project to inform them of study details to get feedback on study development. The ACAB is comprised of approximately 16 youth, 16–23 years and from different organisations and schools in and around Soweto. The members of the ACAB were consulted and involved in the formulations of the initial survey questions. This was to ensure that questions were developmentally and culturally appropriate for male adolescents, understandable in English and translated appropriately to the local languages (isiZulu and Sesotho). The ACAB advised on recruitment strategies and continue to advise the study team throughout the study period.
The survey was pilot tested among 10 adolescent boys 15–19 years, including those living with HIV and those living without HIV infection. Potential participants were recruited from Soweto and signed informed consent to participate in weekly surveys over a four week period. Following their participation, all pilot participants were interviewed telephonically for feedback on their experiences and suggestions to improve implementation.
Survey structure and training with participants
The survey (See Fig 1) included 23 and 25 questions for those living without HIV infection and for persons living with HIV, respectively. The survey included items on mental health (depression, anxiety), alcohol use, physical and sexual violence exposure, intimate partner violence (IPV) perpetration [35–37], sexual activity, condom use. Additional questions on antiretroviral treatment adherence were included for persons living with HIV. Questions on COVID-19 (for example, symptoms, and testing) were also included once a month.
Entry-level smartphones were provided to participants who did not have their own phones or those whose phones did not meet the requirements to download and run the SurveyCTO app [31]. Participants could keep these study phones upon study completion. During study enrolment, mobile phone educators helped participants install the SurveyCTO app on their phones and demonstrated how to complete the survey. Participants were provided with a printed manual on the SurveyCTO app set up and survey completion. In addition, a once-off automated SMS with their individual PIN to access the survey and a link to the survey manual was sent to the participant’s phone. Both the survey and the manual are available in three languages, English, isiZulu and Sesotho.
Surveys started the Monday following study enrolment. Data to the value of 100MB is preloaded on the phones in the afternoon (15:00) of the day a survey request is made through an online app programming interface SIM control [38]. The preloaded data was to ensure that participants would (1) not incur any costs and (2) have sufficient data to be able to submit their survey. An automated survey request SMS was initiated at 15:00 on the day of the participant’s expected submissions day; a reminder SMS followed at 19:00 that day as well as at 15:00 the following day, if the survey was not submitted. Different survey request and reminder SMS wordings were generated in collaboration with the ACAB and were randomly sent out for each time point. Following successful survey submission, an automated confirmation SMS was sent. Participants received a 10ZAR (~70 US cents) airtime voucher incentive if submitted within a two-day window after the initial request. Participants were contacted by the mobile phone educators telephonically and/or via WhatsApp after completing the first survey to troubleshoot any technical problems.
Continuous quality improvement strategies
Weekly team meetings were held to review study progress review survey completion rates and to discuss challenges of implementation. At the time, the EMA data collection was a new data collection technique incorporating sensitive questions. Thus, the monitoring of survey completion rates during the team meetings and a rapid qualitative evaluation were critical to establish barriers to survey completion. Following a text message prompt to participants, a successful survey submission was defined uploading the survey in the SurveyCTO app on the same day or within a 2-day period.
The qualitative evaluation included in-depth interviews with 20 of the Tsamaisano participants attending a follow-up visit 3–6 months after enrolment. The sample size was sufficient to reach data saturation [13,15]. A Tsamaisano staff member received a list of pre-selected participants based on the month of enrolment, survey performance, age group and HIV status. Interviewers conducted a 30–60 minute in-depth interview in a private room at the PHRU.
Participants signed a separate informed consent or assent form for participation in the interviews. Participants were able to select a trained male or female multilingual interviewer. A semi-structured interview guide addressed topics on overall study participation experience, barriers and facilitators to survey completion, and recommendations for improving the survey implementation. All interviews were audio-recorded, transcribed to English, and uploaded in Nvivo [19]. Data were coded and analysed by three coders using a framework data analysis approach [20,21]. First transcripts were read for data familiarisation, then the same initial coder developed a codebook a-priori for barriers to survey completion and recommendations for improvement. Participant demographic characteristics included age, work and income, mobile phone ownership and use, and internet use. Descriptive statistics were performed to calculate frequencies and percentages by HIV status using Stata version 17 (Stata Corp).
Ethical considerations
All study procedures were approved by the Institute Research and Ethics Committees of the University of the Witwatersrand, South Africa (Approval ID: 191001) and Stony Brook University, United States (Approval ID: FWA#00000125 and IRB2019-00567). Written study informed consent and assent for participants younger than 18 years were obtained from all participants. Parents/legal guardians provided written informed consent for participants younger than 18 years. Participants and parent/legal guardian signed separate forms for audio-recording for in-depth interviews. All methods were carried out in accordance with relevant guidelines and regulations. All changes made to improve survey submission rates were approved by Ethics Committees of the University of the Witwatersrand and Stony Brook University. Participants were reimbursed R250 (~ $16) for scheduled visits, which for some included qualitative interviews. A PHRU counsellor was available for those requiring psychosocial support.
Results
Survey completion
Enrolment started in November 2020, prior to the second COVID-19 wave in South Africa. From November 2020—April 2022, 468 participants were enrolled; 241 persons living with HIV (median age was 16 (15–19)), 227 without HIV (16 (15–19)) (Table 1). Fig 2 shows overall survey performance. We define successful completion as submitting a survey via the SurveyCTO app with 48 hours of the request or after of a given week and transmission thereof into the study database. Since study start, the study team closely monitored participants’ survey submission rates and identified that the submission rates were below 50%. Our continuous quality improvement process helped increase survey response rates from 40% in December 2020 to 65% by April 2022.
Lessons learned
1) Studies need to provide most participants with smartphones. These cannot be entry-level phones given the technological requirements of the survey application.
Study phones were provided to participants who did not have a smartphone or did not own a phone that met the minimum software and hardware requirements to run the SurveyCTO app. The study team quickly realised that although participants owned or had access to a mobile phone, many young men did not own an adequate smartphone. Study phone allocation, for the period December 2020 to April 2022, was 87% (see Fig 3). Of 468 participants enrolled at this time, 410 required study phones. In addition, participants were provided with a replacement study phone when the personal or initial study phone had been lost, damaged, stolen, the app never functioned or the phone stopped working. The study originally purchased entry level Android smartphones that met the basic requirements for operating the SurveyCTO app with the survey form. However, the phones required many system updates over time, which slowed down the set-up of the phone, the SurveyCTO app, and survey submission. Therefore, the study team purchased better quality phones with updated Android operating systems and tested different smartphone brands to ensure adequate operation. Finally, the mobile phone network connectivity of the participants was reviewed during enrollment. If it was not adequate, a SIM card was purchased through a mobile phone provider network with better connectivity.
2) Participants require ongoing, multimodal technological support.
Participants encountered a myriad of technical challenges preventing them from submitting surveys. Participants communicated these through direct requests with the mobile phone educators, in follow-up calls with first time and non-submitters, and during the qualitative interviews. The main support requests included: forgetting their app pin and/or individual survey passcode, challenges with using the SurveyCTO app, lack of sufficient data (provided by the study) and poor network connection to submit the weekly survey through the SurveyCTO app, phone issues (i.e. phone being slow, freezing, or broken), and reporting a lost phone. Some participants stated:
“The person that explained, explained it very well but when I opened my phone…I struggled with the pin thing…So I entered it at the end and then I placed the pin. There was a problem because I did not [find] my paper where they wrote the pin for me.” (IDI-02-16 years)
“The app [CTO app] was troublesome, when I would try to send the [survey] form. It would just stay there and not be sent.” (IDI-03-16 years)
Interviewer: “Why do you think some participants don’t always respond to the surveys, like they don’t answer the surveys?”
Participant: “Because lack of… lack of … when there is no network.” (IDI-016-18 years)
In response to participant challenges, the SurveyCTO app and survey completion training was intensified, and more time was taken during the enrollment visit to ensure that participants understood each step of the survey submission process. WhatsApp was also downloaded onto the phone at enrollment for receiving quick study support. WhatsApp was chosen due to the popularity and frequency of use among South African youth. As a last step during the training component, the mobile phone educators saved important study numbers, including the study support WhatsApp number.
From the beginning of the study, telephonic check-in and support was provided to first time submitters after the enrolment visit as a quality assurance measure, to identify and resolve any challenges early, and to provide additional SurveyCTO app survey training where required. In addition, when real-time data monitoring revealed low survey submission rates, we began calling to check in with all participants who had not submitted two or more surveys consecutively. During these calls, technical support was provided, and training points were identified and resolved to encourage survey submission.
The original survey included a counsellor support request button at the end. However, many participants mis-used this to access technical support. Thus, a separate technical support request button was added at the end of survey form. This initiated request sent an SMS to the study technical support phone, and prompted a call from the staff. After implementing the technical support button, staff noted a high volume of technical requests that were placed by mistake. Thus, a confirmation button for technical support was then added, which streamlined the requests and reduced errors.
3) Adolescent participants respond better to peer communication via WhatsApp or text, as compared to phone calls from staff.
In the process of improving our various technical support strategies as well as improving the response rates of the surveys, a male peer-educator to advise on study processes and oversee participant engagements activities (e.g., WhatsApp broadcast messages and content) was recruited. This peer-educator was well placed to respond to additional requests voiced by participants during the in-depth interviews: these included receiving regular study check-ins, study updates, reminders on the weekly survey submission process, and reminders of the support available. We found that the best way for the peer-educator to communicate was by using the anonymous WhatsApp broadcast messaging service. In addition, community engagement was intensified by including a young male CAB member on the bi-weekly operational Tsamaisano team calls to get insights and feedback on study related updates.
4) Facilitating a guided discussion around mobile phone access and survey completion during the enrolment visit can help ensure participants fully understand the commitment required.
There was a substantial number of participants with very low to no survey completion, despite the above efforts. Thus, a greater attempt was made to ensure enrollment of ‘best fit’ participants: adolescents had to understand and express commitment to the mobile component of the study as well as report sufficient mobile phone usage (an indicator they would likely submit surveys). To ensure that both the adolescent participant and their caregiver understood the commitment required for successful EMA, we created and implemented a guided discussion around mobile phone access during the enrolment visit. The team recommended that it was critical to involve the caregiver, as they often control access their child’s mobile phone, particularly when disciplining their child. The outcome of the discussion was an agreement between the caregiver and the child that facilitated survey submissions, such as when participants should check their phones for survey requests, and how punishments should be managed to allow for submissions. Moreover, some adolescents shared a phone with other family members, which could make it difficult to track survey request and reminder SMSs and submit on time. Through this more nuanced discussion, staff could assess whether a dedicated study phone should be provided. Survey completion commitment was reiterated throughout the enrolment visit and through a written commitment. These additions to the eligibility criteria for study participation were reviewed and approved by the institutional review boards at University of the Witwatersrand and Stony Brook University
5) Participants’ non-responses and preferring a fixed survey submission day to a random submission day.
During the interview feedback sessions, participants reported reasons for not submitting surveys: it was boring to answer the same weekly questions; they found it difficult to answer sensitive questions about violence and sex; and they simply forgot to submit surveys. Some participants stated: “I can say some find it boring … so that is why they answer late or maybe just leave it like that [not answer the survey at all].” (IDI-02-16 years)
“Maybe another reason can be the same questions every week.” (IDI-011-17 years)
In response to questions about random versus set submission days, participants stated that it would be easier to remember a fixed weekly submission day. They stated a fixed day would allow them to build a routine over time. For example some reported:
“It’s Friday [Best day to receive the survey] …Yes, I knock off early at school so I have plenty of time…5pm…Because that is when I am no longer doing anything, I am just busy with my phone or watching TV or I could just go out. (IDI-07-19 years)
“It is better Thursday, so it is better that Thursday I didn’t answer the survey [In case he forgot to answer the survey on Thursday]. Friday I know that I am home, they call me, they find me they remind me then I do it. (IDI-03-16 years)
To ensure that the study still had adequate coverage from all days of the week, we took the following steps starting in July 2021: one half was assigned to a randomly selected set week day and the other to a randomly selected set weekend day. After six months, those who were randomly assigned to a weekday for the first six months were then assigned a weekend day for the second six months and vice versa (see Fig 4). This assured adequate coverage of all seven days for the study, while still enabling the participants to establish an individual routine around a set survey day.
Participants also suggested reminder messages be changed, and suggested times that they would be most likely see them. The excerpts below illustrate participant suggestions:
“Maybe I would want it [reminder SMSs] to say, ‘You have less than an hour to submit the survey.’ …Or ‘One day left to submit.’ Or ‘Don’t forget to submit the survey today.” (IDI-09-19 years)
“It will be easy if they maybe change it [SMS] and say: ‘It’s time to answer’…”(IDI-010-16 years)
The SMS times were changed from receiving the survey request SMS in the afternoon to receiving the request SMS in the early morning (i.e. 6:00), followed by reminders in both the same day afternoon (17:00) and the following day afternoon (17:00). The wording for the SMSs were revised on participant request to be more instructive on when surveys had to be submitted to still be eligible for the R10 airtime incentive. Additionally, the survey request SMS also informed participants that their data bundle had been loaded. In feedback sessions, participants noted that receiving data prior to the survey completion and a small airtime incentive after successfully submitting a survey record were motivators to continue submitting surveys. When asked about alternative incentives, most said they were satisfied with the current arrangement.
6) A clear strategy for delivering data is needed to enable survey submissions, particularly among low-income youth.
Submission of surveys requires data. Among South African young people from low-income settings, unlimited data plans are rare. In our context, the majority owned mobile phones requiring pre-paid airtime and data and the study phones require the same. At the beginning of the study, 100 MB of data was automatically sent to participants together with the first survey submission request. As noted earlier, the SMS survey request timing was changed to the early mornings to correspond to the participants wake up time. This was to ensure that participants had a full first day to submit their surveys. However, in both instances, participants often used their allocated data volume prior to submitting their survey, which resulted in non-submission or completed surveys being stuck in the outbox of the SurveyCTO app as they could not be successfully transmitted without mobile phone data. In response, the 100 MB (~$1.5) data allocation was split to send in the morning and the remainder data allocation in the afternoon. This increased the likelihood that participants would have enough mobile data to successfully submit their survey if they chose to do it in the afternoon.
7) Studies need to be prepared for the large volume of psychosocial support.
Importantly, participants mentioned that they appreciated that the survey was anonymous and therefore they could answer sensitive questions, including those around violence and sex, without judgments by a study staff member. However, some participants reported that it was difficult answering sensitive questions every week, especially when it was connected to one’s own recent experiences or that of a close family or community member. This underscores the need for robust psychosocial support. Below are quotes stating participants reports on answering the surveys.
“This survey is thought provoking because there are questions which made me realise and think that things are happening to people… You see but I would have a question on how they solve this problem and I realise that I think this is the best place to solve issues like physical abuse.” (IDI-07-19 years)
“The questions for me are clear but it is up to me how I respond to them know. It becomes difficult to answer them. You find that the question is asking something that you have experienced and you find that the question takes you back to that space … Because I think about all the things that I used to witness when I was still young.” (IDI-07-19 years)
For all participants enrolled in the surveys, support was available 24 hours a day. At the end of the survey, there was a button that the participant could use to request psychosocial support. Once initiated, an automated SMS was sent with the local contact telephone numbers for service organizations, including Childline [39] and the South African Depression and Anxiety Group [40]. Additionally, an automated SMS was sent to the social worker or psychologist on call. They would contact the participant to debrief and/or arrange services. Finally, a question within the survey on being physically hurt or being sexually assaulted was programmed to trigger an automated notification to the study team, promoting a counsellor or social worker to reach out. If a participant revealed that they were in immediate danger or a danger to others, they were brought into the clinic for urgent support, referred to any necessary additional services, and the South African PI was notified. While we anticipated the need for these services, we did not anticipate the volume of need. Many participants utilized the counsellor request button within the survey. Since the separate technical support button was implemented from the week of 22 February 2021, on average, of the 9502 submitted surveys, 256 (2.7%) requested a counsellor, 712 (7.5%) were flagged to have reported being physically hurt and/or sexually assaulted and 1179 (12.4%) requested technical support.
Table 2 provides an overview of barriers to survey completion and recommendations with illustrative key quotes from IDI participants.
Discussion
We describe key lessons learned to inform future study designs with mobile phone EMAs, drawing on our experience implementing such among adolescent boys in a low-and-middle income setting. Our lessons point to the WHO youth-centered digital health interventions framework, to ensure that users are included throughout the research life cycle by leveraging youth-adult partnerships [41]. More than half of the participants required study phones, which was more than was expected, but perhaps is a more realistic picture among participants from lower socio-economic settings. The high volume of technical support requests included common problems that could arise in any setting (e.g., forgetting pins and passwords, pressing the technical request button in error) as well as unique barriers specific to those from low-socio-economic settings (e.g. owning lower end android phones, poor network connectivity, low data access).
Mobile and remote data collection have proven to be feasible but are also not without challenges. A review of low and middle-income countries revealed that the major challenges included: network shortcomings, the need to constantly train users, and the collection of sensitive information [15,42]. Furthermore, a recent Soweto study with similar data collection methods revealed other important limitations of using mobile phones; there were challenges around phones getting lost or breaking during the duration of the study, which resulted in the issuing of new phones [14,15]. Similar challenges were observed during our study, which had to be addressed and strategies developed in an ongoing process. However, many studies evaluating mobile-technology based EMA methods, including with young people, do not specifically report on technical and logistical challenges [18]. This study adds value to the existing literature around using mobile EMA amongst young people from low socio-economic backgrounds, as we describe both technical and logistical challenges since study inception as well as adaptations performed.
Young men in our study reported response fatigue to the same weekly questions as one of the reasons for non-response of the surveys. Response fatigue has been observed when collecting survey data and can stem from various reasons, such as completion length, questions topics, survey questions complexity and type [43]. Short surveys with only a few questions can reduce respondent fatigue [14]. Even though the survey in our study is short and simple, some participants still reported boredom with answering the same weekly survey. For future studies incorporating EMA could explore youth-friendly design to ensure study engagement over time.
In our study, participants expressed a preference for set weekly survey days versus randomly selected days. They stated it would help them better fit the survey completion into their schedule and be able to add their own individual reminders over and above the study SMS reminders. In another study conducted in Soweto, young women who completed a daily survey over a period of 90 days, with an overall response rate of 82%, also reported implementing their own reminder strategies. This included setting an additional alarm on their phones or building the daily survey completion into their morning routine [14].
Communication with participants was important for multiple aspects of study implementation, including survey completion, technical support, retention and engagement. Initially, phone calls and text messages were used as the primary medium of communication. However, WhatsApp was a more successful strategy to increase communication between the study site and participants. In South Africa, WhatsApp is the most popular messaging application used for socialization, business, educational purposes and health interventions. In a population of 58.93 million people, 38 million South Africans are active WhatsApp users [44]. The General Data Protection Regulation stipulates the minimum age for WhatsApp use in South Africa is 13 years [45]. The 2015 student tech survey showed that 92% of South African students were WhatsApp users [46]. Through Tsamaisano, we started a WhatsApp broadcast group, to transmit important news, updates, and check-ins with the aim to increase study engagement and improve survey submission performance. The WhatsApp broadcast function ensured privacy because those receiving the broadcast messages cannot see the contact details of those receiving the messages.
Limitations
Although the data for the interviews can not necessarily be generalized to all the Tsamaisano participants, the data are in agreement with previous research conducted in Soweto and data saturation was reached [13,14]. Further, our data are restricted to adolescent boys, lessons can be applied to youth in low-income South African settings. Although many the findings are applicable to mobile phone EMA studies in general, some of the technical challenges related to are specific to the SurveyCTO app. Our study did not apply a specific youth-centred framework but included youth engagement strategies. Future EMA studies involving reporting of sensitive information could incorporate a youth-centred framework upfront to guide and inform the implementation of the EMA design [47]. Our lessons point to the WHO youth-centered digital health interventions framework, to ensure that users are included throughout the research life cycle by leveraging youth-adult partnerships.
Conclusions
We describe key lessons learned to inform future study designs with mobile phone EMAs, drawing on our experience implementing such among adolescent boys, including persons living with HIV, in a low-and-middle income setting. The key lessons learned through the Tsamaisano study are important to inform future study designs with EMA using mobile phones, electronic data collection among adolescent boys in low-and-middle-income settings. Participants expressed a preference for set weekly survey days versus randomly selected days. We identified a need for increased and ongoing technical support; addressed by creating technical guides, implementing a standard two-week check-in call after enrolment, adding an automated request button for call-back assistance, creating a WhatsApp messaging stream, and proactively reaching out to all participants failing to submit two sequential surveys.
Acknowledgments
We would like to acknowledge all the study participants and their parents for the time and information they shared with us. We thank the research field team and interviewers, including Campion Zharima.
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health, the South African Medical Research, or the other funders.
References
- 1. Laurenceau J, Bolger N. Using diary methods to study marital and family processes. Journal of Family Psychology. 2005;19(1):86–97. https://doi.org/10.1037/0893-3200.19.1.86.
- 2. Doherty K, Balaskas A, Doherty G. The design of ecological momentary assessment technologies Interacting with Computers. 2020;32(3):257–78. https://doi.org/10.1093/iwcomp/iwaa019.
- 3. Wen C, Schneider S, Stone A, Spruijt-Metz D. Compliance with mobile ecological momentary assessment protocols in children and adolescents: a systematic review and meta-analysis. Journal of Medical Internet Research. 2017;19(4):1–14. https://doi.org/10.2196/jmir.6641.
- 4.
United Nations. Technology and Innovation Report 2021. Geneva, Switzerland: United Nations, 2021. Available from: https://unctad.org/system/files/official-document/tir2020_en.pdf.
- 5. Jandoo T. WHO guidance for digital health: What it means for researchers. Digit Health. 2020;6:2055207619898984.
- 6. Silver R. Smartphone ownership is growing rapidly around the world, but not always equally. 2019 [cited 2022 07 Mar]. Available from: https://www.pewresearch.org/global/2019/02/05/smartphone-ownership-is-growing-rapidly-around-the-world-but-not-always-equally/.
- 7. Curran K, Mugo NR, Kurth A, Ngure K, Heffron R, Donnell D, et al. Daily short message service surveys to measure sexual behavior and pre-exposure prophylaxis use among Kenyan men and women. AIDS and Behavior. 2013;17(9):2977–85.
- 8. Farmer S, Mindry D, Comulada WS, Swendeman D. Mobile phone ecological momentary assessment of daily stressors among people living with HIV: elucidating factors underlying health-related challenges in daily routines. Journal of the Association of Nurses in AIDS Care. 2017;28(5):737–51.
- 9. Weinhardt L, Forsyth A, Carey M, Jaworski B, Durant L. Reliability and validity of self-report measures of HIV-related sexual behavior: progress since 1990 and recommendations for research and practice. Archives of Sexual Behavior. 1998;27(2):155–80.
- 10. Burke LE, Shiffman S, Music E, Styn MA, Kriska A, Smailagic A, et al. Ecological momentary assessment in behavioral research: addressing technological and human participant challenges. Journal of Medical Internet Research. 2017;19(3):e77–e.
- 11. Thunnissen MR, Aan Het Rot M, van den Hoofdakker BJ, Nauta MH. Youth psychopathology in daily life: systematically reviewed characteristics and potentials of ecological momentary assessment applications. Child Psychiatry and Human Development. 2021.
- 12. Smiley SL, Milburn NG, Nyhan K, Taggart T. A systematic review of recent methodological approaches for using ecological momentary assessment to examine outcomes in U.S. based HIV research. Current HIV/AIDS Reports. 2020;17(4):333–42.
- 13. Dietrich JJ, Benade GL, Mulaudzi M, Kagee A, Hornschuh S, Makhale LM, et al. "You are on the right track with the app:" qualitative analysis of mobile phone use and user feedback regarding mobile phone sexual risk assessments for HIV prevention research. Front Digit Health. 2021;3:576514.
- 14. Dietrich JJ, Hornschuh S, Khunwane M, Makhale LM, Otwombe K, Morgan C, et al. A mixed methods investigation of implementation barriers and facilitators to a daily mobile phone sexual risk assessment for young women in Soweto, South Africa. PloS One. 2020;15(4):e0231086.
- 15. Dietrich JJ, Lazarus E, Andrasik M, Hornschuh S, Otwombe K, Morgan C, et al. Mobile phone questionnaires for sexual risk data collection among young women in Soweto, South Africa. AIDS and Behavior. 2018;22(7):2312–21.
- 16. Lemos MP, Lazarus E, Isaacs A, Dietrich J, Morgan C, Huang Y, et al. Daily vaginal swabs and mobile phone sex report for assessing HIV virion exposure prospectively among a cohort of young sexually active women in South Africa (HVTN 915). Journal of Acquired Immune Deficiency Syndromes. 2019;81(2):e39–e48.
- 17. Shiffman S, Stone A, Hufford M. Ecological Momentary Assessment. Annual Review of Clinical Psychology. 2008;4(1):1–32. https://doi.org/10.1146/annurev.clinpsy.3.022806.091415.
- 18. Heron KE, Everhart RS, McHale SM, Smyth JM. Using mobile-technology-based ecological momentary assessment (EMA) methods with youth: a systematic review and recommendations. Journal of Pediatric Psychology. 2017;42(10):1087–107.
- 19. Tyler KA, Olson K. Examining the feasibility of ecological momentary assessment using short message service surveying with homeless youth: lessons learned. Field Methods. 2018;30(2):91–104.
- 20. Abdool Karim Q, Baxter C. The dual burden of gender-based violence and HIV in adolescent girls and young women in South Africa. South African Medical Journal. 2016;106(12):1151–3.
- 21.
WHO. Violence Against Women and HIV/AIDS: Critical Intersections Intimate Partner Violence and HIV/AIDS. Geneva, Switzerland: World Health Organization, 2004. Available from: https://data.unaids.org/gcwa/gcwa_fs_vaw2_who_01may05_en.pdf.
- 22. Kuchukhidze S, Panagiotoglou D, Boily MC, Diabaté S, Eaton JW, Mbofana F, et al. The effects of intimate partner violence on women’s risk of HIV acquisition and engagement in the HIV treatment and care cascade: a pooled analysis of nationally representative surveys in sub-Saharan Africa. Lancet HIV. 2023;10(2):e107–e17.
- 23. Otwombe KN, Dietrich J, Sikkema KJ, Coetzee J, Hopkins J, Laher F, et al. Exposure to and experiences of violence among adolescents in lower socio-economic groups in Johannesburg, South Africa. BMC Public Health 2015.
- 24. Meinck F, Cluver LD, Boyes ME, Loening-Voysey H. Physical, emotional and sexual adolescent abuse victimisation in South Africa: prevalence, incidence, perpetrators and locations. Journal of Epidemiology and Community Health. 2016;70(9):910–6.
- 25. Kidman R, Palermo T. The relationship between parental presence and child sexual violence: Evidence from thirteen countries in sub-Saharan Africa. Child Abuse and Neglect. 2016;51:172–80.
- 26. Medley A, Garcia-Moreno C, McGill S, Maman S. Rates, barriers and outcomes of HIV serostatus disclosure among women in developing countries: implications for prevention of mother-to-child transmission programmes. Bulletin of the World Health Organization. 2004;82(4):299–307.
- 27. Okafor CN, Christodoulou J, Bantjes J, Qondela T, Stewart J, Shoptaw S, et al. Understanding HIV Risk Behaviors Among Young Men in South Africa: A Syndemic Approach. AIDS and Behavior. 2018;22(12):3962–70.
- 28. Otwombe K, Dietrich J, Laher F, Hornschuh S, Nkala B, Chimoyi L, et al. Health-seeking behaviours by gender among adolescents in Soweto, South Africa. Global Health Action. 2015;8(1):25670.
- 29. Statistica. Largest cities in South Africa in 2021, by number of inhabitants (in 1000s). 2021 [cited 2022 26 February]. Available from: https://www.statista.com/statistics/1127496/largest-cities-in-south-africa/.
- 30. Perinatal HIV Research Unit. About Us. 2023 [cited 2023 30 October]. Available from: https://www.phru.co.za.
- 31. SurveyCTO. 2022 [cited 2022 26 February]. Available from: https://www.surveycto.com.
- 32. ikapadata. Our services. 2022 [cited 2022 08 February]. Available from: http://ikapadata.com.
- 33. Dietrich J, Coetzee J, Otwombe K, Mdanda S, Nkala B, Makongoza M, et al. Adolescent-friendly technologies as potential adjuncts for health promotion. Health Education. 2014;114(4):304–18.
- 34. Dietrich JJ, Laher F, Hornschuh S, Nkala B, Chimoyi L, Otwombe K, et al. Investigating sociodemographic factors and HIV risk behaviors associated with social networking among adolescents in Soweto, South Africa: a cross-sectional survey. JMIR Public Health Surveill. 2016;2(2):e154.
- 35. WHO. Adverse Childhood Experiences International Questionnaire (ACE-IQ). 2020 [cited 2023 30 October]. Available from: https://www.who.int/publications/m/item/adverse-childhood-experiences-international-questionnaire-(ace-iq).
- 36. Koss MP, Abbey A, Campbell R, Cook S, Norris J, Testa M, et al. Revising the SES: A Collaborative Process to Improve Assessment of Sexual Aggression and Victimization. Psychology of Women Quarterly. 2007;31(4):357–70.
- 37.
WHO. WHO multi-country study on women’s health and domestic violence against women: initial results on prevalence, health outcomes and women’s responses. Geneva, Switzerland: World Health Organization, 2005. Available from: https://www.who.int/publications/i/item/9241593512.
- 38. Simcontrol. 2022 [cited 2022 26 February]. Available from: www.simcontrol.co.za.
- 39. Childline South Africa. About. 2022 [cited 2022 16 May]. Available from: https://www.childlinesa.org.za/about/how-we-help/crisis-line/.
- 40. South African Depression and Anxiety Group. About SADAG. 2022 [cited 2022 16 May]. Available from: https://www.sadag.org/index.php?option=com_content&view=article&id=2022&Itemid=138.
- 41.
WHO. Youth-centred digital health interventions: a framework for planning, developing and implementing solutions with and for young people. Geneva, Switzerland: WHO, 2020. Available from: https://www.who.int/publications/i/item/9789240011717.
- 42. Muthiah G, Prashant S, Jhunjhunwala A. A review on challenges in implementing mobile phone based data collection in developing countries. The Journal of Health Informatics in Developing Countries. 2011;5:366–74.
- 43. O’Reilly-Shah VN. Factors influencing healthcare provider respondent fatigue answering a globally administered in-app survey. PeerJ. 2017;5:e3785.
- 44. Ceci L. Most popular mobile apps used in South Africa as of February 2020, by reach. 2021 [cited 2022 08 February]. Available from: https://www.statista.com/statistics/1103151/most-popular-mobile-apps-south-africa/#:~:text=Most%20popular%20mobile%20apps%20in%20South%20Africa%202020%2C%20by%20reach&text=The%20most%20popular%20app%20in,app%20as%20of%20February%202020.
- 45. Parker B. Parent’s guide: WhatsApp increases the minimum age to use the WhatsApp from 13 to 16 and look at what SA parents need to know. 2019 [cited 2022 08 February]. Available from: https://www.news24.com/parent/family/parenting/parents-guide-whatsapp-increases-the-minimum-age-to-use-the-app-from-13-to-16-and-we-look-at-what-sa-parents-need-to-know-20190506.
- 46. World Wide Worx. 2015 Student Tech Survey highlights popularity of Facebook. 2015 [cited 2022 21 February]. Available from: https://www.bizcommunity.com/Article/196/347/125525.html.
- 47.
SAMHSA’s Trauma and Justice Strategic Initiative. SAMHSA’s concept of trauma and guidance for a trauma-informed approach. Rockville, MD: Substance Abuse and Mental Health Services Administration, 2014. Available from: https://ncsacw.samhsa.gov/userfiles/files/SAMHSA_Trauma.pdf.