Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

“Endless opportunities”: A qualitative exploration of facilitators and barriers to scale-up of two-way texting follow-up after voluntary medical male circumcision in Zimbabwe

  • Chelsea Elkins,

    Roles Formal analysis, Investigation, Writing – original draft

    Affiliation Department of Global Health, University of Washington, Seattle, Washington, United States of America

  • Sandra Kokera,

    Roles Formal analysis, Writing – original draft

    Affiliation Zimbabwe Technical Training and Education Centre for Health (Zim-TTECH), Harare, Zimbabwe

  • Phiona Vumbugwa,

    Roles Writing – original draft

    Affiliations Department of Global Health, University of Washington, Seattle, Washington, United States of America, Zimbabwe Technical Training and Education Centre for Health (Zim-TTECH), Harare, Zimbabwe

  • Jacqueline Gavhera,

    Roles Investigation, Writing – original draft

    Affiliation Zimbabwe Technical Training and Education Centre for Health (Zim-TTECH), Harare, Zimbabwe

  • Kathleen M. West,

    Roles Supervision, Validation, Writing – original draft

    Affiliation Health Systems & Population Health, University of Washington, Seattle, Washington, United States of America

  • Katherine Wilson,

    Roles Supervision, Writing – original draft

    Affiliations Department of Global Health, University of Washington, Seattle, Washington, United States of America, Department of Global Health, International Training and Education Center for Health, University of Washington, Seattle, Washington, United States of America

  • Batsirai Makunike-Chikwinya,

    Roles Supervision, Writing – original draft

    Affiliation Zimbabwe Technical Training and Education Centre for Health (Zim-TTECH), Harare, Zimbabwe

  • Lewis Masimba,

    Roles Supervision, Writing – original draft

    Affiliation Zimbabwe Technical Training and Education Centre for Health (Zim-TTECH), Harare, Zimbabwe

  • Marrianne Holec,

    Roles Project administration, Writing – original draft

    Affiliation Department of Global Health, International Training and Education Center for Health, University of Washington, Seattle, Washington, United States of America

  • Scott Barnhart,

    Roles Project administration, Supervision, Writing – original draft

    Affiliations Department of Global Health, University of Washington, Seattle, Washington, United States of America, Department of Global Health, International Training and Education Center for Health, University of Washington, Seattle, Washington, United States of America, Department of Medicine, University of Washington, Seattle, Washington, United States of America

  • Sulemana Matinu,

    Roles Writing – original draft

    Affiliation Department of Global Health, University of Washington, Seattle, Washington, United States of America

  • Beatrice Wassuna,

    Roles Writing – original draft

    Affiliation Medic, Nairobi, Kenya

  • Caryl Feldacker

    Roles Conceptualization, Formal analysis, Methodology, Project administration, Supervision, Writing – original draft, Writing – review & editing

    cfeld@uw.edu

    Affiliations Department of Global Health, University of Washington, Seattle, Washington, United States of America, Department of Global Health, International Training and Education Center for Health, University of Washington, Seattle, Washington, United States of America

Abstract

In Zimbabwe, the ZAZIC consortium employs two-way, text-based (2wT) follow-up to strengthen post-operative care for voluntary medical male circumcision (VMMC). 2wT scaled nationally with evidence of client support and strengthened follow-up. However, 2wT uptake among healthcare providers remains suboptimal. Understanding the gap between mobile health (mHealth) potential for innovation expansion and scale-up realization is critical for 2wT and other mHealth innovations. Therefore, we conducted an exploratory qualitative study with the objective of identifying 2wT program strengths, challenges, and suggestions for scale up as part of routine VMMC services. A total of 16 in-depth interviews (IDIs) with diverse 2wT stakeholders were conducted, including nurses, monitoring & evaluation teams, and technology partners–a combination of perspectives that provide new insights. We used both inductive and deductive coding for thematic analysis. Among 2wT drivers of expansion success, interviewees noted: 2wT care benefits for clients; effective hands-on 2wT training; ease of app use for providers; 2wT saved time and money; and 2wT strengthened client/provider interaction. For 2wT scale-up challenges, staff shortages; network infrastructure constraints; client costs; duplication of paper and electronic reporting; and complexity of digital tools integration. To improve 2wT robustness, respondents suggested: more staff training to offset turnover; making 2wT free for clients; using 2wT to replace paper VMMC reporting; integrating with routine VMMC reporting systems; and expanding 2wT to other health areas. High stakeholder participation in app design, implementation strengthening, and evaluation were appreciated. Several 2wT improvements stemmed from this study, including enrollment of multiple people on one number to account for phone sharing; 2wT inclusion of minors ages 15+; clients provided with $1 to offset SMS costs; and reduced SMS messages to clients. Continued 2wT mentoring for staff, harmonization of 2wT with Ministry e-health data systems, and increased awareness of 2wT’s client and provider benefits will help ensure successful 2wT scale-up.

Introduction

Voluntary medical male circumcision (VMMC) is a critical component of comprehensive HIV prevention and scaled nationally across much of Sub-Saharan Africa [1]. VMMC is safe with few serious adverse events (AEs) [28]. Post-operative follow-up mitigates the risk of Aes and ensures quality care. In accordance with global guidance, national VMMC guidelines recommend in-person follow-up visits in the two weeks following procedure [9]. However, with few Aes, in-person follow-up compounds pressure on already constrained systems in resource-scarce settings, like Zimbabwe, with severe national healthcare worker (HCW) shortages [10].

The International Training and Education Center for Health (I-TECH) formed the ZAZIC Consortium (an acronym composed of letters from member partners: Zimbabwe Technical Assistance, Training and Education Center for Health (Zim-TTECH); Zimbabwe Association of Church Related Hospitals (ZACH); and Zimbabwe Community Health Intervention Research Project (ZiCHIRe)) to conduct VMMC in partnership with the Ministry of Health and Child Care (MoHCC) [10]. ZAZIC has been a leader in VMMC innovations since 2013, working to integrate VMMC services into routine healthcare delivery and ensuring quality service delivery by exploring and addressing factors that contribute to Aes [2,11,12]. ZAZIC’s VMMC program is efficient and safe, reaching over 500,000 males with an AE rate of 0.13% [6].

In response to few Aes and required post-operative visits that burdened overstretched VMMC providers, I-TECH designed, tested, evaluated and scaled an app using two-way texting (2wT) to improve the quality of post-operative care while reducing the burden on HCWs and reducing costs. The 2wT system was rigorously tested via a randomized control trial (RCT) in 2019, providing evidence on 2wT safety, usability, and workload reduction [10,1315]. In brief, the 2wT system for VMMC is a mobile health (mHealth) innovation that was built using the open-source Community Health Toolkit (CHT). This 2wT system is optimized for local Zimbabwean users, allowing VMMC clients to opt into post-operative follow-up via SMS (text message) with VMMC nurses instead of scheduled, in-person, post-operative visits [16]. 2wT provides automated educational messages about the healing process and facilitates interactive SMS with a VMMC nurse for concerns or reassurance, triaging clients to in-person care when needed. 2wT supports VMMC scale-up by allowing the vast majority of interviewees to heal at home while those in need of in-person review for potential Aes are promptly referred to on-site care by the 2wT nurse. Since 2019, while overcoming the restrictions and complications of service delivery within the COVID-19 context, ZAZIC transitioned the 2wT follow-up approach from the RCT to national scale-up, reaching over 33,000 men by 2023 [17]. 2wT was also successfully evaluated in a RCT in both rural and urban South Africa, with similar findings in safety, usability, efficiency, and improved quality of follow-up [1820].

2wT remains a safe and effective follow-up method, with high usability among providers and clients: over 50% of all ZAZIC VMMC clients receive 2wT-based follow-up. 2wT adheres to the mHealth principles [21] and aligns with recommendations gleaned from recent mHealth literature [2225], including: 1) 2wT follow-up is brief, reducing risks of poor retention, including challenges related to phone theft/loss; 2) 2wT’s hybrid approach of one-way educational messages to meet client desires for action-oriented knowledge combined with interactive, client/nurse telehealth improves engagement in care; 3) VMMC context reduces confidentiality concerns; 4) locally-led, human-centered design (HCD) tailors content to meet client and HCW needs; 5) SMS meets expressed client preferences while WhatsApp-like interface for providers reduces concerns of low digital literacy; 6) and, the “Hub-and-Spoke” model provides a dedicated central 2wT nurse, known as the “Hub Nurse”, to support site-based teams (“spokes”), reducing response delays.

Despite building upon digital health successes, support for scale-up, and clear evidence of 2wT advantages, the pace of 2wT expansion remains lower than expected. 2wT is not intended to meet the needs of all males in all VMMC locations. Yet, adequate network coverage and high phone ownership suggest that uptake should be closer to 75% of VMMC clients. Elucidation of some of the causes and possible strategies to help 2wT fulfill its promise could identify barriers and potential solutions for other evidence-based apps aimed at improving client care while reducing HCW workload–a critical issue across many low- and middle-income countries (LMIC).

Qualitative research that engages diverse stakeholders, including HCWs, can help better understand app dissemination opportunities and challenges [24]. Soliciting widespread user feedback as apps move from research to routine practice is also in line with World Health Organization (WHO) guidance to promote mHealth equity and provide quality evidence for decisions surrounding expansion, scale-up, and institutionalization [22,26,27]. In the mHealth field, especially during COVID-19, many apps did not transition from research or niche implementation to routine practice [28]. Moreover, users who are given multiple opportunities to provide feedback may be more invested and engaged in digital health programs, an important consideration as 2wT continues to generate multi-level buy-in for expansion [14,19,29,30]. User feedback is not a goal that can be accomplished, but rather a core component of frequent and flexible adjustment and adaptation to specific programmatic context [19]. Encouraging insights and feedback from HCWs, in particular, raises the likelihood of understanding what works in real-world clinical settings, aiding implementation success [31]. As 2wT has successfully been brought to scale and maintained for over 2 years in the Zimbabwe setting, there is a unique opportunity for inquiry to share with others in the growing mHealth field.

To improve the current program as well as inform future 2wT scale-up efforts, we conducted an exploratory qualitative study among 2wT stakeholders, including 2wT supervisors, enrollment nurses at 2wT VMMC sites, ZAZIC collaborators, and staff from 2wT technology partner, Medic. The main objective of the study was to identify 2wT program strengths, challenges, and suggestions for 2wT scale up as part of routine ZAZIC VMMC services from the perspective of these 2wT stakeholders. Engaging this broad group of digital health users is critical to inform successful scale up from RCT to widespread use in the routine VMMC setting. Incorporation of more voices from LMIC public health settings also enriches the optimization process and further improves fit of digital health where these innovations are most needed.

Methods

Setting

The ZAZIC Consortium provides comprehensive VMMC services in partnership with MoHCC in 14 districts, with a mix of urban, peri-urban and rural sites in Zimbabwe. VMMC demand creation and service delivery are also aimed at populations at higher risk of HIV, like artisanal mining and high migratory communities. We interviewed 16 2wT program partners for this study. We defined a 2wT program partner as an individual from an organization that participated in the design, implementation, scale up, and/or maintenance of ZAZIC’s 2wT program for VMMC follow-up. Interviewees included staff in a variety of roles across the ZAZIC Consortium; employees from the program’s technology partner, Medic Mobile; and MoHCC nurses implementing 2wT on site. ZAZIC and MoHCC site nurses are all based in Zimbabwe’s Harare, Midlands, Mashonaland Central, Mashonaland East, Mashonaland West, Masvingo, Matabeleland North and South Provinces. Medic Mobile employees interviewed are located in Kenya, Ghana, Uganda, and Nigeria.

2wT intervention at scale

The 2wT system has been described in detail previously [14,15,17]. Within the ZAZIC program, clients are enrolled into 2wT on the day of VMMC by VMMC site teams. On the day of VMMC, 2wT clients receive augmented post-operative counseling about how to interact with the 2wT system and reminders on signs of complication to monitor during the follow-up period. After the procedure, clients receive a daily text with an educational message or a prompt for a response. On interactive prompt days, clients have the option to respond with “0” (no issues) or “1” (possible problem), indicating whether they have concerns about their healing. A 2wT nurse interacts with all males who report a problem or those who write and SMS into the 2wT system with a concern. If the men do not respond with a concern, there is no immediate follow-up. Men who do not respond at all by day 7 are contacted with a call to confirm there are no issues.

Clients can send an SMS message to the 2wT system at any time. After-hours coverage on weekends and holidays ensures a timely response for urgent matters, with the regular 2wT nurse responding during routine clinic hours. Fig 1 illustrates the 2wT interaction flow [17]. The 2wT implementation operates as a Hub (the central 2wT “Hub Nurse”) and spokes (all sites), referred to as a “Hub-and-Spoke model”. Sites enroll 2wT clients and conduct follow-up visits, completing forms on in-person visits and observed AEs. The 2wT Hub Nurse interacts with men from all sites on a daily basis, providing reassurance, triaging men to in-person reviews using a “referral for care” form, and contacting men who do not respond by SMS or phone. The Hub Nurse refers men who have not responded to sites for follow up on Day 8.

2wT training for scale-up

For scale-up, 2wT training was conducted by a core team of four ZAZIC HCWs over a 1-week period and included 30 administrative staff from the MoHCC national-, province-, and district-levels. The training was cascaded down to facility levels by ZAZIC teams with two nurses per site attending a two-day training. Trainings were conducted virtually or in-person on-site, depending on COVID-19 protocols and logistics issues. In both formats, training included facility nurses, data clerks, site managers, and demand creation staff. The ‘trainer-of-trainers’ approach with on-the-job mentorship was used: the first group of trained staff subsequently trained other staff at their sites as staff turned over. The 2wT training curriculum includes theory and practical sessions conducting post-operative counseling; enrolling clients; documenting visits and client tracing outcomes into the 2wT system; and routine monitoring and evaluation. In-person trainings resumed immediately after COVID-19 precautions were reduced.

Data collection

We conducted interviews from December 17th, 2021 to February 27, 2022. We selected 16 interviewees, using a combination of purposive and snowball sampling from ZACH, ZICHIRE, I-TECH, MoHCC and Medic Mobile teams who provided direct management, support, or implementation of 2wT at national-, district-, or health facility-level [10,32]. The sample size is reflective of research demonstrating that a high quantity of interviews is often not necessary for high caliber qualitative research [33]. As well as a recent review of qualitative studies that noted studies with 9–17 individual interviews can be sufficient for thematic saturation [34]. Furthermore, our goal was to hear from key implementing partners in the 2wT program, and we successfully interviewed the majority of partners (16/21) across all organizations that supported the program implementation. We conducted and recorded the individual in-depth interviews (IDIs) via Zoom or WhatsApp using a semi-structured interview guide [21]. Interviews were conducted remotely due to COVID-19 health concerns and travel restrictions. Most interviews lasted 20–35 minutes, with an average length of 27 minutes. Interview data are stored in a password protected shared drive approved for research through our institution. The primary interviewer transcribed the interviews, editing for clarity as needed.

Data analysis

We undertook a thematic analysis approach [35]. We iteratively developed a codebook, initially identifying codes deductively based on study goals and interests. Additional inductive, or open, coding allowed us to capture unanticipated ideas regarding program barriers [36]. After several iterations, a program member reviewed and commented on the draft before we finalized the codebook. The primary coder (CE) applied the codebook to all transcripts using ATLAS.ti version 9 [35,36]. A secondary coder (SK) reviewed the primary codes and identified additional open codes and minor discrepancies in interpretation that were resolved through a common writing process [37]. The study team identified, a priori, three domains according to study goals. We sorted codes into these domains and examined the coded data for patterns within and across stakeholder groups, to develop into themes under each domain. Again, we undertook a member checking process with individuals in the program, as well as the larger study team, to ensure accurate interpretation and communication of ideas.

Positionality and trustworthiness.

The lead interviewer is not employed by Zim-TTECH, nor any partner organization, but is affiliated with the University of Washington Department of Global Health, a funder of the 2wT program in Zimbabwe. This quality improvement study received no specific funding. This may have impacted how interviewees responded to questions. The remaining authors are either from, or work closely with, the Zim-TTECH organization which may have influenced their interpretation of the responses. Another factor is that all interviews took place virtually rather than in-person due to COVID-19 travel restrictions. Interviewees may have been more or less comfortable with virtual interviews, impacting their responses. All researchers were attentive to iterative process evaluations of trustworthiness [25], assessing and strengthening the study’s methodological rigor, transparency, and inclusion across data collection, analysis, and results reporting phases.

Ethics

The human subjects division of the University of Washington’s Institutional Review Board determined that this human subjects research was for the purposes of quality improvement that qualified for exempt status (IRB ID: STUDY00014561). In Zimbabwe, the Medical Research Council of Zimbabwe is required when engaging in non-exempt research. The ZAZIC Consortium and Medic approved participation of employees who wished to volunteer in this study. All participants provided informed verbal consent to be interviewed and recorded prior to participating in this study.

Results

Interviewee characteristics

Among 21 partners contacted, 16 participated in the study, representing the five organizations responsible for the 2wT implementation process. Interviewees either worked for the technology partner, Medic Mobile (n = 5); as MoHCC site nurses (n = 2); or within the ZAZIC Consortium (n = 9), including Zim-TTECH (n = 3); ZACH (n = 3), and ZiCHIRe (n = 3). Most interviewees (n = 11) lived in Zimbabwe, except for Medic Mobile interviewees who were based in Kenya, Ghana, Nigeria, or Uganda (Table 1).

thumbnail
Table 1. Characteristics of participating program partners (N = 16).

https://doi.org/10.1371/journal.pone.0296570.t001

MoHCC site nurses conduct 2wT related tasks, such as client recruitment, reviewing clients referred for suspected AEs or loss to follow-up, and 2wT oversight at the health facility-level. ZAZIC partner responsibilities include facilitating the 2wT training; providing overall 2wT system operations oversight, serving as the 2wT “Hub” to triage clients for care, following up on clients lost to follow-up or with suspected Aes, referrals for in-person follow-up, monitoring 2wT system challenges, maintaining documentation, and conducting quality assurance. Technology partners at Medic Mobile provide technical support to the 2wT program, including development of software, upgrading 2wT systems, and supporting continued program staff and HCW use. Our results are organized by three domains: Program Strengths, Program Challenges, and Suggested Improvements, with themes under each domain as presented in Table 2.

Domain 1: 2wT program strengths at scale

2wT improves quality of client follow-up care.

Interviewees overwhelmingly reported that the 2wT program is largely a success, as one interviewee remarked, “we have a lot to really celebrate.” [IDI-15] HCW interviewees noted advantages for client care, since now “[clients] have the motivation and are willing to respond…People are interested in their own health and are willing to communicate directly with the two-way texting nurse. [IDI-01] HCWs expressed that 2wT increases close post-operative monitoring and improves identification of Aes over in-person visits:

[2wT] has increased the follow-up rate. It also has improved adverse event surveillance. I think that close interaction with clients would always [allow us] to know if there is an adverse event. [IDI-07]

2wT also appears to improve follow-up data quality since the system both stores interactive messages and alerts nurses for timely documentation of referrals to care outcomes. Instead of clerks transcribing clinical notes from multiple sources, the 2wT nurse completes reporting. “With two-way texting…it’s the service provider who is in contact with the client. [They are] getting first-hand information and entering the data.” [IDI-09]

Continuous, on-the-job 2wT training is effective for 2wT scale-up support.

Interviewees described the 2wT curriculum content of in-class role plays, 2wT-based practical exercises, and on-the-job training, as simple and useful. “The training has been quite successful in giving all the information within a day, and [HCWs are] able to use it soon afterward.” [IDI-04] The one-day format caused little disruption to the clinicians’ duties, and the practical training enhanced learning the content. One interviewee noted, “hands-on [training] sped up the learning process, and also improved the buy-in and appreciation of the technology.” [IDI-01] When COVID-19 restrictions led to a temporary change to virtual training, HCWs missed on-site support and repetitive practice. Still, virtual training did have upsides, because “all players are there, everyone is there, they exchange ideas. [Training by] visiting HCWs in their facilities takes time and people don’t have one platform to exchange ideas. [IDI-08] Interviewees remarked on the importance of continuous trainings, and voiced that the role of the Hub Nurse should “[provide] continuous mentorship support” [IDI-08]

2wT saves provider, client, and program costs, creating enthusiasm for continued scale-up.

Interviewees reported that the 2wT program contributed to resource savings, both financial and human. For HCWs, 2wT saved travel time over in-person follow-up, leaving time for demand creation or other health services. On the program level, 2wT also decreases costs, “because for you to go for follow-ups, you need a vehicle, you need to support the service providers…it involves a lot of personnel. But if there’s someone just sitting there and in charge of the clients [via 2wT], it’s a lot easier. [IDI-02] Respondents noted that they believe that 2wT largely eliminates the need for clients to travel and take time off work. “Two-way texting came at a time when clients, especially adult males, were not willing to have the commitments to meet the service delivery teams and have them conduct reviews. So, I think it’s popular. It’s very convenient for clients.” [IDI-07] Several HCWs confirmed that client demand for 2wT has grown, so that “clients will just come in and they’ll ask, ‘Oh, I want to be part of the clients that will be texting back and forth. I won’t be able to come [into the clinic] because I’ll be at work. [IDI-03]

The 2wT app is easy to use, making it attractive for healthcare workers.

For HCWs, most interviewees identified the ease and usability of the 2wT app as a major strength of the program: “It’s just very simple steps to follow.. It’s actually very friendly for the healthcare staff. [IDI-03] HCWs also remarked on the multi-lingual 2wT system that reaches clients in their preferred language. HCWs felt the system is easy for clients to use, regardless of their education. “The [clients] easily understand it, even those who we were not sure would be able to use it because of literacy levels. [IDI-06]. The simplicity of the app for HCW and client users was attributed to the 2wT co-design approach. A respondent from technology partner, Medic Mobile, noted: “We designed a system with our users, for our users…. That’s something that I would say I’m really proud of, just from the start: making sure that the digital health intervention really resonates with its users and that it’s really solving for their pain points. [IDI-15]. Interviewees also appreciated ongoing 2wT adaptation and revision based on ongoing user feedback. For example, to improve messaging speed, Medic Mobile developers leveraged the Community Health Toolkit (CHT) integration with another open-source tool, RapidPro, “to allow more [2wT] messages to be sent out at a given time. I think that’s probably one of the big things…that might be going well with the scale-up. [IDI-16]. Frontline HCW appreciated this consistent effort to simplify and improve their own user experience, that of their VMMC clients.

2wT is widely accepted by clients, communities and MoHCC.

Most interviewees’ perception was that the 2wT program was generally well received by clients and HCWs alike, resulting in a high program uptake. “The response rate has been good. Even in areas where there is no good [cellular] network.” [IDI-01] Interviewees identified promotion of the 2wT follow-up method at community and facility levels as a program strength, including distribution of 2wT branded masks and hats, sensitization on 2wT during community health talks, 2wT training for village health workers, and VMMC outreach to adult working men highlighting availability of 2wT-based follow-up. As a result, one interviewee stated that “two-way texting is now a household name.” However, HCWs noted that broad support was not immediate. HCWs initially had their own doubts about client safety and concerns about clients’ digital literacy, but with time, “our population kind of embraced [2wT]. [IDI-01] While success among clients and HCWs was encouraging, respondents were most impressed with MoHCC acceptance. “The fact that the Ministry really appreciated this two-way texting intervention and supported the scale up [is a] huge success.” [IDI-15] Another respondent considered MoHCC partnership as critical for success, noting that “[2wT is] a good innovation [which] is telling policymakers that these [physical] follow-ups are not necessary, and policy should be informed by this finding. [IDI-02]

Domain 2: 2wT approach challenges for expansion maintenance

Difficult maintaining core numbers of 2wT trained or motivated HCWs at site level.

Multiple respondents reported lack of sufficient trained HCW to support the 2wT program. Although sites had at least one trained staff member, those with only one found that enrolment was disrupted if the trained nurse was not on duty or took a different job. “Once our trained nurses have gone away, we have no one enrolling our clients. [IDI-04] Multiple partners identified staff shortages due to attrition, or a shortage of trained HCWs at sites, as a constraint to scale-up. However, even with on-site 2wT training, staff must perceive 2wT as part of routine duties, and not as a special or additional task. Multiple interviewees pointed out that some sites had HCWs who appear reticent to use to 2wT since, “in some sites, you would get someone who is not very keen with two-way texting.” [IDI-02]

Persistent technological barriers to sustained 2wT roll-out.

Poor cellular network connectivity and access to devices slow 2wT expansion. Of the 16 program partners interviewed, 13 discussed poor mobile networks as a constraint to scale-up that deprived program partners and clients access to the system. Especially in rural areas, “some of the areas are very remote and network coverage is very poor…It’s difficult to implement when there is no mobile network. [IDI-07]. Still, urban areas could face power outages that affect network availability. Interviewees also reported that the 2wT app was slow when high volumes of new clients were added. “After a while, the system starts to slow down because of the volume of data on it. [Now], we purge old data from the devices, so the system doesn’t have to filter through so much data before it loads. [IDI-14] System delays can lead HCWs to enroll clients after hours, causing syncing issues that result in data gaps, or, infrequently, some clients not receiving messages on time or at all.

Clients struggle with 2wT costs and communication.

Interviewees described their observations of client challenges, such as low technology literacy and phone sharing as barriers to scale-up. Some stakeholders felt that clients lacked a phone or shared a phone with others in their household, particularly in rural areas or among minors. IDI-09 commented that, “Most of them share phones with their parents, their siblings, and sometimes you may get responses, not from the client, but from another person. Multiple interviewees also stated that cost was a potential barrier as clients had to purchase their own airtime for communicating with the nurses via 2wT. “The issue of just the availability of the dollar, to be texting back and forth [is a challenge]. [IDI-03]

Complexity of data management between MoHCC paper forms and 2wT digital database.

To date, MoHCC mandates paper reporting for VMMC procedures, resulting in data duplication for 2wT clients. IDI-02 remarked on “the duplication of the system in the enrollment. Because we have paper-based client intake forms, and we have this two-way texting system, which is collecting the same data. IDI-03 identified the additional work this created, noting the need “for the hard copy for the circumcision itself. As much as we are going to enter them into the system, we are still going to enter them in another physical form. Making changes to global or national policies such as discontinuation of paper forms is not simple. “There is not much control in terms of determining the pace of moving things. It’s all reliant on the Ministry of Health. [IDI-13]

Domain 3: Suggestions to facilitate wider scale-up of the 2wT approach

Improve the technology for HCWs and clients.

Several interviewees stated that previous clients should be archived to free the system memory and increase the application speed on HCW devices. Features such as improved data visualization and audible notifications for new messages could promote timely nurse responses. Several interviewees proposed 2wT be delivered on WhatsApp, versus traditional SMS, “because with WhatsApp, [clients] say it’s cheaper, and they no longer communicate using an SMS. [IDI-05] Additionally, multiple interviewees discussed needing system updates to allow multiple clients to enroll with the same phone number or f multiple phone numbers per person, accommodating those with multiple SIM cards. “The platform should allow a number to be entered twice, especially when the surname is the same, to show that these [clients] are related and they’re using the same number. [IDI-11] Lastly, considerations of improved security, such as two factor authentication for access to 2wT client data were voiced as being needed.

Train, and refresher train, more 2wT staff.

While almost every interviewee stated that the 2wT training was an overall success, some cautioned, “if you are to expand, there is a need to cascade the training down, so that you can also have provincial-based trainers, or district, or even site-based trainers. [IDI-07] Incorporating the 2wT program training into the standard VMMC training package for all new VMMC staff could also ensure that all MOH service providers become skilled with 2wT: “Because when we, if we, are going into sustainability, that is going to be a huge barrier. [IDI-11] Lastly, improving the training’s reach by extending 2wT training to community health workers, and offering consistent refresher training, will help fill personnel gaps: “when one [trained healthcare worker] is absent, or unwilling, another can take over.” [IDI-06]

Replace paper with 2wT for all VMMC data.

Duplication of paper and electronic data increases HCW burden, potentially compromising data completeness in either system. "Maybe if we could do away with the paper based and we just go digital, it might improve our adverse event reporting? [IDI-04] Another interviewee suggested that if all VMMC data were entered into the same system, 2wT enrollments could be automated, and those not enrolled into 2wT could still have their routine VMMC data entered digitally. “Those with phone numbers automatically go into two-way texting. Those without, you have to indicate that they didn’t bring a phone. Electronically moving away from the use of paper [client intake forms]. [IDI-11] Comprehensive electronic VMMC data could ensure data linkage “to other databases that are used by Ministry of Health as well, so that it’s available to a wider group, [IDI-08], making data more readily available to decision-makers.

Transition 2wT to more MoHCC ownership and integrate 2wT into Zimbabwean e-health.

Several respondents noted a desire to offer 2wT to other VMMC implementing partners and transferring implementation to central MoHCC leadership. 2wT could be “made a national thing, through the Ministry of Health through sensitization…to say, now everybody’s using two-way texting through the Ministry of Health, I think that would improve uptake. [IDI-04] The open-source (no fee) CHT 2wT software gave respondents optimism for potential MoHCC transition, “because we were worried that if its proprietary software, once the other partners would be off [stop funding 2wT], the Ministry of Health would not be able to sustain its use. [IDI-13] To support transition, respondents noted that local software engineers from partner organizations and/or MoHCC must be trained to competence and confidence in the CHT, including the 2wT system, and that 2wT would need local server hosting. “Transiting software engineering skills to the local team for 2wT maintenance is a priority for consistently meeting the needs of end users. [IDI15]. Lastly, there is a huge leap between recognizing 2wT benefits and integrating 2wT into MoHCC e-health architecture. “Trying to sell [2wT] to the other people, especially the health informatics department, they appreciate it, but they had huge expectations that two-way texting would seamlessly integrate with all aspects of the electronic health system. [IDI-01] The pathway forward is neither simple nor straightforward.

Expand 2wT follow-up to other health conditions beyond VMMC.

All respondents noted that 2wT could benefit clients and HCWs beyond VMMC. “There are so many opportunities of which we haven’t done much in terms of demand creation… [2wT] information dissemination, there is a huge gap and opportunity. [IDI-07] For example, 2wT supported VMMC service delivery during COVID-19 and could be leveraged for subsequent pandemic waves. “Two-way texting has come at the right time, because we are able to review clients without physical reviews… during this outbreak of COVID.” [IDI-05] As COVID-19 wanes, but client demand for improved communication with HCWs grows, interviewees identified areas for 2wT growth. “For example, growth monitoring, HIV care and treatment, immunizations and so on. It will be very useful because it informs the health worker how the client is and will keep the client informed about when to make a certain visit or take drugs. [IDI-06] Another interviewee suggested that 2wT can provide “follow-up for outpatients with chronic conditions, say the diabetics or hypertensive clients, they can measure their own blood pressure and give feedback via 2wT and avoid unnecessary movements going back to the hospital or clinic. [IDI-07]

Discussion

This paper shares perspectives, insights, and suggestions from a unique set of 2wT app stakeholders, including frontline HCWs and software developers, to increase understanding of the process, pitfalls, and pathways forward to bring 2wT to scale in routine VMMC settings in Zimbabwe. These lessons learned should be of great interest in South Africa where 2wT is scaling after evidence of safety, usability, and efficiency gains [1820]. Interviewees noted that drivers of 2wT scale-up included enhanced promotion of client care and cost benefits, comprehensive and continuous HCW training, and ease of app use for HCWs and clients. In contrast, lack of available HCWs, technology challenges, client communication costs, and complexities in the transition to MoHCC ownership stymied progress. Interviewees suggested that national 2wT scale-up would benefit from an expanded training cascade, transitioning from paper to digital VMMC reporting, MoHCC ownership of 2wT, and using the 2wT model for other short-term client follow-up. Several lessons learned merit additional consideration.

First, the 2wT training and mentorship model appears to play a critical role in creating cadres of confident, competent, 2wT HCW users. Similar to an evaluation from Sierra Leone demonstrating the critical importance of both initial and refresher training for HCWs for institutionalization, [38] 2wT respondents believed that the combination of brief (<1 day) on-site training with continuous and accessible mentoring was effective to engage providers and facility teams in 2wT implementation and maintenance. During COVID-19, remote technology-based training (Zoom or WhatsApp) was an advantageous stop-gap and fit easily into HCWs personal schedules [39]. However, it is clear that HCWs preferred on-site in-person training and follow-up mentoring when possible. To facilitate in-person training, more decentralized 2wT training cascades across more districts are needed to more swiftly respond to shortages of 2wT-trained HCWs and sustain implementation at scale [40].

Second, as with other mHealth apps, 2wT enhanced provider to client communication and built relationships. Men typically underutilize healthcare services [41,42]. But, mHealth appears to strengthen the relationships between male clients and their HCWs [17,19]. HCWs, therefore, want to use mHealth to better engage their male clients in care [43]. The 2wT approach helped clinicians better care for their rural and distant clients by providing a platform for timely client/provider communication [44]. These 2wT findings largely complement a recent review that found mHealth could also improve coordination among HCWs, promote flexibility in follow-up care, and enhance relationships between clients and their care givers, engaging clients and providers in a shared responsibility for healing [45].

While technology still presents some barriers, client and provider creativity can help overcome challenges reaching clients in more remote or less connected areas. 2wT is offline-first, allowing clients to enroll even without network coverage. Data are stored and then synced later when providers return to connectivity. Although 2wT requires that clients receive an enrollment message to confirm their correct phone number for follow-up, many facilities now utilize solar power and providers access facility WIFI to connect daily, minimizing delays in messages being sent and received. Provider confidence in reaching clients is maintained despite short-term delays in messaging. Moreover, even in areas with poor connectivity, clients are aware of where to receive a signal near their homes or workplaces. Clients seek out these network “spot zones” for other communication needs and can receive and respond to 2wT messages concurrently, meeting the program reporting requirements and preventing client loss-to-follow-up.

Results from this qualitative study led to several program improvements since study conclusion. First, ZAZIC now enrolls minors aged 15–17 years in 2wT via the minor’s or the parent/guardian’s phone, expanding access to 2wT. Second, in response to client and provider feedback, daily text message with required response volume was reduced from 15 to 11 and language offerings were expanded to include Ndebele. These changes increased the educational content, reduced SMS volume overall, and responded to client communication preferences. Third, clients are offered $1 for initial transport reimbursement and to offset 2wT airtime costs, decreasing cost barriers to 2wT uptake. Fourth, system improvements allow clients to register with a primary and secondary number, responding from either number as network connectivity fluctuates. 2wT enrolls surgical and device-based VMMC clients (e.g., ShangRing), allowing men to have 2wT follow-up regardless of VMMC method. Lastly, technology improvements, such as weekly data purging and procuring phones with larger memory, have sped up processing for providers and facilities. We recommend a follow-up study to track the impact of these new program changes on 2wT outcomes and support from program partners.

This study has several limitations. First, we aimed to explore 2wT uptake challenges in the scale-up phase from the perspective of healthcare providers and software developers. Additional study among clients to gather their perspective on facilitators and barriers to uptake are warranted in future study. Second, interviewees who volunteered for this study may have had more time, capacity, and interest in the program than their counterparts who declined to participate. Additionally, while all partner organizations were represented, it is likely that views within partner organizations vary, and we likely did not capture the full range of views. Further study focusing on the technology perspective could further illuminate specific technical areas that were not well explored. Finally, these interviews took place over Zoom, sometimes with a poor connection, which may have led to some lack of clarity in transcription or reduced conversation comfort, potentially impacting the interview quality. Although these biases cannot be quantified, the research team feels that the strengths of this exploratory study outweigh the weaknesses, contributing understanding to mHealth scale up in Sub-Saharan Africa.

Conclusion

There is rigorous evidence of 2wT’s positive impact on client care quality and acceptability among HCWs, but widespread scale-up remains incomplete. Despite enrollments of over 33,000 males ages 15 and older, and broad approval from diverse client, provider, partner, and MoHCC stakeholders, digital innovations still present complex challenges to further scale up. For long-term success, how and when to integrate 2wT into current e-health platforms, including potential integration into the Electronic Health Records (EHR) system, should be further explored. Continuous collaboration between mHealth program implementers across all VMMC partners, from ZAZIC to MoHCC, would help foster continuous 2wT improvements for clients and providers across districts. Although the 2wT app is free to use, highly skilled and paid workers are needed to adapt, maintain, and manage the system. In the short-term, ZAZIC is growing its capacity to independently manage the CHT-based 2wT app with support from Medic Mobile, creating a transition plan for local 2wT management, supervision, and maintenance on the pathway to MoHCC ownership. However, for longer-term 2wT transition to MoHCC, human resources, funding for SMS, server space, and availability of trained technology teams are required. If leveraged well, as noted by one respondent, 2wT could be utilized in a variety of settings and health conditions: “The opportunities are really endless.”

Supporting information

S1 File. PLOS ONE inclusivity in global research questionnaire.

https://doi.org/10.1371/journal.pone.0296570.s001

(PDF)

Acknowledgments

The authors would like to thank the members of the ZAZIC Consortium, including the International Training and Education Center for Health (I-TECH,) the Zimbabwe Association of Church Related Hospitals (ZACH), Zimbabwe Community Health Intervention Research (ZiCHIRe), and Zimbabwe Technical Assistance Training & Education Center for Health (Zim-TTECH) for their support of 2wT. The authors would also like to appreciate the collaboration of the MoHCC and the facility teams for their dedication to quality VMMC service delivery, including 2wT based follow-up.

References

  1. 1. UNAIDS, WHO. Voluntary Medical Male Circumcision [Internet]. Geneva, Switzerland; 2021 Feb [cited 2021 Nov 14]. (UNAIDS and WHO Progress Brief). Available from: https://www.malecircumcision.org/resource/unaids-and-who-progress-brief-voluntary-medical-male-circumcision-february-2021.
  2. 2. Bochner AF, Feldacker C, Makunike B, Holec M, Murenje V, Stepaniak A, et al. Adverse event profile of a mature voluntary medical male circumcision programme performing PrePex and surgical procedures in Zimbabwe. Journal of the International AIDS Society. 2017 Mar 1;20(1). pmid:28362066
  3. 3. Hellar A, Plotkin M, Lija G, Mwanamsangu A, Mkungume S, Christensen A, et al. Adverse Events in a Large-Scale VMMC Programme in Tanzania: Findings from a Case Series Analysis. Journal of the International AIDS Society. 2019 Jul 1;22(7):e25369. pmid:31368235
  4. 4. Muchiri E, Charalambous S, Ginindza S, Maraisane M, Maringa T, Vranken P, et al. Description of adverse events among adult men following voluntary medical male circumcision: Findings from a circumcision programme in two provinces of South Africa. PLoS ONE. 2021 Aug 17;16(8):e0253960–e0253960. pmid:34403409
  5. 5. O’Bryan G, Feldacker C, Ensminger A, Nghatanga M, Brandt L, Shepard M, et al. Adverse event profile and associated factors following surgical voluntary medical male circumcision in two regions of Namibia, 2015–2018. PLoS ONE. 2021 Oct 20;16(10):e0258611–e0258611. pmid:34669709
  6. 6. Omollo V, Marongwe P, Murenje V, Madoda T, Paidamoyo G, Gwenzi F, et al. Adverse Event Trends Within a Large-Scale, Routine, Voluntary Medical Male Circumcision Program in Zimbabwe, 2014–2019. JAIDS Journal of Acquired Immune Deficiency Syndromes. 2021 Oct 1;88(2):173–80. pmid:34173789
  7. 7. Lucas T, Cooney C, Prainito A, Godfrey C, Kiggundu V, Thomas AG, et al. Consolidated Overview of Notifiable Adverse Events in the U.S. President’s Emergency Plan for AIDS Relief’s Voluntary Medical Male Circumcision Program Through 2020. Current HIV/AIDS reports. 2022;19(6):508–15. pmid:36348185
  8. 8. Brito S, Korn A, Monteiro L, Mudender F, Maiela A, Come J, et al. Need for improved detection of voluntary medical male circumcision adverse events in Mozambique: a mixed-methods assessment. BMC Health Services Research. 2019 Nov 21;19(1):85. pmid:31752838
  9. 9. U.S. Department of State. PEPFAR Country/Regional Operational Plan 2022 Guidance (COP/ROP 2022). 2022. (The United States President’s Emergency Plan for AIDS Relief Country Operational Plan Guidance). Available from: https://www.state.gov/wp-content/uploads/2022/02/COP22-Guidance-Final_508-Compliant-3.pdf
  10. 10. Feldacker C, Murenje V, Holeman I, Makunike-Chikwinya B, Korir M, Gundidza PT, et al. Reducing Provider Workload While Preserving Patient Safety: A Randomized Control Trial Using 2-Way Texting for Postoperative Follow-up in Zimbabwe’s Voluntary Medical Male Circumcision Program. JAIDS. 2020 Jan 1;83(1):16–23. pmid:31809358
  11. 11. Feldacker C, Bochner AF, Murenje V, Makunike-Chikwinya B, Holec M, Xaba S, et al. Timing of adverse events among voluntary medical male circumcision clients: Implications from routine service delivery in Zimbabwe. PLoS One. 2018 Sep;13(9):e0205113. pmid:30192816
  12. 12. Feldacker C, Murenje V, Makunike-Chikwinya B, Hove J, Munyaradzi T, Marongwe P, et al. Balancing competing priorities: Quantity versus quality within a routine, voluntary medical male circumcision program operating at scale in Zimbabwe. PLoS ONE. 2020 Oct 13;15(10):e0240425–e0240425. pmid:33048977
  13. 13. Babigumira JB, Barnhart S, Mendelsohn JM, Murenje V, Tshimanga M, Mauhy C, et al. Cost-effectiveness analysis of two-way texting for post-operative follow-up in Zimbabwe’s voluntary medical male circumcision program. PLoS ONE. 2020 Sep 30;15(9):e0239915–e0239915. pmid:32997710
  14. 14. Feldacker C, Holeman I, Murenje V, Xaba S, Korir M, Wambua B, et al. Usability and acceptability of a two-way texting intervention for post-operative follow-up for voluntary medical male circumcision in Zimbabwe. PLoS ONE. 2020 Jun 16;15(6):e0233234–e0233234. pmid:32544161
  15. 15. Feldacker C, Murenje V, Barnhart S, Xaba S, Makunike-Chikwinya B, Holeman I, et al. Reducing provider workload while preserving patient safety via a two-way texting intervention in Zimbabwe’s voluntary medical male circumcision program: study protocol for an un-blinded, prospective, non-inferiority, randomized controlled trial. Trials. 2019 Jul 23; 20: 451 [cited 2023 Jun 6];20(1). pmid:31337414
  16. 16. Community Health Toolkit [Internet]. [cited 2023 Feb 13]. The Community Health Toolkit (CHT). Available from: https://communityhealthtoolkit.org
  17. 17. Marongwe P, Wasunna B, Murenje V, Gavera J, Gwenzi F, Hove J, et al. Transitioning a digital health innovation from research to routine practice: two-way texting for male circumcision follow-up in Zimbabwe. PLOS Digital Health. 2022. pmid:36812548
  18. 18. Feldacker C, Pienaar J, Wasunna B, Ndebele F, Khumalo C, Day S, et al. Expanding the Evidence on the Safety and Efficiency of 2-Way Text Messaging-Based Telehealth for Voluntary Medical Male Circumcision Follow-up Compared With In-Person Reviews: Randomized Controlled Trial in Rural and Urban South Africa. Journal of medical Internet research. 2023;25:e42111–e42111. pmid:37159245
  19. 19. Day S, Ncube V, Maja L, Wasunna B, Pienaar J, Setswe G, et al. Centering Frontline Health Care Workers in Digital Innovation Design to Inform the Optimization of an App for Improved Male Circumcision Follow-up in South Africa: Qualitative Usability Study. JMIR formative research. 2023;7: e44122–e44122. pmid:36947127
  20. 20. Pienaar J, Day S, Setswe G, Wasunna B, Ncube V, Ndebele F, et al. ‘I understood the texting process well’. Participant perspectives on usability and acceptability of SMS-based telehealth follow-up after voluntary medical male circumcision in South Africa. Digital health. 2023;9: 1–10. pmid:37654716
  21. 21. Principles for Digital Development 2022. Principles for Digital Development [Internet]. 2022. [cited 2022 August 01] Available from: https://digitalprinciples.org/
  22. 22. World Health Organization. WHO guideline: recommendations on digital interventions for health system strengthening: web supplement 2: summary of findings and GRADE tables. World Health Organization; 2019. Available from: https://apps.who.int/iris/bitstream/handle/10665/324998/WHO-RHR-19.7-eng.pdf?ua=1
  23. 23. World Health Organization. Classification of Digital Health Interventions v1. 0: A Shared Language to Describe the Uses of Digital Technology for Health [Internet]. 2018. [cited 2021 October 1] Available from: https://apps.who.int/iris/bitstream/handle/10665/260480/WHO-RHR-18.06-eng.pdf
  24. 24. Duffy A, Christie GJ, Moreno S. The Challenges Toward Real-world Implementation of Digital Health Design Approaches: Narrative Review. JMIR Hum Factors. 2022 Sep 9;9(3). pmid:36083628
  25. 25. Elo S, Kääriäinen M, Kanste O, Pölkki T, Utriainen K, Kyngäs H. Qualitative Content Analysis: A Focus on Trustworthiness. SAGE Open. 2014;4(1).
  26. 26. Guo C, Ashrafian H, Ghafur S, Fontana G, Gardner C, Prime M. Challenges for the evaluation of digital health solutions—A call for innovative evidence generation approaches. npj Digit Med. 2020 Aug 27;3(110). pmid:32904379
  27. 27. Holst C, Sukums F, Radovanovic D, Ngowi B, Noll J, Winkler AS. Sub-Saharan Africa—the new breeding ground for global digital health. Lancet Digital Health. 2020;2(4):e160–162. pmid:33328076
  28. 28. UNICEF. Pulse Check On Digital Learning [Internet]. 2022 Dec [cited 2023 Feb 13]. Available from: https://www.unicef.org/reports/pulse-check.
  29. 29. Bazzano AN, Martin J, Hicks E, Faughnan M, Murphy L. Human-centred design in global health: A scoping review of applications and contexts. PloS one. 2017;12(11):e0186744–e0186744. pmid:29091935
  30. 30. Norman MK, Hamm ME, Schenker Y, Mayowski CA, Hierholzer W, Rubio DM, et al. Assessing the Application of Human-Centered Design to Translational Research. Journal of Clinical and Translational Science. 2021;5(1):e130–e130. pmid:34367675
  31. 31. Beres LK, Simbeza S, Holmes CB, Mwamba C, Mukamba N, Sharma A, et al. Human-Centered Design Lessons for Implementation Science: Improving the Implementation of a Patient-Centered Care Intervention. Journal of acquired immune deficiency syndromes (1999). 2019;82 Suppl 3(3):S230–43. pmid:31764259
  32. 32. Noy C. Sampling Knowledge: The Hermeneutics of Snowball Sampling in Qualitative Research. International Journal of Social Research Methodology. 2008 Oct;11(4):327–44.
  33. 33. Pope C, Mays N. Qualitative research in health care. 3rd ed. Malden, Mass: Blackwell Pub./BMJ Books; 2006.
  34. 34. Hennink M, Kaiser BN. Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Social science & medicine (1982). 2022;292:114523–114523. pmid:34785096
  35. 35. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. 2006 Apr;3(2):77–101.
  36. 36. Elliot V. Thinking about the Coding Process in Qualitative Data Analysis. The Qualitative Report. 2018;23(11):2850–61.
  37. 37. Savage J. One voice, different tunes: issues raised by dual analysis of a segment of qualitative data. Journal of Advanced Nursing Science. 2000;31(6):1493–500. pmid:10849163
  38. 38. Mc Kenna P, Babughirana G, Amponsah M, Egoeh SG, Banura E, Kanwagi R, et al. Mobile training and support (MOTS) service-using technology to increase Ebola preparedness of remotely-located community health workers (CHWs) in Sierra Leone. mHealth. 2019;5:35–35. pmid:31620462
  39. 39. Thomas MP, Kozikott S, Kamateeka M, Abdu-Aguye R, Agogo E, Bello BG, et al. Development of a simple and effective online training for health workers: results from a pilot in Nigeria. BMC public health. 2022;22(1):551–551. pmid:35313834
  40. 40. Bakibinga P, Kamande E, Kisia L, Omuya M, Matanda DJ, Kyobutungi C. Challenges and prospects for implementation of community health volunteers’ digital health solutions in Kenya: a qualitative study. BMC health services research. 2020;20(1):888–888. pmid:32957958
  41. 41. Prata N, Weidert K, Soro DR. A mixed-methods study to explore opportunities and challenges with using a mHealth approach to engage men who have sex with men in HIV prevention, treatment and care in Lomé, Togo. mHealth. 2021;7:47–47. pmid:34345624
  42. 42. Beia T, Kielmann K, Diaconu K. Changing men or changing health systems? A scoping review of interventions, services and programmes targeting men’s health in sub-Saharan Africa. International journal for equity in health. 2021;20(1):87–87. pmid:33789688
  43. 43. Mbokazi N, Madzima R, Leon N, Lurie MN, Cornell M, Schmidt BM, et al. Health Worker Experiences of and Perspectives on Engaging Men in HIV Care: A Qualitative Study in Cape Town, South Africa. Journal of the International Association of Providers of AIDS Care. 2020;19. pmid:32597712
  44. 44. Fairbanks J, Beima-Sofie K, Akinyi P, Matemo D, Unger JA, Kinuthia J, et al. “You Will Know That Despite Being HIV Positive You Are Not Alone”: Qualitative Study to Inform Content of a Text Messaging Intervention to Improve Prevention of Mother-to-Child HIV Transmission. JMIR mHealth and uHealth. 2018;6(7):e10671. pmid:30026177
  45. 45. Odendaal WA, Watkins JA, Leon N, Goudge J, Griffiths F, Tomlinson M, et al. Health workers’ perceptions and experiences of using mHealth technologies to deliver primary healthcare services: a qualitative evidence synthesis. Cochrane Database of Systematic Reviews. 2020;(3). pmid:32216074