Introduction

Treatment advances for HIV over the last four decades have transformed HIV into a chronic condition [1, 2]. Seropositive individuals are now experiencing the additional burden of multi-morbidity due to long exposure to HIV, antiretroviral treatment, and aging with concomitant effect on health-related quality of life (HRQoL) [3]. With the global progress in reaching and maintaining the 90% target for the diagnosis, treatment, and viral suppression, the HIV community is now emphasizing on beyond viral suppression by adding good HRQoL as the fourth 90% target [4]. Self-management is fundamental to improvement in HRQoL as it is focused on delivering meaningful and tailored information to patients so that they can set health-related goals and set in place appropriate action plans aimed to improve their health outcome over time [5].

Self-management interventions that incorporated goal setting component have shown to be effective in increasing patients’ progress toward better health outcomes in the context of chronic diseases [6, 7]. Until now, studies of goal setting to improve health outcomes for people with chronic diseases have used a collaborative approach to goal setting. Time, energy, and active engagement of the patient and the clinician are key factors in forming an effective goal setting process [8]. With limited time allocation during clinic visits and medical topics competing with patients’ priorities [9], it is unclear how much goal setting can be accomplished. Patient-formulated goals, without clinicians’ input, have not received much attention. At the population level (i.e., outside clinical setting), where there is no direct input from the clinicians, setting good quality goalsFootnote 1 and making plans for actions are difficult and can impede achievement of targeted health outcomes.

Over the past decade, eHealth technology has helped with the delivery of educational and motivational self-management support. One possible strategy to empower patients to self-manage their condition is by providing them with structured information about their health profile (i.e., feedback) to stimulate setting specific goals [10]. Previous studies reported that personalized feedback helps with improvement of patients’ adherence to risk-reducing behaviors (diet, physical activity) [11,12,13]. Personalized rather than general feedback received more attention by patients as they could develop expectations according to their own health targets and set goals by making informed choices about risk assessment and lifestyle modification [14]. Employing two behavioral change techniques of feedback and goal setting, this study contributes to the understanding of the quality of patient-formulated self-management goals (hereafter self-defined goals) by testing the effectiveness of providing feedback using a personalized health profile. The objective of this study was to estimate, among people living with HIV, to what extent providing feedback on their health outcomes, compared to no feedback, will affect the number and specificity of self-defined goals.

Methods

The protocol for this trial has been previously published [15]. We used the reporting guideline for pragmatic trials [16]. The study was registered on November 25, 2019 on Clinicaltrials.gov (NCT04175795).

Study design

This was a blinded, stratified, and pragmatic randomized controlled trial.

Participants

This study was a sub-study of the Positive Brain Health Now (+BHN) studyFootnote 2 across five HIV sites in Canada [17]. Participants for this study were HIV+ individuals who were enrolled in the BHN study with two assessments (first and most recent), agreed to be contacted for further sub-studies, and had access to internet. Briefly, cohort members were adults of ≥ 35 years who were HIV+ for at least 1 year. People with dementia, co-morbidities affecting cognition, substance abuse, or life-threatening illnesses were excluded.

Intervention

The intervention tested in this study was providing feedback on modifiable contributors of health and quality of life. For this goal, a Dashboard comprised of 15 different actionable items (selected from pre-existing validated measures [18]) was developed for all BHN participants (Fig. 1). Details on the development and usability of the Dashboard has been published elsewhere [18]. Cognitive interviewing with 15 seropositive individuals from Montreal and Vancouver showed that the Dashboard was deemed helpful as a feedback tool. Our hypothesis was that the Dashboard helps participants to realize aspects of their life that are amenable to intervention and set specific goals.

Procedures

All eligible participants were first contacted by designated trial staff either by phone or e-mail. After providing their e-mail addresses, participants received an e-mail with trial information, their unique access code (token), and the link to access the goal setting survey platform (in French and English). The intervention group received feedback in the form of their personal health profile (hereafter Dashboard) via e-mail along with instructions on goal setting and tips to improve brain health. The control group received only the goal setting instructions and tips. The brain health tips included general advice about factors that have an impact on brain health such as sleep, social engagement, etc.

Data collection was conducted over six months from January 2020 until mid-August 2020 through a web-based goal setting platform (LimeSurvey hosted on a McGill server). Upon electronically signing the consent form (Extended data) [19], participants were directed to write their goals (in free text) and answer questions that followed each goal. Participants were encouraged to think of the top 3 to 5 actions they would like to take to improve their health condition in the assigned boxes. Following each goal, a section was assigned for perceived barriers and enablers. Participants were also asked to rate, on the scale from 1 (low) to 10 (high), the importance and difficulty of their goals as well as their perceived self-efficacy in achieving the goals.

Participants were given a maximum of 2 weeks to answer and submit the survey (a reminder e-mail was sent after the first week). A second reminder e-mail was sent to those who did not answer the survey within the two weeks. At the end of the study (4 weeks/participant), participants in the control group received their Dashboard via e-mail. Communication with participants were only via research assistants. There was no in-person visit for this project. Step-by-step instruction as to how fill out the survey was also made as a PDF file in English and French for participants who needed some guidance for filling out the survey. A brief post-survey assessment for evaluation of intervention compliance was also conducted.

This study was approved by institutional research ethics (McGill University Health Center Research Ethics Board, Canada, ABHN_Goals 2020-5728).

Outcomes

We tested the hypothesis that providing feedback, using the Dashboard, results in formulation of more specific goals. The two main outcomes were the number and specificity of self-management goals. For testing goal specificity, text mining techniques were used. This outcome is reported as a discrete variable—number of specific words (matched to a domain-specific lexicon that we created) per person-words. It was expected that the intervention group will set more goals and have more words matching the lexicon than the control group. The numerator for this estimator was total number of specific words per group; the denominator was the sum of cumulated number of words per person.

Regardless of the goal setting approach, SMART criteria [20]—Specific, Measurable, Actionable, Realistic, and Time-bound—have been widely used for goal formulation. There are no standardized and validated measuring criteria for goal formulation. To test goal quality outside the clinical setting where goals are self-defined and most often lack the structure and format of clinically elaborated SMART goals, techniques of text mining can be helpful. In this study we used text mining techniques to extract and understand specific goal criteria from the textual data.

Text mining procedure

Python programming language and relevant libraries were used for the text mining process. Text mining included three phases of preparation (defining a goal setting lexicon), preprocessing, and classification. Details on these phases has been provided in Attachment File.

To check the reliability of text mining results, 20% of the goals were randomly selected and manually evaluated by 5 experts in the field of rehabilitation. Manual evaluation involved experts making their own judgment regarding the classification of words. Each expert evaluated 20 goals and were all blinded to the lexicon. Interrater reliability (between the raters and text mining classification) was tested using Cohen’s Kappa.

To understand goal areas and participants’ priorities, topic modeling was tested which did not provide meaningful information. This could be due to short length of sentences and corpus volume. Instead, thematic analysis was performed to identify goal areas. Emerged themes were then mapped to the International Classification of Functioning, Disability and Health (ICF) [21]. Top 10 goal areas for both groups were ranked.

Statistical analysis

Descriptive statistics were used to analyze the characteristics of the participants as well as total number of goals (first outcome) set by each group. The second outcome—goal specificity—was defined as the rate of specific words per person-words. Text mining process provided quantifiable data for this purpose—cumulated number of specific words and total words used by each group in their goals. Thus, the analysis aimed to compare this rate across groups. Using negative binomial regression, we estimated the rate ratio and 95% confidence intervals. Negative binomial was a more appropriate model to use due to overdispersion of our count data (count of specific words). Pearson Chi2 dispersion statistics for the Poisson model had a ratio of 2.17 (presenting greater variability) and the negative binomial model showed a value of 1.02 for the corresponding entry.

Cognitive ability in terms of memory, information processing, or comprehension, etc. is one component that could affect goal formulation [22]. In order to see whether cognitive ability of participants affected goal specificity, cognition was added to the model as an interaction term. Data on this variable were gathered through the BHN study. Cognitive ability was measured using a computerized test—Brief Cognitive Ability Measure (B-CAM) [23]. We have used the most recent B-CAM score of the participants measured in 2020. B-CAM is a continuous variable with a maximum score of 36, with higher values reflecting better cognition.

No gender-based analysis was done because of small count for this variable to have any meaningful information. The proportion of women in this study was ~ 10% (n = 6 per group). Thus, gender was not modeled in the regression equation:

$${\text{Log }}\left( {E\left( {{\text{Rate}}\,{\text{of}}\,{\text{Goal}}\,{\text{Specificity}}} \right)} \right) = \left( {{\text{groups}}} \right) + \left( {{\text{B-CAM}}\,{\text{scores}}} \right) + \left( {{\text{groups}}*{\text{B-CAM}}\,{\text{scores}}} \right)$$

Correlations between goal specificity (rate of specific words/person-words—continuous variable) and participants’ education and cognition were also calculated. Polyserial correlation was calculated with education (5 levels) and Pearson correlation with B-CAM (continuous). Distribution of ranks for the common goal areas between groups was tested using Wilcoxon rank sum test. All analyses were done using SAS 9.4 software (SAS Institute Inc.).

Sample size

The estimated sample size for this trial was 210 per arm based on an estimated rate ratio of 1.5 (calculated based on results a pilot study), power of 80%, and response rate of 20% [24]. This sample size was calculated to provide a 95% confidence interval that excludes 1 (i.e., 1.23, 1.83). Based on the available sample of > 800 HIV + men and women enrolled in the BHN this sample size deemed practical. However, only 350 participants confirmed their consent to enroll in further sub-studies under the BHN.

Due to the COVID-19 pandemic and closure of outpatient clinics, recruitment was temporarily shut down for nearly 4 months (March to June 2020). Recruitment restarted in July with slow progression due to remote work and ongoing closure of some sites. Therefore, recruitment terminated by mid-August. This study received a high response rate of 60% (three times more than the initial assumption). Recalculating estimated confidence interval for the rate ratio of 1.5 and 30% loss to follow-up with sample of 176 participants provided a 95% confidence interval that excludes 1 (i.e., 1.09, 2.03).

Randomization and masking

With equal allocation ratio (1:1) participants were randomized into one of two groups, the intervention or control group, stratified based on gender. The randomization was computer generated by the study statistician using randomization.com. Given the intervention for this study was participants’ personalized health profile, it was not possible to keep participants blinded to group assignment. To avoid bias in assessing outcomes, those collected and analyzed data were kept blind to group assignment.

Results

Figure 2 shows the study flow diagram. The available BHN sample who agreed to be enrolled in the BHN’s sub-studies was 350. This sample was from four sites, two in Montreal, one in Vancouver, and one in Toronto. A total of 176 (87 English/89 French) participants were included in the trial and randomly assigned to the feedback group (n = 88) and control group (n = 88). Non-participation was due to not having access to the Internet or lack of interest in the study. Of those randomized, nearly 40% (n = 66) of the participants dropped out—17 withdrew from the study and 49 did not respond to the survey. This led to a total of 97 participants. Responses of 13 more participants were also added to the analysis because they had filled out but not submitted the survey. This was after confirmation with participants. Thus, a total of 110 survey responses were analyzed (56 English/54 French).

Fig. 1
figure 1

Dashboard

Full size image
Fig. 2
figure 2

Flow diagram for “goal setting in HIV” trial

Full size image

Characteristics of participants were similar between the two groups with the exception of co-morbidities and living status. Presence of additional condition aside from HIV was more common in the feedback group (55 [80%] vs 34 [63%]) compared to the control group. Living alone was less common among the feedback group (23 [41%] vs 29 [54%]) than the control group (Table 1).

Table 1 Characteristics of the participants
Full size table

Control group responded to the survey faster than the feedback group (average of 8.3 days ± 8.3 vs 10.2 days ± 9.1). For both groups, the majority of responses were received after the first reminder.

Number and characteristics of the goals

Table 2 shows goals’ characteristics. A total of 421 goals were formulated (227 in English/194 in French). Average number of goals was similar for both groups (3.7 for feedback group vs 3.9 for control group). Equal combination of outcome (focus on result) and behavioral goals (focus of performance) were set by feedback group. Less than 10% of the goals were mixed goals (mix of outcomes with some intended behaviors on different topics). Also, less than 10% of the goals were unclear and could not be categorized. Control group, on the other hand, set more behavioral goals than outcome goals (45% vs 37%) and 2% of their goals were vague and unclear.

Table 2 Characteristics of the goals
Full size table

Formulated goals were of highly important to both groups with mean goal importance of 8.5 out of 10. Perceived difficulty for the set goals were average for both feedback and control group (5.8 vs 5.3 out of 10, respectively). Feedback group’s perceived self-efficacy was slightly higher than control group (7.4 vs. 6.9).

Table 3 shows the top 10 goal areas defined by each group. Both groups had almost similar goal areas as shown by overlaps in 8 goal areas and little difference in rank. These 8 goal areas were consistent with 7 items on the Dashboard. More than 50% of the goals for both groups were mainly around self-care, managing diet and fitness, and cognition. Other common goal areas were mobility, sleep, handling stress, regulation of emotions, and socializing. Wilcoxon rank sum test for 8 common goal areas showed a difference in ranking of these areas between the two groups (p = 0.02). This difference in rank was related to two goal areas—socializing and mobility. Socializing was more observed in goals defined by feedback group than control group (ranked 7 vs 3.5) while mobility was mainly seen in goals set by control group (ranked 6.5 vs 4). This difference was consistent with participants’ profile. Dealing with negative feeling such as depression and anxiety was reported by 54% of feedback group, while mobility issues such as climbing stairs were more prevalent in control group (61%). Table 4 shows proportion of participants below the optimal level for Dashboard items. Detailed description of goal areas for each group has been provided in Supplementary Table 1.

Table 3 Top 10 goal areas and shared goal areas between groups
Full size table
Table 4 Proportion of participants below the optimal level on Dashboard items
Full size table

Text mining output

Total number of unique words (words remained after removing stop words) used by feedback group was 2187 compared to 2502 for the control group—a difference of 245 words. The shortest goal for both groups was formulated using one word which was only a broad definition of goal domain (e.g., diet, exercise). The longest goals set by the feedback and control group had 60 and 47 words, respectively. Total number of specific words matching the goal setting lexicon was also more for the goals set by the control group (739) compared to the feedback group (642). Proportion of specific words used by both groups were ~ 30%. Average number of specific words/goal was similar for both groups (3.5 vs. 3.1). Specific nouns and actionable verbs were present to some extent. Goals were mainly missing “measurable” and “time-bound” words in both groups (Table 2).

Manual evaluation of 20% of the goals (100 goals) achieved a moderate degree of agreement with text mining output—Cohen’s Kappa of 0.63 (95%CI 0.53, 0.73). Interrater agreement for sub-section of goals (20 goals) rated by each expert also showed a moderate degree agreement with Cohen’s Kappa ranging from 0.54 to 0.72.

Goal specificity

Both crude and estimated rate ratio for goal specificity have been presented. Table 5 shows rate ratio of specific words (goal specificity) calculated using total number of words/person as the denominator. Negative binomial regression showed no difference in goal specificity among groups. Rate ratio of specific words/person-words was 0.93 (95% CI 0.78–1.10) showing the rate of specific words for the feedback group was almost similar (though slightly lower) to the control group. In other words, the Dashboard did not improve goal quality. Adding cognitive ability as an interaction term to the regression equation did not improve predictability of the model (RR = 1.13, 95% CI 0.45, 2.82). The correlation between goal specificity and education (r = 0.21, 95% CI 0.02, 0.38) and cognition (r = 0.17, 95% CI − 0.01, 0.35) was weak (Table 6).

Table 5 Negative binomial regression analysis on number of specific words for feedback and control groups with and without an interaction term
Full size table
Table 6 Correlation between specific words/person goals and B_CAM and education
Full size table

Post-survey assessment with the feedback group (response rate of 61%) showed degrees of non-compliance with the intervention. While all reported they had read their Dashboard, only 54% used it for goal setting. Despite variability in intervention uptake, no difference in goal specificity was observed.

Post-survey assessment

Participants in the feedback group reported satisfaction with the Dashboard and perceived it as an easy and nice visual presentation of brain health and health-related quality of life. For some participants the Dashboard was a proof for their ideas on goals and some found it an inspiration to share with their partners, families, or friends.

Discussion

In this randomized controlled trial, feedback by means of a personalized health profile did not improve number and specificity of the goals set by HIV+ adults in Canada. Participants in the control group seemed eager to receive their Dashboard as seen by their faster reply to the survey. While variability in levels of adherence seen in the feedback group did not affect the outcome of interest in this trial, it informed of participants’ low engagement with their Dashboard. The issue of “low engagement” with online or digital interventions have been previously reported particularly when no obvious benefit is perceived by the user [25]. Also, active engagement with digital feedback seems to happen when feedback is received at the time of need, i.e., “when needed” [26]. It could be that, at the time of the study, participants did not consider dashboard information “as needed.” Nonetheless, in this trial a pragmatic approach was chosen to determine whether the intervention works under the usual condition of our target group. Similar performance of both groups in terms of goal priorities highlights participants’ insight of their health situation. Regardless of group assignment, participants goal areas corresponded to 7 items on the Dashboard. These areas were those that more than 50% of participants in both groups were below the optimal level.

To our knowledge, goal quality as a primary outcome of personalized feedback interventions has not been studied before particularly in the context of chronic conditions. Previous studies on communication of personalized feedback showed mixed results. For example, systematic reviews led by Krebs et al. [13] and Teasdale et al. [12] both showed positive, though small, effect of tailored digital feedback (in forms of reports or SMS) on improvement of several lifestyle behaviors, such as physical activity and diet modification. Contrary to these results, visual feedback or personalized estimates of a disease risk (e.g., cardiovascular diseases, cancer) made no difference on uptake of risk-reducing behaviors [27, 28]. Despite contradictory findings, potential benefit of personalized feedback for shared and informed decision-making and reducing ambivalence was acknowledged [29].

In this study, we considered Dashboard as a promising feedback tool to improve goal setting. Items on the Dashboard are fundamental factors influencing health-related quality of life [30, 31] and could primarily be modified by non-pharmacological management including self-management. Goal setting plays a dominant role in self-management by laying a roadmap pointing where one stands and where they are heading [32]. The Dashboard provides the basic information for setting goals by providing a reference point to patients’ HRQoL according to their visit data. Offering optimal range on the Dashboard gives a choice to the individual to decide whether they want to make a change. Having a choice is empowering and allows the person feels they are in control of their decisions and have the ability to switch between their options [33, 34].

This study has a number of strengths and limitations. The main strength of this trial is its new methodological approach for measuring goal quality. Techniques of text mining extracted useful and meaningful information from unstructured goal data in a shorter period of time and informed goal criteria. It also helped with expansion of goal setting lexicon (from 694 words representing four goal criteria to 994 words) by identification of unclassified words. Even though we did not use topic detection for identification of goal domains, text tokens alone were informative of participants’ most common health-related concerns. Word cloud for French and English goals are provided in supplementary materials. Finally, while goal setting lexicon has been tested and expanded with HIV population, it is relevant to other chronic health conditions. Text mining approach used in this trial as well as the domain-specific (goal setting) lexicon created in this study could be useful for the future trials using text mining for evaluation of voluminous patient-defined textual data. The trial had high retention rate and the randomized controlled design supports the strength of the findings. Population characteristics in the two groups were similar. Contamination was not an issue due to personalized nature of the Dashboard.

One limitation of this trial is its low sample size. As explained earlier, closure of HIV clinics due to COVID-19 pandemic interfered with recruitment. A bigger sample size would have helped with precision of text mining algorithms and further expansion of the goal setting lexicon, although change in the main outcome remains unlikely on the basis of low participation. Another limitation concerns complexities of natural languages. Classification of words for different goal criteria was based on the guideline defined by our research team. Other researchers or language experts might disagree, particularly with classification of some “nouns” and “verbs” and get different results for goal specificity. Evaluation of text data is based primarily on human judgment and is not exempt of measurement errors. As shown in our trial, manual evaluation of goals by different raters provided different agreement estimates. Raters or guidelines are not considered as reference standard [35], yet a consensus-based application orientated guidelines—as the case in our trial—lead to a uniform approach and evaluation of data.

To conclude, based on our findings there is a necessity to scale up feedback tools to help with SMART goal setting. Text mining has the potential to help with difficulties of goal setting in a non-face-to-face setting. A larger patient-formulated goal data would be needed to provide a real-time information on what words are commonly being used and what goal criteria are missing. This could potentially lead to a unique and valuable dataset of its kind. Future comparative studies could highlight the strengths and weaknesses of using text mining for both goal evaluation and goal setting.