http://orcid.org/0000-0002-5606-2046
Figures
Abstract
Objective
EQ-5D-3L is a generic QOL tool used mainly in economic evaluations. Burden of Chronic Kidney Disease (CKD) is rising in Sri Lanka. Assessing the validity of generic QOL tools creates new opportunities of their utilization among patients with CKD.
Methods
A cross-sectional study was conducted among 1036 CKD patients, selected using the simple random sampling technique. The validity was tested with six a-priori hypotheses. These included construct validity assessments, evaluating convergent validity and performing known group comparisons. EQ-5D-3L, Short Form-36 (SF-36) were used to assess QOL. Center for Epidemiological Studies Depression Scale (CES-D-20) and General Health Questionnaire-12 (GHQ-12) were used to assess the presence of depression and psychological distress respectively. Internal consistency of the whole tool and when each item is removed was assessed by Cronbach alpha.
Results
The response rate was 99.2%. Majority of participants were males (n = 646,62.4%) in the age category of 41–60 (n = 530; 51.2%). Most were in either stage 4 or 5 of CKD (n = 646,75.1%). The summary measures of SF-36, positively and significantly correlated with the EQ-5D-3L index and VAS scores (p<0.001). EQ-5D-3L QOL scores were significantly different between the group with depression and without as measured by CES-D-20 (p<0.001). Assessed using GHQ-12, similar significance was detected between the group with psychological distress and without (p<0.001). The Cronbach alpha was 0.834 and when each item was removed, ranged from 0.782 to 0.832.
Citation: Senanayake S, Mahesh PKB, Gunawardena N, Graves N, Kularatna S (2019) Validity and internal consistency of EQ-5D-3L quality of life tool among pre-dialysis patients with chronic kidney disease in Sri Lanka, a lower middle-income country. PLoS ONE 14(6): e0211604. https://doi.org/10.1371/journal.pone.0211604
Editor: Senaka Rajapakse, University of Colombo Faculty of Medicine, SRI LANKA
Received: January 16, 2019; Accepted: June 12, 2019; Published: June 26, 2019
Copyright: © 2019 Senanayake et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The study was funded by the Ministry of Health, Sri Lanka [MRI/40/2015] to SS. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Chronic Kidney Disease (CKD) has become a major global burden of disease accounting for a significant mortality and disability adjusted life years [1]. In the Global Burden of Disease Study-2015, it was ranked as the “12th most common cause of death” with an increased overall mortality over 30% in the previous decade [2]. This rise has been “alarming” in the low- and middle-income settings [3]. Additional emphasis on CKD has been recommended in relation to these settings owing to the existing deficiencies in health policies and practices [2].
Sri Lanka which is an agricultural lower-middle-income country, is affected by the rising trend of chronic non-communicable diseases (NCDs) due to demographic and epidemiological transitions [4,5]. CKD has become a major burden of the healthcare system in Sri Lanka [6]. With the contribution of occurrence of unexpected types of it like CKD of unknown etiology as well as due to the concomitant increase of incidence of other NCDs, its burden is expected to rise further [6.7]. In addition, CKD impose numerous social and economic threats in Sri Lanka [8].
Quality of life (QOL) reflects a person’s subjective evaluation of his or her position of life in the living contexts [4]. The QOL related to impact of health conditions is referred to as health-related QOL [9]. Measurement of QOL become utmost important among patients with CKD, due to the problems they are forced to face by the disease condition. They face a diverse range of symptoms, distresses and even depression [1,10]. Furthermore, the condition would make them suffer from social and financial inabilities as well. They need to be in a potential waiting period for getting the dialysis done, even when they enter that stage.
Health-related QOL can be measured by using generic as well as disease specific tools [11]. Short-Form-36 (SF-36 and) EQ-5D-3L are such generic QOL tools [12]. Assessment of validity of QOL tools, for specific disease conditions would create many opportunities for utilization within specific settings [13]. This is particularly beneficial for generic QOL tools, as this allows QOL comparisons between different conditions [14]. Validity measures to what extent the tool measures what it is expected to measure [15,16]. Reliability reflects the reproducibility of the same concept when repeated measures are observed. It can be assessed by methods including internal consistency and test-retest method [17].
In the absence of gold standard tests for measurements like QOL, the validity assessments could be done for construct validity with methods such as “convergent/divergent validity assessments” and “known group comparisons” [13,15,18]. The EQ-5D-3L health states have been valued using preference of general population in Sri Lanka [19]. Furthermore using it, health related QOL has been assessed in relation to several main chronic NCDs in Sri Lanka [4,20]. Yet the validity of it, in relation to CKD has not been assessed.
The purpose of this study is to assess the validity and the internal consistency of EQ-5D-3L among patients with kidney diseases who are in their pre-dialysis stage.
Materials and methods
Selection of the sample
A descriptive cross-sectional study was conducted in Anuradhapura district of Sri Lanka. This district, located in the North Central Province, is experiencing a rising trend of CKD. Patients with CKD confirmed by a medical specialist based on the recommended guidelines (ie. Glomerular Filtration Rate (GFR) being less than 60ml/min per 1.73m2 body surface area in two measurements made three months apart) comprised the study population. All patients above 18 years were included. Patients who had renal transplantations were excluded.
The sample size (n) for the estimation of the construct validity was based on the publication of Cohen (1992) for the “product-moment correlation coefficient”[21]. Cohen (1992) recommends a sample size of 783 for a small effect size [21]. With an assumed response rate of 75%, minimal sample size was calculated as 1044. Patients with CKD in the district of Anuradhapura are registered in a population-based CKD register according to the geographic health units in which they reside. The district comprise 19 health units and the required number of study units were allocated to the 19 health units in proportionate to the number of registered patients in each health unit. Using the population based register as the frame, simple random sampling technique was used to select the study units from each the 19 health units within the district.
Assessment of validity and internal consistency
The construct validity of the EQ-5D-3L was assessed using the following six a-priori hypotheses.
- The EQ-5D-3L index scores would significantly correlate with SF-36 summary scores in the positive direction with acceptable strength
- The EQ-5D-3L Visual Analogue Scale (VAS) scores would significantly correlate with SF-36 summary scores in the positive direction with acceptable strength
- The EQ-5D-3L index scores would be significantly different between the groups with depression and without depression
- The EQ-5D-3L VAS scores would be significantly different between the groups with and without depression
- The EQ-5D-3L index scores would be significantly different between the groups with and without psychological distress
- The EQ-5D-3L VAS scores would be significantly different between the groups with and without psychological distress
The internal consistency was assessed using Cronbach alpha measurement for all the items as well as when each item is removed.
EQ-5D-3L developed by the EuroQol group consist of a descriptive system and a VAS. The descriptive system comprises of five questions (on mobility, self-care, daily activities, pain and mood) with each carrying three levels of possible responses. Each level reflects a level of impairment as “with no problem”, “with some problems” and “with severe problems”. An index score can be obtained from these questions which is expressed as a negative or positive fraction. VAS may range from 0 to 100 and reflects the general health as perceived by the participant [22]. EQ-5D-3L has been previously translated to Sinhalese language and population norms has been developed in 2014 (19). Other tools used comprised interviewer administered forms of SF-36, CES-D-20 and GHQ-12, in addition to the tool to assess socio-demographic and medical details.
SF-36 was used in the assessment of a-priori hypothesis 1 and 2. It includes 36 items out of which 35 are used to calculate eight QOL domain scores. Four each of these domains give rise to the “physical-component” and “mental-component” summary measures. SF-36 focuses on the period of previous 28 days in getting the responses [23]. Its validity has been assessed and proved to be satisfactory for several conditions in Sri Lankan setting [24,25]. Furthermore it has been used in this setting to explore QOL of several conditions [4,20].
The presence of depression was screened by the CES-D. It has 20 questions and a maximum score of 60 is allocated. A score above 15 is indicative of depression. Its sensitivity is over 80% and the specificity is over 90% in relation to the local setting [26]. In screening for psychological distress, GHQ-12 questionnaire was used. It gives a maximum score of 12. A cut-off value of two or more has been recommended to the local setting with a sensitivity and specificity more than 70% [27].
Data analysis
Since the normality assessment showed non-normal distributions non-parametric techniques were used in the analysis. For the a-priori hypothesis 1 and 2, Spearman correlation coefficient was used [28]. For the a-priori hypotheses No. 3 to 6, Mann Whitney U test was used [29]. The significance level was considered as 5%, but when lower p values were observed, relevant significant levels were reported.
Ethics approval was obtained from the Ethics Review Committee of the Faculty of Medicine, University of Colombo prior to the data collection (EC-15-081). Informed written consent was taken from participants.
Results
The response rate was 99.2%. The median (IQR) age of the participants was 59 (52 to 66) years. More than half (51.2%) the population were between 41–60 years old. The majority were males (62.4%) and were unemployed (64.7%). Of those who are not employed, 48.0% were females and of them 77.2% are housewives who are not formally employed. Nearly four fifth were in stages of 4 or 5 in the CKD staging. Of the population 64.2% were screening positive for depression while 74.4% were positive for psychological distress (Table 1).
The descriptive system of the EQ-5D-3L comprises of five questions (on mobility, self-care, daily activities, pain and mood) with each carrying three levels of possible responses. Except in the self-care domain, in all other four domains, the majority had stated level 2 or 3 responses (i.e. either having some or major problems). For the pain and the mood domains, responses with some-or-severe problems were reported by more than three fourth. In addition, in more than half of the participants, the mobility and the usual activities have been affected to a greater or lesser degree (Table 2).
Role-limitation-emotional domain had recorded lowest scores out of the eight domains of the SF-36. The mental-component summary score has recorded a slightly higher score than that of the physical-component. Mean EQ-5D-3L index score was 0.52 (SD 0.33). Median scores of all the eight domains of both physical health and mental health summary components had the scores below 50.0. The highest mean score was for the Vitality domain (45.29; SD 11.53) while the lowest was for the Role-limitation-emotional (23.94; SD 33.25) (Table 3).
The Spearman correlation coefficients between SF-36 summary measures versus the EQ5D-3L index and VAS scores were statistically significant (p<0.001) as well as positive in direction, even though being lower in strength (Table 4). The index values showed relatively stronger associations than the VAS scores.
The comparison of the EQ-5D-3L and SF-36 scores between the depressed and non-depressed groups were carried out to test the 5th a-priori hypotheses (Table 5). It shows that statistically significant differences are observed between the QOL measures between the two groups as detected by both the tools (p<0.001).
Similar statistically significant differences are shown between the groups with and without psychological distress (p<0.001) (Table 6).
Data on validity were disaggregated by age and education level to assess whether validity is affected by age or education level (S2 Appendix and S3 Appendix). Analysis indicated that the results do not change depending on different age categories (age < 60 versus age > 60) and different education levels (Education level up to Grade 5 versus Education level above Grade 5
The internal consistency reflected by the Cronbach alpha was 0.834. The alpha values when each item was removed ranged from 0.782 to 0.832.
Discussion
This is the first study documenting the validity of the EQ-5D-3L among pre-dialysis patients with CKD. All the six a-priori hypotheses were found to be true in relation to this generic QOL tool. Hence this study pave the way towards opportunities of using EQ-5D-3L on patients with CKD, which is becoming a rising burden in this lower-middle income setting.
Higher proportion (i.e. more than 75%) of participants has responded as having some-or-severe problems in relation to the pain and mood domains. This is compatible with the other documented literature depicting that mental well-being of CKD affected participants must be given more emphasis [10]. Additionally, it is in par with the findings of the present study with 75% being psychologically distressed and nearly 65% being depressed.
All the individual summary scores had means and medians less than 50. The figures are in general lower than the post-myocardial infarction and post-stroke QOL measurements in Sri Lanka [4,20]. Comparatively the VAS values of the EQ-5D-3L too has remained around the value of 50, further proving the negative impact, CKD imposes on the lifestyles of the affected patients.
Swank and Mullen (2017) state that in interpreting convergent validity by this method, must consider three factors; significance, direction and the effect size [28]. The Spearman correlation coefficients showed a significant association between the SF-36 summary scores versus EQ-5D-3L index and VAS scores in the positive direction. This shows that the parameters of EQ-5D-3L have measured the relevant constructs in the similar manner as that of SF-36 proving its construct validity. However, the effect sizes of the associations were approximately in the range of 0.2 to 0.4. This range can be classified as “acceptable and of medium strength”[28,30]. The relatively weak strengths of associations are acceptable due to the high number of other factors affecting the variability of these outcome parameters. Furthermore, due to the complexities of constructs, the interpretation of findings in this regard is recommended to be not exactly similar to interpreting other bivariate correlations [31].
The index values showed relatively stronger associations compared to VAS figures. This may be explained by the fact that the index scores include more domains whereas the VAS score is a single general estimation of the participant. Since the SF-36 summary scores too include multiple domains in them, they can be assumed to be more strongly correlated with the EQ-5D-3L index score.
Both the depressed and psychologically distressed groups had relatively lower EQ-5D-3L scores compared to their counterparts. This proves that the EQ-5D-3L is valid in differentiating two groups which are known or assumed to be different in relation to QOL. The depressed and distressed people would perceive the position of their life at a lower level than those who are not. Hence the QOL of depressed and distressed groups can be assumed to be lower. In the present study EQ-5D-3L has been able to detect this difference. The SF-36 values too were mentioned to prove that actually there was a difference of QOL between the two groups compared.
In the current study 64.6% of the participants were unemployed. Evidence indicate that the unemployment is associated with psychological distress and depression and one would expect the validation results to be affected by the high unemployment status of the study population. However, of those who were not employed, 25.1% were housewives who have not been employed in their lifetime. Of the unemployed males 57.5% were retired. Thus authors believe that the high unemployment rate in the study population is unlikely to be a source of psychological stress to a majority of distressed in the study population. Thus it is unlikely that the high unemployment would have any impact on the validation results.
The study population is from an area of Sri Lanka which has a preponderance of Chronic Kidney Disease on Unknown etiology (CKDu) which affects a younger age group in agricultural communities. However, present study showed that the instrument is equally valid for both younger and older study groups (S2 Appendix) and different education levels indicating that the results are valid to all forms of CKD. Furthermore it is expected that the Visual Analogue Scales of the instrument would depend on the education level of the respondents. However, results disaggregated by age and education level (S3 Appendix) show that they do not differ depending on the educational status of the respondent. This indicate that EQ-5D-3L is equally valid among different education levels.
The highest internal consistency was seen when all the items are present in the tool as reflected by the Cronbach alpha values. This proves that even when all the five questions are included in the tool, homogeneity of the responses is preserved. The alpha value being more than 0.7, reflects a satisfactory internal consistency as categorized conventionally [32].
There were several limitations of the study. Firstly the reliability of the EQ-5D-3L was only tested using the internal consistency in the study. To measure the reliability by test-retest method, a clinically stable period is needed [33,34]. If inter-rater methods are used, a minimum time period which would eliminate the answers given to the previous responder is needed in between the data collections [35]. Since EQ-5D-3L captures the QOL at the “time of completion” [22], the administration of other methods becomes debatable. On the other hand internal-consistency, though not being the sole representation, could be assumes to reflect the reliability characteristics of tools [36]. However, to prevent any over-generalizability of the findings, in the title of the current study, the word “internal consistency” was used instead of the word “reliability”.
Secondly the QOL scores were not adjusted to the morbidities those were present among the participants. Majority of the participants were suffering from different comorbid conditions (nearly 71%) with different severities. Hence it was not feasible to adjust for these. However as shown in literature, concomitant non-communicable medical conditions are common among the patients with CKD [6,7] and the sample of this study too represent more or less similar characteristics.
Conclusions
EQ-5D-3L is a valid generic QOL tool with satisfactory internal consistency to be used among CKD patients in the pre-dialysis stage
Supporting information
S2 Appendix. Subgroup analysis according to age.
https://doi.org/10.1371/journal.pone.0211604.s002
(DOCX)
S3 Appendix. Subgroup analysis according to education level.
https://doi.org/10.1371/journal.pone.0211604.s003
(DOCX)
Acknowledgments
Authors acknowledge Asanga Ranasinghe, Priyantha Kumara and Anura Ranasinghe for the support rendered during the study.
References
- 1. Senanayake S, Gunawardena N, Palihawadana P, Bandara P, Haniffa R, Karunarathna R, et al. Symptom burden in chronic kidney disease; a population based cross sectional study. BMC Nephrol [Internet]. 2017 Dec 10;18(1):228. Available from: pmid:28693434
- 2. Neuen BL, Chadban SJ, Demaio AR, Johnson DW, Perkovic V. Chronic kidney disease and the global NCDs agenda. BMJ Glob Heal [Internet]. 2017 Jul 6;2(2):e000380. Available from: http://gh.bmj.com/lookup/doi/10.1136/bmjgh-2017-000380
- 3. Gifford FJ, Gifford RM, Eddleston M, Dhaun N. Endemic Nephropathy Around the World. Kidney Int Reports [Internet]. 2017 Mar;2(2):282–92. Available from: http://linkinghub.elsevier.com/retrieve/pii/S2468024916301681
- 4. Mahesh PKB, Gunathunga MW, Jayasinghe S, Arnold SM, Haniffa R, De Silva AP. Pre-event quality of life and its influence on the post-event quality of life among patients with ST elevation and non-ST elevation myocardial infarctions of a premier province of Sri Lanka. Health Qual Life Outcomes [Internet]. 2017 Dec 1;15(1):154. Available from: pmid:28764724
- 5. Mahesh PKB, Gunathunga MW, Jayasinghe S, Arnold SM, Mallawarachchi DS V., Perera SK, et al. Financial burden of survivors of medically-managed myocardial infarction and its association with selected social determinants and quality of life in a lower middle income country. BMC Cardiovasc Disord [Internet]. 2017 Dec 19;17(1):251. Available from: http://bmccardiovascdisord.biomedcentral.com/articles/10.1186/s12872-017-0687-y pmid:28927380
- 6. Rajapakse S, Shivanthan MC, Selvarajah M. Chronic kidney disease of unknown etiology in Sri Lanka. Int J Occup Environ Health [Internet]. 2016 Jul 2;22(3):259–64. Available from: https://www.tandfonline.com/doi/full/10.1080/10773525.2016.1203097 pmid:27399161
- 7. Navaratne KV, Gopalan S, Engelgau M, Okamoto K. Prevention and control of selected chronic NCDs in Sri Lanka: policy options and action. HNP Discuss Pap [Internet]. 2010;1(October):1–136. Available from: http://documents.worldbank.org/curated/en/2010/10/12932732/prevention-control-selected-chronic-ncds-sri-lanka-policy-options-action
- 8. Wijetunge S, Ratnatunga NVI, Abeysekera TDJ, Wazil AWM, Selvarajah M. Endemic chronic kidney disease of unknown etiology in Sri Lanka: Correlation of pathology with clinical stages. Indian J Nephrol [Internet]. 2015;25(5):274. Available from: http://www.indianjnephrol.org/text.asp?2015/25/5/274/145095 pmid:26628792
- 9. Thompson DR, Yu C-M. Quality of life in patients with coronary heart disease-I: Assessment tools. Health Qual Life Outcomes [Internet]. 2003;1(1):42. Available from: http://hqlo.biomedcentral.com/articles/10.1186/1477-7525-1-42
- 10. Senanayake S, Gunawardena N, Palihawadana P, Suraweera C, Karunarathna R, Kumara P. Depression and psychological distress in patients with chronic renal failure: Prevalence and associated factors in a rural district in Sri Lanka. J Psychosom Res [Internet]. 2018 Sep;112:25–31. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0022399918300758 pmid:30097132
- 11. Ware JE, Gandek B, Guyer R, Deng N. Standardizing disease-specific quality of life measures across multiple chronic conditions: development and initial evaluation of the QOL Disease Impact Scale (QDIS®). Health Qual Life Outcomes [Internet]. 2016 Dec 2;14(1):84. Available from: http://hqlo.biomedcentral.com/articles/10.1186/s12955-016-0483-x
- 12. Kotecha D, Ahmed A, Calvert M, Lencioni M, Terwee CB, Lane DA. Patient-Reported Outcomes for Quality of Life Assessment in Atrial Fibrillation: A Systematic Review of Measurement Properties. Quinn TJ, editor. PLoS One [Internet]. 2016 Nov 1;11(11):e0165790. Available from: http://dx.plos.org/10.1371/journal.pone.0165790 pmid:27802324
- 13. Muller AE, Skurtveit S, Clausen T. Validating the generic quality of life tool “QOL10” in a substance use disorder treatment cohort exposes a unique social construct. BMC Med Res Methodol [Internet]. 2016 Dec 23;16(1):60. Available from: http://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-016-0163-x
- 14. Frendl DM, Ware JE. Patient-reported Functional Health and Well-Being Outcomes With Drug Therapy. Med Care [Internet]. 2014 May;52(5):439–45. Available from: http://content.wkhealth.com/linkback/openurl?sid=WKPTLP:landingpage&an=00005650-201405000-00010 pmid:24714581
- 15. Wells GA, Russell AS, Haraoui B, Bissonnette R, Ware CF. Validity of Quality of Life Measurement Tools—From Generic to Disease-specific. J Rheumatol Suppl [Internet]. 2011 Nov 1;88:2–6. Available from: http://www.jrheum.org/cgi/doi/10.3899/jrheum.110906 pmid:22045972
- 16.
Abramson J.H., & Abramson Z. Survey methods in community medicine. 5th ed. Edinburgh: Churchil Livingstone; 1999.
- 17. Lin X-J, Lin I-M, Fan S-Y. Methodological issues in measuring health-related quality of life. Tzu Chi Med J [Internet]. 2013 Mar;25(1):8–12. Available from: http://linkinghub.elsevier.com/retrieve/pii/S1016319012000961
- 18. Banks JL, Marotta CA. Outcomes validity and reliability of the modified rankin scale: Implications for stroke clinical trials—A literature review and synthesis. Vol. 38, Stroke. 2007. p. 1091–6. pmid:17272767
- 19. Kularatna S, Whitty JA, Johnson NW, Jayasinghe R, Scuffham PA. EQ-5D-3L Derived Population Norms for Health Related Quality of Life in Sri Lanka. Bammann K, editor. PLoS One [Internet]. 2014 Nov 3;9(11):e108434. Available from: http://dx.plos.org/10.1371/journal.pone.0108434 pmid:25365171
- 20. Mahesh PKB, Gunathunga MW, Jayasinghe S, Arnold SM, Liyanage SN. Factors influencing pre-stroke and post-stroke quality of life among stroke survivors in a lower middle-income country. Neurol Sci. 2018;39(2):287–95. pmid:29103178
- 21. Cohen J. A power primer. Psychol Bull. 1992;
- 22.
Reenen M., & Oppe M. EQ-5D-3L user guide. EuroQol Research Foundation: Netherland.; 2015.
- 23. Ware JE, Snow KK, Kosinski M, Gandek B. SF-36 Health Survey Manual and Interpretation Guide. Bost New Engl Med Cent [Internet]. 1993;1 v. (various pagings). Available from: http://books.google.com/books/about/SF_36_health_survey.html?id=WJsgAAAAMAAJ
- 24. Gunawardena N.S., Seneviratne S.R.A. & Atauda T. An approach to validation of a multi-dimensional tool. J Coll Community Physicians Sri Lanka. 2003;8:18–26.
- 25.
Mallawarachchi DSV. Quality of life of stroke patients presenting to selected hospitals in the Colombo district and the possibility of common mental disorders among the principal informal caregivers. MD Community Medicine, Post Graduate Institute of Medicine, University of Colombo.; 2007. 2006 p.
- 26. De Silva V, Ekanayake S, Hanwella R. Validity of the Sinhala version of the Centre for Epidemiological Studies Depression Scale (CES-D) in out-patients. Ceylon Med J. 2014;
- 27. Abeysena H, Jayawardana P, Peiris U, Rodrigo A. Validation of the Sinhala version of the 12-item General Health Questionnaire. J Postgrad Inst Med. 2014;
- 28. Swank JM, Mullen PR. Evaluating Evidence for Conceptually Related Constructs Using Bivariate Correlations. Meas Eval Couns Dev [Internet]. 2017 Oct 2;50(4):270–4. Available from: https://www.tandfonline.com/doi/full/10.1080/07481756.2017.1339562
- 29. Hart A. Mann-Whitney test is not just a test of medians: differences in spread can be important. BMJ [Internet]. 2001 Aug 18;323(7309):391–3. Available from: http://www.ncbi.nlm.nih.gov/pubmed/11509435 pmid:11509435
- 30. Sink C, Stroh H. Practical Significance:The Use of Effect Sizes in School Counseling Research. Prof Sch Couns [Internet]. 2006 Apr;9(4):401–11. Available from: http://professionalschoolcounseling.org/doi/10.5330/prsc.9.4.283746k664204023
- 31.
Watson J. C., & Flamez B. Counseling assessment and evaluation: Fundamentals of applied practice. Thousand Oaks, CA: Sage; 2015.
- 32. Kim H, Ku B, Kim JY, Park Y-J, Park Y-B. Confirmatory and Exploratory Factor Analysis for Validating the Phlegm Pattern Questionnaire for Healthy Subjects. Evidence-Based Complement Altern Med [Internet]. 2016;2016:1–8. Available from: http://www.hindawi.com/journals/ecam/2016/2696019/
- 33. Paiva CE, Barroso EM, Carneseca EC, de Pádua Souza C, dos Santos FT, Mendoza López RV, et al. A critical analysis of test-retest reliability in instrument validation studies of cancer patients under palliative care: a systematic review. BMC Med Res Methodol [Internet]. 2014;14(1):8. Available from: http://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-14-8
- 34. Fayers PM, Machin D. Quality of Life: The Assessment, Analysis and Interpretation of Patient-Reported Outcomes: Second Edition [Internet]. Quality of Life: The Assessment, Analysis and Interpretation of Patient-Reported Outcomes: Second Edition. 2007. 1–544 p. Available from: http://www.scopus.com/inward/record.url?eid=2-s2.0-84889360388&partnerID=40&md5=2f81647008e54415dae40fe07a447902
- 35. Richardson J, Iezzi A, Khan MA, Maxwell A. Validity and Reliability of the Assessment of Quality of Life (AQoL)-8D Multi-Attribute Utility Instrument. Patient—Patient-Centered Outcomes Res [Internet]. 2014 Mar 23;7(1):85–96. Available from: http://link.springer.com/10.1007/s40271-013-0036-x
- 36. Kimberlin CL, Winterstein AG. Validity and reliability of measurement instruments used in research. Am J Heal Pharm [Internet]. 2008 Dec 1;65(23):2276–84. Available from: http://www.ajhp.org/cgi/doi/10.2146/ajhp070364