http://orcid.org/0000-0002-4069-5746
Figures
Abstract
The Low Back Activity Confidence Scale (LoBACS) assesses the self-efficacy to perform activities in individuals with chronic low back pain (CLBP). As self-efficacy appears to directly influence the patient’s functional capacity and prognosis, it is important to develop a scale that evaluates this attribute to guide treatment strategy and monitor the clinical course of patients. This study aimed to evaluate the reliability, construct validity, and responsiveness of the Brazilian version of the LoBACS. The scale was applied to 112 male and female patients (age, 18–65 years) with specific and nonspecific CLBP. For evaluating the interobserver reliability, the scale was applied twice on the first evaluation day by two trained evaluators (A and B). Within 48–72 h of the first evaluation, assessor A reapplied the scale to evaluate intraobserver reliability (test–retest), which was analyzed by intraclass correlation coefficient (ICC). The first LoBACS applied in the baseline evaluation was also used to assess the construct validity of the scale by factor analysis. For responsiveness, the scale was applied 5 times at 2-week intervals and the change in scores was analyzed by the repeated measures ANOVA. Although factor analysis indicated three subscales, they did not present acceptable values of convergent and divergent validity. Reliability ranged from good to excellent, with ICC values of .90 (95% CI, .84; .93) and .85 (95% CI, .77; .91) for inter- and intraobserver variability for total score. Moreover, the total score was responsive in all comparisons, with no floor or ceiling effects. Thus, only the total score of the Brazilian version of LoBACS proved to be reliable, valid, and responsive.
Citation: Dias JM, Silva MF, Dela Bela LF, Pelegrinelli ARM, Olkoski MM, Guenka LC, et al. (2020) Assessment of measurement properties of the Brazilian-Portuguese version of the low back activity confidence scale (LoBACS) in patients with chronic low back pain. PLoS ONE 15(9): e0239332. https://doi.org/10.1371/journal.pone.0239332
Editor: Alison Rushton, Western University, CANADA
Received: January 22, 2020; Accepted: September 4, 2020; Published: September 22, 2020
Copyright: © 2020 Dias et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The original study was partially funded by the Foundation for Physical Therapy (American Physical Therapy Association) and the present study by a grant from CNPq (National Council for Scientific and Technological Development – 14/2013), a productivity scholarship for the last author, Araucaria Foundation/PPSUS/SESA-PR/MSDecit/CNPq [grant number # 04/2013], and by the Coordination for the Improvement of Higher Education Personnel (CAPES – Brazil) – [Finance Code 001]. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
In Brazil, chronic low back pain (CLBP) affects nearly 18.5% (95% confidence interval [CI], 17.8–19.1) of individuals >18 years of age, with 16.4% of them presenting with severe disability [1]. The global economic burden from the disease is highly significant, with about $200 billion annual spend [2].
CLBP is attributed to intricate biopsychosocial mechanisms and multiple risk factors [3, 4]. Several studies have evaluated the relationship between CLBP and self-efficacy (individual’s confidence in their ability to perform a task) and found that cognitive tests, such as assessment of functional self-efficacy, have greater explanatory power than specific pain-related assessments [5–9]. In addition, associations were found between low self-efficacy and higher disability rates [7, 10]. Patients with CLBP and low self-efficacy are about 113% more likely to present with disability than patients with high self-efficacy [11]. Moreover, psychological factors have been established as predictors of chronicity in patients with low back pain (LBP) [12].
Recently, some studies have evaluated self-efficacy as a treatment method for patients with LBP [13–15]. Khodadad et al. compared the effectiveness of cognitive functional approach and lumbar stabilization and found no difference between the methods [14]. Both methods were able to improve pain and movement control [14]. Ferrari et al. evaluated the effect of physical therapy centered on cognitive and behavioral principles on patients with lumbar spondylolisthesis and CLBP. Their results indicated an improvement in lumbar function, pain self-efficacy, and clinical tests [15].
Self-efficacy seems to be an important outcome in evaluating patients with CLBP as it can predict disability [5–11], adherence to treatment [16], and the likelihood of chronicity [12]. Therefore, it is important to develop a self-efficacy evaluation questionnaire that can assist in providing better physical and biopsychosocial treatment for these patients.
In Brazil, this outcome has been used to assess different health conditions and for specific populations. However, there are no questionnaires formulated specifically for patients with LBP or for evaluating the self-regulation domain [17–20]. It is also well known that questionnaires formulated in a foreign language are not suitable for use in all populations because of differences in beliefs, definitions, and lifestyles in different cultures [21, 22]. The Low Back Activity Confidence Scale (LoBACS) was translated into and cross-culturally adapted to Brazilian–Portuguese [23]. The changes made during this adaptation require evaluation of measurement properties to verify both internal and external validity. The present study aimed to evaluate the reliability, construct validity, and responsiveness of the Brazilian version of LoBACS in patients with CLBP.
Materials and methods
This is a measurement property evaluation study [22, 24]. All changes made during the process of evaluation of LoBACS into Portuguese were authorized by the authors of the original scale. Additionally, this study was approved by the Universidade Estadual de Londrina–Institutional Review Board #17918313.4.0000.5231.
Sample
A consecutive sampling method was used. The sample was selected through television and radio advertisements. Patients were assessed for inclusion criteria by two physical therapists trained for evaluation. A medical diagnosis of CLBP (more than 12 weeks) was requested [25]. Inclusion criteria were individuals with specific and nonspecific CLBP, of both genders, aged between 18 and 65 years, and referred to physical therapy. Patients were excluded if they had fibromyalgia, ankylosing spondylitis, or rheumatoid arthritis; were immediately postoperative for lumbar spine or lower limbs or refused to sign the informed consent.
The sample size calculation was based on the goodness of fit test through the G*Power program and accounted for a 0.4 effect size, an error of 5%, a power (1-β) of 95%, and two degrees of freedom. The sample size was 110 patients, considering 10% of dropouts.
Instruments
LoBACS is a tool that measures the self-efficacy of patients with LBP, can be self-applicative, and consists of 15 items divided into three subscales: Functional self-efficacy (FnSE) composed of seven items–related to the confidence to perform functional activities; Self-regulatory self-efficacy (Self-RegSE) consisting of three items, which assesses the confidence to handle and control problems related to the lumbar spine, and Exercise self-efficacy (ExSE), consisting of five items related to the confidence and proactivity to perform physical exercises [26, 27]. This instrument consists of a Likert scale of 11 points, ranging from 0% (no confidence) to 100% (complete confidence) with increments of 10% for each point. Scale scoring can be calculated globally (e.g., the average score of all items), or domain by domain (e.g., the average score of items 1–7 for FnSE subscale), and the total score may range from 0 to 100, where the higher the score, the better the self-efficacy [26]. When a questionnaire item is not answered, it should be excluded from the average in the final calculation.
The process to create the Brazilian-Portuguese version of the LoBACS was performed according to the already established protocol [28]: Preparation and Forward translation—four professionals who are native speakers of Portuguese with English as a second language conducted an independent translation from English to Portuguese (T1; T2; T3; T4); Reconciliation—the Portuguese versions were analyzed and discussed, with discrepancies documented in summary, to produce a single version. At this stage, some unit of measurement amendments were performed; for example, the use of kg instead of lb; Back translation—the single version in Portuguese was submitted to a new translation into English by three translators (BT1, BT2, and BT3), who were native English speakers with a second language of Portuguese; Back translation review and Harmonization—back translated versions were compared with the original scale to obtain a pretesting version (PTV) in Portuguese [23].
This PTV was applied to a selected sample of 37 individuals. They were questioned about their understanding, doubts, and suggestions during the Cross-cultural adaptation stage or Cognitive debriefing. After this, a Review of cognitive debriefing results was conducted, and if appropriate, amendments were made. Modifications were as follows: Question 1 –change the word “box” to “object”; Question 2 –change the word “phone book” to “book”; Question 3 –change the word “carpet” to “rough surface (such as carpet)”; Question 4 –the word “plane” was removed; and Question 5 –the number of steps in a flight of stairs was included. After this stage, Proofreading and Final report of the final Brazilian-Portuguese version of the scale was obtained (S1 File) [23].
Procedures
The reliability of the Brazilian version of LoBACS was evaluated by applying it twice on the first evaluation day: two protocol-blinded and trained evaluators (A and B) applied the LoBACS with an interval of two hours (inter-observer reliability). After the first assessment, between 48 and 72 hours, assessor A reapplied the scale to evaluate intra-observer reliability (test-retest). The first LoBACS-Brasil applied in the baseline evaluation was also used to evaluate the construct validity of the scale by factor analysis.
To assess responsiveness, the scale was applied five times, over eight weeks of treatment, with two-week intervals between each application, and the change in the score was analyzed. The sequence of treatment was showed in the S1 Table.
Data analysis
Numerical variables were assessed for the distribution of normality by the Shapiro-Wilk test and are presented as medians (Md) and quartiles (25–75%). Categorical variables are presented as absolute and relative frequencies. For evaluation of intra- and inter-rater reliability, the intraclass correlation coefficient (ICC3,1) two-way mixed model, with 95% CI, which is based on one-way ANOVA, was used. The standard error of the measurement (SEM) was calculated using the formula = SD x √(1-CCI), where the SD was derived from the mean square of the residuals [29]. Reliability was considered poor when the value of the ICC was < .40, good when .40 ≤ ICC ≥ .75, and excellent for values >.75 [30].
Davenport et al. evaluated latent factors of the original scale through principal component analysis and found 3 factors [27]. In this study, the quality of the structure found was tested using confirmatory factor analysis (CFA) through the maximum likelihood estimation with robust standard errors, followed by a mean and variance adjusted test statistic (WLSMV). The model was adjusted until acceptable values were found in these fit tests: comparative fit index (CFI), Tucker-Lewis index (TLI), and root mean square error of approximation (RMSEA). For CFI to TLI the values were considered great when > .95, traditional when > .90, and sometimes permissible when > .80. For RMSEA the values were good when < .05, moderate between .05-.10 and bad when >.10 [31].
Both the reliability and validity of the scale structure were evaluated through the following tests: composite reliability (CR) (>.70), average variance extracted (AVE) (>.50), and maximum shared variance (MSV) (MSV < AVE) [32]. The systematic error of the data was analyzed by the common method bias through the common latent factor.
Repeated measures ANOVA was performed to assess responsiveness. The Mauchly test of sphericity was used, and when it was violated, technical corrections were performed using the Greenhouse-Geisser test. When the F test was significant, the Bonferroni test was used. Both floor and ceiling effects were evaluated descriptively through the 5th and 95th percentiles. Statistical significance was set at 5%. All analyses were performed using IBM® SPSS® 25.0 and R Project for Statistical Computing (lavaan package).
Results
In total, 112 patients were evaluated for both reliability and construct validity of the Brazilian version of LoBACS. There were no missing data for these patients. The median age of the included patients was 48 (39.5–55.5) years, and the median duration of symptoms was 96 (36–210) months. Additional information about the study sample is presented in Table 1.
Regarding reliability, all the values were rated as excellent, except for the assessment of the intraobserver reliability of the Self-RegSE factor, which was considered good (Table 2).
The 3 factors found at the EFA were evaluated for their suitability using confirmatory factor analysis (CFA) [27]. Three models were set up to obtain a robust model (Fig 1). The results of the model fit tests were as follows: ᵪ2(81) = 179.79; P≤ .001; CFI = .95; TLI = .93 and root mean square error of approximation (RMSEA) = .10.
Regarding CR, the results were .85, .66, and .77 for FnSE, Self-RegSE, ExSE subscales, respectively. The Self-RegSE subscale presented problems of convergent validity, and the AVE value was .49. The AVE values for the FnSE and Self-RegSE subscales were .57 and .50, respectively. Subscales also presented divergent validity problems, with MSV values higher than AVE values for FnSE (.77), Self-RegSE (.77) and ExSE (.71). Regardless of this bias, LoBACS-Brasil presented satisfactory results in the comparison of standardized regression weights of the model, without versus with the common latent factor.
For the assessment of responsiveness, only 48 patients were evaluated because the remaining patients discontinued treatment for personal reasons. When the total score was evaluated, statistically significant differences were found in comparisons between the baseline evaluation (BE) and all subsequent applications (BE x 2nd mean difference [MD] = −4.89; P = .03; BE x 4th MD = −9.18; P < .001; BE x 6th MD = −10.53; P < .001; BE x 8th MD = −13.25; P < .001). The same pattern was found for the FnSE subscale. Unsatisfactory responsiveness patterns were found for the Self-RegSE and ExSE subscales. The 5th and 95th percentile values were, respectively, 38 and 95 for total score; 1 and 99 for FnSE; 24 and 100 for Self-RegSE, and 13 and 100 ExSE. These results indicate ceiling effects for both Self-RegSE and ExSE subscales.
Discussion
The LoBACS-Brazil scale demonstrated excellent results for test–retest reliability, in line with the original study (FnSE, .92; Self-RegSE, .62; ExSE, .71; and total score, .85) [26]. The Self-RegSE subscale presented lower values of reliability in both the studies, and even lower values in the intrarater evaluation in our study. The inconsistent values can be explained by the subjective nature of the subscale and it being easily affected by the patient’s mood. Because data collection was performed on different days, patient mood could have varied. In addition, on the first evaluation day, the patients underwent few functional tests to provide timely treatment. This may have caused some pain to the patients, leading to the lower intrarater reliability results.
In our study the values in the model fit tests were acceptable. Furthermore, we found no bias in the database by the common latent factor, and the reliability as assessed by the composite reliability was adequate. However, subscales presented problems of both divergent and convergent validity, meaning that items did not correlate well with each other within their parent factor and that items correlated more highly with items outside their parent factor than with items within their parent factor. Hence, although the model indicates the existence of the three subscales, they are not independent of each other, thereby limiting their use.
Concerning responsiveness, we found statistical differences between comparisons of the BE and all other applications, for total score. These results indicate that the scale is responsive and that if the therapist applies the scale on the day of the BE, improvement of the patient’s self-efficacy throughout the treatment may be observed. The same pattern was observed for the FnSE subscale.
In contrast, the Self-RegSE and ExSE subscales did not present adequate standards of responsiveness. These findings can be justified by two reasons. First, the convergent and divergent validity of these subscales suggests that the LoBACS-Brasil should not be separated into subscales as it may lead to statistical problems. Second, the treatment to which the patients were submitted was not for self-efficacy and maybe that is why we did not find changes in the subscale scores. We could only observe improvements in aspects related to the treatment, specifically, the subscale related to functionality (FnSE) and total score. The problems of divergent and convergent validity may also explain the poor results related to ceiling effects of subscales.
Importantly, the original scale was validated in patients who recently received single-level lumbar microdiscectomy for the first time and the factorial analysis was performed in patients with post-acute LBP, which may result in differences found between the studies [26, 27].
Our study has some limitations. Although covariates such as the degree of pain and depression are known to influence self-efficacy [5, 26, 27, 33], they were not controlled in this study. Personal self-efficacy develops from sources such as previous experiences, observations, verbal/society persuasion, and physiological state [34] and is mediated by the interpretation of specific events [35]. Therefore, this cognitive factor is highly important and demonstrates how situations such as depression can influence self-efficacy [35]. In addition, the questionnaires were applied through interview because of the educational condition of the study sample, whereas the original version was self-administered. This difference could have influenced the results.
Implications for research
Further studies must control covariates such as pain and depression to clarify their effects on the LoBACS-Brasil score. The validity of the scale must be evaluated using a more specific tool. Prospective studies with a specific objective of self-efficacy improvement are required to provide better insights into responsiveness and to establish cutoff points for the scale.
Implications for practice
Self-efficacy is an important outcome to be evaluated in patients with CLBP, especially when considering the biopsychosocial model of the disease. Validation of LoBACS in Brazilian–Portuguese presents the professional with the possibility of a differentiated assessment of the patient, provides information on confidence in performing functional activities and coping strategy adopted by the patient, and predicts their possible adherence to exercise treatment. In addition, the use of objective and reliable tools assist in thorough evaluation, evidence-based prescription, and a systematic strategy to monitor the clinical course.
Conclusions
Only the total score of the Brazilian version of LoBACS proved to be reliable, valid, and responsive for use in patients with CLBP. Although factorial analyses indicate the existence of three subscales, these should not be used due to problems associated with convergent and divergent validity, responsiveness, and ceiling effects.
Supporting information
S1 File. Low Back Activity Confidence Scale–LoBACS-Brasil.
https://doi.org/10.1371/journal.pone.0239332.s001
(PDF)
S4 File. LoBACS-Brasil_ Responsiveness database.
https://doi.org/10.1371/journal.pone.0239332.s004
(SAV)
S1 Table. Aquatic and Deep-Water Running Exercises Program.
https://doi.org/10.1371/journal.pone.0239332.s005
(PDF)
References
- 1.
Instituto Brasileiro de Geografia e Estatística (IBGE). Pesquisa nacional de saúde 2013: percepção do estado de saúde, estilos de vida e doenças crônicas. Rio de Janeiro:2014.
- 2. Ma VY, Chan L, Carruthers KJ. Incidence, prevalence, costs, and impact on disability of common conditions requiring rehabilitation in the United States: stroke, spinal cord injury, traumatic brain injury, multiple sclerosis, osteoarthritis, rheumatoid arthritis, limb loss, and back pain. Arch Phys Med Rehabil. 2014; 95(5): 986–995. pmid:24462839
- 3. Delitto A, George SZ, van Dillen L, Whitman JM, Sowa G, Shekelle P et al. Low back pain clinical practice guidelines linked to the international classification of functioning, disability, and health from the orthopaedic section of the American Physical Therapy Association. J Orthop Sports Phys Ther. 2012; 42(4): 1–57.
- 4. Ramond A, Bouton C, Richard I, Roquelaure Y, Baufreton C, Legrand E et al. Psychosocial risk factors for chronic low back pain in primary care: a systematic review. Fam Pract. 2011; 28(1): 12–21. pmid:20833704
- 5. Lackner JM, Carosella AM, Feuerstein M. Pain expectancies, pain, and functional self-efficacy expectancies as determinants of disability in patients with chronic low back disorders. J Consult Clin Psychol. 1996; 64(1): 212–220. pmid:8907101
- 6. Lackner JM, Carosella AM. The relative influence of perceived pain control, anxiety, and functional self-efficacy on spinal function among patients with chronic low back pain. Spine (Phila Pa 1976). 1999; 24(21): 2254–2260.
- 7. Vieira EBM, Salvetti MG, Damiani LP, Pimenta CAM. Self-efficacy and fear avoidance beliefs in chronic low back pain patients: coexistence and associated factors. Pain Manag Nurs. 2014; 15(3): 593–602. pmid:23891180
- 8. Riley SP, Bialosky J, Coronado RA. Are changes in fear-avoidance beliefs and self-efficacy mediators of function and pain at discharge in patients with acute and chronic low back pain? J Orthop Sports Phys Ther. 2020; 50(6): 301–308. pmid:31905094
- 9. Ferrari S, Vanti C, Pellizzer M, Dozza L, Monticone M, Pillastrini P. Is there a relationship between self-efficacy, disability, pain and sociodemographic characteristics in chronic low back pain? A multicenter retrospective analysis. Arch Physiother. 2019; 12(9): 1–9.
- 10. Woby SR, Urmston M, Watson PJ. Self-efficacy mediates the relation between pain-related fear and outcome in chronic low back pain patients. Eur J Pain. 2007; 11(7): 711–718. pmid:17218132
- 11. Salvetti MG, Pimenta CA, Braga PE, Corrêa CF. Disability related to chronic low back pain: prevalence and associated factors. Rev Esc Enferm USP. 2012; 46(spec): 16–23.
- 12. Pincus T, Burton KA, Vogel S, Field, AP. A systematic review of psychological factors as predictors of chronicity/disability in prospective cohorts of low back pain. Spine. 2002; 27(5): E109–E120. pmid:11880847
- 13. O’Neill A, O’Sullivan K, O’Sullivan P, Purtill H, O’Keeffe M. Examining what factors mediate treatment effect in chronic low back pain: a mediation analysis of a cognitive functional therapy clinical trial. Eur J Pain. 2020; Online ahead of print.
- 14. Khodadad B, Letafatkar A, Hadadnezhad M, Shojaedin S. Comparing the effectiveness of cognitive functional treatment and lumbar stabilization treatment on pain and movement control in patients with low back pain. Sports Health. 2020; 12(3): 289–295. pmid:31841078
- 15. Ferrari S, Vanti C, Costa F, Fornari M. Can physical therapy centred on cognitive and behavioural principles improve pain self-efficacy in symptomatic lumbar isthmic spondylolisthesis? A case series. J Bodyw Mov Ther. 2016; 20(3): 554–564. pmid:27634078
- 16. Granlund B, Brulin C, Johansson H, Sojka P. Can motivational factors predict adherence to an exercise program for subjects with low back pain? Scand J Behav Ther. 1998; 27(2): 81–96.
- 17. Rabelo DF, Cardoso CM. Self-efficacy, chronic diseases and functional disability in the old age. Psico-UFS. 2007; 12(1): 75–81.
- 18. Salvetti MG, Pimenta CAM, Lage LV, Oliveira JO Jr., Rocha RO. Self-efficacy and depressive symptoms in chronic pain patients. Rev Psiquiatr Clín. 2007; 34(3): 111–117.
- 19. Sbicigo JB, Teixeira MAP, Dias ACG, Dell’Aglio DD. Psychometric properties of the general perceived self-efficacy scale (EAGP). Psico. 2012; 43(2): 139–146.
- 20. Yassuda MS, Lasca VB, Neri AL. Metamemory and self-efficacy: a validation study of research instruments in memory aging. Psicol Reflex Crit. 2005; 18(1): 78–90.
- 21. Reichenheim ME, Moraes CL. Operationalizing the cross-cultural adaptation of epidemiological measurement instruments. Rev Saude Publica. 2007; 41(4): 665–673. pmid:17589768
- 22. Wild D, Grove A, Martin M, Eremenco S, McElroy S, Verjee-Lorenz A et al. Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (pro) measures: report of the ispor task force for translation and cultural adaptation. Value Health. 2005; 8(2): 94–104. pmid:15804318
- 23.
Dela Bela LF. Translation and cultural adaptation of the Low back activity confidence scale (LoBACS) to Brazilian-Portuguese. M.Sc. Thesis, Universidade Estadual de Londrina. 2014.
- 24. Kottner J, Audigé L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A et al. Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. J Clin Epidemiol. 2011; 64(1): 96–106. pmid:21130355
- 25. Hayden JA, van Tulder MW, Malmivaara A, Koes BW. Exercise therapy for treatment of non-specific low back pain. Cochrane Database Syst Rev. 2005; 20(3): CD000335.
- 26. Yamada KA, Lewthwaite R, Popovich JM Jr, Beneck GJ, Kulig K; Physical Therapy Clinical Research Network. The Low Back Activity Confidence Scale (LoBACS): preliminary validity and reliability. Phys Ther. 2011; 91(11): 1592–1603. pmid:21852520
- 27. Davenport TE, Cleland JA, Yamada KA, Kulig K. Measurement Properties of the Low Back Activity Confidence Scale (LoBACS). Eval Health Prof. 2016; 39(2): 204–214. pmid:24686745
- 28. Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976). 2000; 25(24): 3186–3191.
- 29. Weir JP. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Cond Res. 2005; 19(1): 231–249. pmid:15705040
- 30.
Fleiss JL. The design and analysis of clinical experiments. 1st ed. New York: Wiley; 1986.
- 31. Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Modeling. 2009; 6(1):1–55.
- 32.
Hair J, Black W, Babin B, Anderson RE. Multivariate data analysis. 7th ed. Upper Saddle River: Prentice-Hall Inc.; 2010.
- 33. Salvetti MG, Pimenta CAM, Braga PE, McGillion M. Prevalence of fatigue and associated factors in chronic low back pain patients. Rev Lat Am Enfermagem. 2013; 21(Spec): 12–19.
- 34.
Amaral JC. Self-efficacy, self-regulation and performance in achievement of cognitive tasks. M.Sc. Thesis, Applied Psychology Institute; 1993.
- 35.
Flavell J, Ross L. Social cognitive development: frontiers and possible futures. 1st ed. Cambridge: Cambridge University Press; 1981.