School health assessment tools: a systematic review of measurement in primary schools

Protocol and registration

The review protocol of the study used the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) guidelines. This review was registered with the International Prospective Registry of Systematic Reviews (PROSPERO) (Registration ID: CRD42020158158) before starting the research according to the PRISMA guideline recommendation (Moher et al., 2009).

Eligibility criteria

The criteria for the selection of the studies included in this systematic review were that the studies: (1) developed or evaluated an instrument to measure school health; (2) the instruments should measure the school health factors; (3) evaluated at least one of the psychometric properties described in COSMIN Risk of Bias checklist for a school health instrument; (4) the population of the study should be primary school children; (5) were original research journal articles (i.e., book chapters, thesis and case studies were excluded); and (6) were in English. All articles that were not available online in full text were also excluded from the study.

Search procedures

Two of the authors (Maryam Kazemitabar and Mojtaba Habibi) performed, independently from each other, a systematic search strategy to find articles with the specified criteria. The articles were divided into three groups of “completely related”, “somewhat related”, and “not related”. The authors selected the articles based on the relatedness of the title and abstract of the articles, then the full texts of those placed in the “completely related” and “somewhat related” groups were studied. In case of any disagreement, the second author (Ali Moghadamzadeh) intervened until consensus was reached.

The studies were searched through the EBSCO(host), ProQuest, PubMed, and Wily databases, also a gray literature search was performed in PROSPERO, OpenGrey, and Google Scholar. No publication date was considered. Each database was queried from November 2019 using the Boolean operators (AND/OR). The terms searched were: “school health”, “health-promoting schools”, “coordinated school health”, “psychometric properties”, “reliability”, “validity”, “tool”, “scale”, “index”, “instrument”, “evaluation”, and “questionnaire”. We have also conducted hand search on the references lists of the selected studies. Table 1 shows the databases and their search algorithms applied to the search strategy.

We have sent an Email to the corresponding author of one of the articles to access the full-text of it, but we have not received any reply so it has been excluded from this Systematic Review. The search for articles was finalized by December 2019. A total of 649 studies were identified for this review.

Study selection

From the 649 articles, we excluded all articles that did not report the psychometric properties of the instrument (cf. Brener, Pejavara & McManus, 2011; Chen & Lee, 2016; Burt et al., 1996; Weiler & Pigg Jr, 2004) or that did not includ middle or high school students (Kristiansen, Holmstrom & Olofsson, 2016; Yun et al., 2018; Sagatun et al., 2019). After this procedure, seven articles were selected as clearly relevant for inclusion in the systematic review.

Data extraction and processing

The selected studies (n = 7) were assessed by the modified COSMIN checklist, which was developed to assess the methodological quality of studies on measurement properties. The original COSMIN checklist contains 10 parameters. In the modified version of the COSMIN checklist used in this systematic review, however, the parameters “criterion validity” and “responsiveness” were omitted because none of the studies reported these parameters. Thus, eight parameters were used in the present systematic review: PROM development, content validity, structural validity, internal consistency, cross-cultural validity/measurement invariance, reliability, measurement errors, and hypotheses testing for construct validity.

Content validity refers to the degree to which the items in an instrument are an adequate reflection of the construct to be measured. Structural validity is the degree to which the scores of an instrument are an adequate reflection of the dimensionality of the construct to be measured (Mokkink et al, 2010). Cross-cultural validity refers to the degree to which the performance of the items on a translated or culturally adapted instrument are an adequate reflection of the performance of the items of the original version of the instrument. Internal consistency refers to the degree of interrelatedness among the items and is often assessed by Cronbach’s alpha (Prinsen et al., 2018) and more recently by using coefficient Omega (Nima et al., in press). Reliability refers to the overall consistency of a measure, a measure has a high reliability that results in similar scores under consistent conditions; in other words, a test is reliable if it gives the same repeated result under the same conditions. Measurement error is the difference between a measure quantity and its true value. There are two types: systematic error and random error. The appropriate statistics for calculating the measurement error are Standard Error of Measurement, Limits of Agreement, and Smallest Detectable Change (Prinsen et al., 2018). Hypotheses testing for construct validity shows the extent of consistency between scores of the measure with hypotheses. It can be measured by comparing the instrument’s score to other instruments or difference between relevant groups; in other words, studies that performed convergent and discriminative (divergent) validity or known-groups validity would be appraised as “very good”.

The option “not applicable” applies for studies that have not investigated a specific psychometric property because it has not been applicable in that study and therefore were not included for scoring. Each parameter’s check box has items to assess each specific psychometric property using a four-point scale (“very good”, “adequate”, “doubtful”, and “inadequate”). The overall quality of each psychometric property is operationalized as the lowest rating of any standard in the box is taken (i.e., “the worst score counts” principle) (Mokkink et al., 2018; Prinsen et al., 2018; Terwee et al., 2018). In this study each “very good” rating received score 1 and the options “adequate”, “doubtful”, and “inadequate” were received score 0. That is, the maximum score for a study was 8. The final quality of any individual study was rated based on high quality for any study with 50% of the scores or more, moderate quality for 30% to 50%, and low quality for below 30% of the scores.

A customized data extraction form was developed by the first reviewer (Maryam Kazemitabar), and then characteristics data of the studies and participants were extracted; the second reviewer (Mojtaba Habibi) also extracted the data independently, the results of the two extractions were compared and in case of any differences the third reviewer (Ali Moghadamzadeh) intervened to reach consensus. The data regarding characteristics of the selected studies included: first author name, year of publication, country, sample size, time to answer the items, number of items, response scale, participant age/grade, participant sex, and instrument factors.

The instruments studied in this systematic review were: Scale for Health-Promoting Schools (SHPS), Quality of Life in School, School Health Policies and Program Study 2000 Questionnaire, Healthy School Indicator Tool (HSIT), and School Health Evaluation Instrument.

Study selection

The search strategy was conducted throughout 6 data bases: EBSCOhost, PubMed, ProQuest, Wily, PROSPERO, and OpenGrey. This resulted in 952 articles; and then in 649 after removing the duplicates. After the screening of titles and abstracts 515 articles were selected; then the full-text of the articles were studied for investigating the eligibility criteria including original journal articles, articles that studies at least one of the COSMIN Risk of Bias checklist measure properties of a school health assessment tool, study’s population should be primary schools, articles of English language and articles with full-text availability. After this procedure, seven articles remained for inclusion in this systematic review (Fig. 1).

Study characteristics

The following descriptive characteristics of the articles included in this systematic review was extracted: first author name, publication year, country, sample size, time to answer the items/number of items, response scale (multiple/forced choice, Likert scale, fill-in-the blank, true or false, yes or no, and text response), participant age/grade, participant sex, instrument’s factors (Table 2).

Risk of bias of school health measurement properties

Table 3 shows the modified COSMIN risk of bias checklist for the studies included in this systematic review. In the modified version of the checklist “criterion validity” and “responsiveness” were omitted because none of the studies reported these parameters. In this modified version, the term “tool development” substituted the term “Patient Reported Outcome Measure (PROM) development” from the original COSMIN checklist, in order to reflect the real target population of the studies which are students, teachers, and principals instead of “patient”.

The articles were assessed by three raters who agreed that none of the seven articles scored “high quality”, three of the articles were assessed as “moderate quality”, and the remaining four articles as “low quality”.

Tool development

There are some requirements for developing an instrument. First of all the theory, framework or model used for developing the instrument should be clearly described, the target population should be specified. Context of use and qualitative data collection method should be clarified. The interviewer should be skilled and the data collected from qualitative study should be recorded and transcribed and coding section should be performed by at least two researchers independently. In addition a pilot study should be conducted to test the comprehensibility of the items, and the appropriate number of examinees for qualitative studies should be ≥7 and for quantitative studies ≥50 to score as “very good”.

Four articles of the seven articles included in this review (Weintraub & Erez, 2009; Lee et al., 2014; Andrews & Conte, 2005; Pinto et al., 2016) were development studies; three of the studies’ quality were assessed as “adequate” (Weintraub & Erez, 2009; Lee et al., 2014; Pinto et al., 2016) because there was not enough information whether the coding was performed independently, if the data collection continued until saturation, and if the group meetings have been recorded and transcribed. The other article (Andrews & Conte, 2005) was scored as “very good” because it fulfilled all criteria in the “tool development” box of the COSMIN checklist, including a clear description of steps for developing the instrument such as five focus groups indulging in determining indicators, table of specifications, creating items and response format, and conducting a pilot study. They have also mentioned that members of the advisory group have independently studied the indicators as well as reviewing relevant literature.

Content validity

According to the checklist the study should examine items both in an expert group and the target population to determine relevance and comprehensiveness by skilled interviewers. Other criteria are similar to the “tool development” parameter.

Five of the seven studies examined content validity of the instrument at hand. Three of them have met these criteria and received “very good” (Lee et al., 2014; Andrews & Conte, 2005; Pinto et al., 2016). One of the studies used the Delphi method and pilot study for evaluating content validity (Pinto et al., 2016); another has used focus group and a pilot study as the method of validation (Andrews & Conte, 2005); one study has examined only the comprehensibility of the items from students’ points of view and did not evaluate content validity by asking professionals, so it was scored as “inadequate” (Weintraub & Erez, 2009), and the last study used only experts and did not perform a pilot study and therefore was scored “inadequate” (Öncel & Sümen, 2018).

Structural validity

Measuring structural validity is only applicable for instruments that are based on a reflective model, something that all instruments included in this systematic review were. The studies that have conducted confirmatory factor analysis and had a sample size that was seven times the number of items and ≥100 get scored as “very good”, studies with exploratory factor analysis and a sample size at least five times the number of items and ≥100 or the sample size at least six times the number of items but <100 get scored as “adequate”.

All of the seven articles have investigated the structural validity of the instrument at hand. Three articles have performed confirmatory factor analysis with a sufficient sample size (seven times the number of items) and were scored as “very good” (Öncel & Sümen, 2018; Weintraub & Erez, 2009; Lee et al., 2014), three studies have performed exploratory factor analysis (Ghotra et al., 2016; Pinto et al., 2016; Andrews & Conte, 2005) and were assessed as “adequate”; the last one was assessed as “inadequate” because it has not performed confirmatory or exploratory factor analysis (Brener, Kann & Smith, 2003).

Internal consistency

In COSMIN Risk of Bias checklist any study that measures internal consistency for each of the subscales independently is considered to have a “very good” quality and any study presenting only the total internal consistency or not calculating the internal consistency at all, receives the score “inadequate”. Five studies have calculated Cronbach’s alpha for each subscale or unidimensional scale and were therefore evaluated as “very good” (Öncel & Sümen, 2018; Weintraub & Erez, 2009; Lee et al., 2014; Pinto et al., 2016; Ghotra et al., 2016), one of the studies was scored as “doubtful” because internal consistency was unclear and was not fully reported (Andrews & Conte, 2005), the last study was scored as “inadequate” because the internal consistency has not been calculated (Brener, Kann & Smith, 2003).

Cross-cultural validity/measurement invariance

Cross-cultural validity or measurement invariance is needed for a translated or culturally adapted instrument to compare performance of the items to the original version of the instrument. The samples should be similar for relevant characteristics except for the group variance; the sample size should be 200 subjects per group to be scored as “very good” and 150 subjects per group to be scored as “adequate”. Cross-cultural validity assesses if the instrument is Measurement Invariance and not Differential Item Functioning occurs; this indicates that examinees from different groups with the same latent trait level act invariant to an item. Appropriate methods for Classical Test Theory (CCT) studies are regression analyses or confirmatory factor analysis, while for Item-Response Theory (IRT), the appropriate method is Differential Item Functioning (Teresi et al., 2009).

Only two of the studies could be investigated for cross-cultural validity. One study was scored as “adequate”, because it did not report comparison between different groups clearly (Ghotra et al., 2016), and the other study has not reported cross-cultural validity at all (Öncel & Sümen, 2018).

Reliability

As the reliability value should be stable on the construct to be measured between repeated test administrations, the time interval should be appropriate, the test condition should be similar to original instrument (e.g., administration, environment, and instructions).

Moreover, the intraclass correlation coefficient should be calculated for continuous scores. For dichotomous/nominal/ordinal scores Cohen’s kappa should be calculated, for ordinal scored weighted kappa should also be calculated and it is recommended to describe weighting scheme for ordinal scores (Prinsen et al., 2018).

Two of the studies received score “very good” because one of them has performed split-half reliability (Pinto et al., 2016) and the other has performed a test-retest reliability for continuous data and Kappa has also been calculated for dichotomous scores (Brener, Kann & Smith, 2003). The rest five studies were scored as “doubtful” because they have not presented a clear description about stability across respondents, time interval, similar conditions or calculating the intraclass correlation coefficient.

Measurement error

Conditions for measurement error is similar to reliability, the same examinees, appropriate time interval, and similar conditions for measurements. Standard Error of Measurement, Small Detectable Change or Limits of Agreements should be calculated. Percentage (positive and negative) agreement should be calculated for dichotomous/nominal/ordinal scores. Calculation of Standard Error Measurement through Cronbach’s alpha is not appropriate, because it does not take the variance between time points into account and it should be calculated by a test-retest design (De Vet et al., 2006). None of the included studies calculated Standard Error of Measurement, Limits of Agreement or Smallest Detectable Change in the instruments being used in each study.

Hypotheses testing for construct validity

Hypothesis testing for construct validity requires comparisons of the instrument under development with other outcome measurement instruments (convergent validity) through Pearson correlation and describing distribution of scores or mean scores. Performing confirmatory factor analysis or structural equation modeling over scales are also proper methods to measure differences between similar constructs. Discrepancy in scores among ‘known groups’ (discriminative or known group validity study) should be also measured. In discriminative or known group validity study significant characteristics of the groups under study such as age, gender, language etc. should be reported.

Only one of the studies (Ghotra et al., 2016) has calculated construct validity and was assessed as “very good” because it reported both convergent validity and discriminative validity for the instrument under investigation. The rest of the studies have not reported construct validity by comparing the school health instruments with another instrument or throughout known group validity.

Implications for school health measurement

Our findings suggest that measuring all aspects or factors of school health is not widespread. Most of the studies have measured just one, two or some factors related to school health. Measuring health at school is necessary, since it affects the policies and decisions of stakeholders of schools. Further investigations are required into measuring comprehensive and precise psychometric testing on existing tools, as well as advanced development of new tools considering appropriate statistical methods for measuring cross-cultural validity, reliability, measurement error, and construct validity. Nevertheless, among the existing tools, the Scale for Health-Promoting Schools (Lee et al., 2014) and the School Health Evaluation Instrument (Pinto et al., 2016) are the best available options for primary schools in terms of methodological evaluation. In fact, Lee et al. (2014) instrument has a good validity and Pinto et al. (2016) instrument has a good reliability. Therefore, although they require more evaluations regarding afore mentioned measurement properties, we recommend researchers, scholars and stakeholders to apply them in primary schools and to continue the psychometric testing and development of these instruments.

Limitations

In sum, we found only a few instruments assessing school health. What is more, there are only a few studies measuring the psychometric properties of these instruments. Importantly, this small number of studies have not considered some of the measurement properties mentioned in the COSMIN checklist. Thus, there is not enough evidence on the quality of these instruments. This systematic review suggests that researchers need to conduct more studies on school health assessment tools in order to acquire better conclusions on these instrument’s measurement properties.