Developing fit-for-purpose self-report instruments for assessing consumer responses to tobacco and nicotine products: the ABOUT™ Toolbox initiative

Development and management of the ABOUT™ Toolbox

The ABOUT™ Toolbox has been developed using best measurement development practices and as the manifest of an underpinning behavioral conceptual model for TNPs.

Several guidelines, including the FDA’s “Guidance for Industry Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims”¹⁸, have been used as the foundation for the creation of the ABOUT™ Toolbox initiative^19,20. These guidelines provide the scientific basis for the development, modification, and validation of patient-reported outcome (PRO) measures in support of medical care research. Although not specifically designed for the tobacco regulatory research field, these recommendations are essential in outlining a wide range of development considerations, such as (1) defining the context of use and identifying the concept(s) of interest, (2) generating items that best capture the concept(s) of interest as expressed by the population of interest, (3) choosing the right response options and recall period, (4) evaluating the content validity of the instrument, and (5) assessing psychometric measurement properties for construct validity. These recommendations also cover the considerations around the adaptation of a self-report measure (e.g., for a different context of use, target population, or cultural group [see below the paragraph “cross-cultural equivalence”]).

The application of these best practices requires the use of mixed-methods research, which can be defined as “research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches, and methods in a single study or program of inquiry”²¹. As noted in a recent paper on rare disease patients²², qualitative methods alone are unable to inform us about the extent to which concepts are measurable. Conversely, quantitative methods alone cannot inform us about which concepts should be measured.

By applying these best measurement practices for the development of the ABOUT™ Toolbox (see Figure 1), the initiative further enhances tobacco regulatory research by combining five major components (described in the paragraphs below) that are paramount for rigorous instrument development.

Figure 1. Iterative process for the development of an ABOUT™ instrument.

1. Generation of a conceptual framework

The development of each instrument starts with the generation of a conceptual framework, which is grounded in theory and supported by the triangulation of evidence data from literature reviews, consumer input, and expert opinions. Furthermore, the development of each measurement instrument in the ABOUT™ Toolbox has been or is being conducted in close partnership with scientific experts from academic and commercial organizations with expertise in the fields of nicotine addiction, motivational aspects of consumer perception, and relevant areas on approaches to measurement (e.g., PROs, cross-cultural adaptation, psychometrics, regulatory submissions). The role of these experts is pivotal not only to provide input during the development of the conceptual framework but also to assist in the objective consensus-building process throughout the entire development of the instrument.

2. Evaluation of content validity

The main goal is to evaluate whether the instrument represents the concepts of interest, and the instructions and item content are appropriate, relevant, comprehensive, and understandable to the target population. This evaluation is performed in accordance with current good research practices^23,24. The content validity of the instruments included in the ABOUT™ Toolbox is supported by the execution of qualitative research with consumers (e.g., 25–27).

3. Use of an appropriate psychometric model

The psychometric assessment of most of the ABOUT™ instruments is based on the use of Rasch measurement methods (RMM)²⁸, supplemented by diagnostic evidence of the dimensionality and reliability properties rooted in classical test theory^29,30. RMM particularly investigate to what extent (1) the items are targeted for the type and range of issues to be measured³¹, (2) the items work well together as a set forming a unidimensional scale^31–33, (3) the response options are working as intended³⁴, and (4) the items do not show differential item functioning across various population groups or TNPs³⁵. This evaluation provides the necessary evidence that the conceptual framework has been converted successfully into a list of items, the responses to which can be summed to form a statistically sufficient total score (i.e., comprising all available information). The score can then be transformed into a linear measure that is comparable across population groups or TNPs, conceptually meaningful and substantively interpretable (e.g., 36,37). The final outcome of the application of RMM is the development of a calibrated scoring table that transfers sum scores to logit measures, which are mapped to a 0–100 scale for ease of interpretability. The conversion is a simple, linear transformation that changes the logit mean of 0 to 50 and converts most extreme measures to 0 and 100, respectively (e.g., 37). The purpose, description, administration, and scoring (including the calibrated scoring table) of the final validated version of an instrument is documented in a user manual that accompanies all the instruments of the ABOUT™ Toolbox.

4. Cross-cultural equivalence of the ABOUT™ instruments

In the context of the globalization of tobacco regulatory research, measures appropriate for use in different cultures are crucial. The demonstration of cross-cultural equivalence requires investigating that the instrument measures the same concepts in a comparable way across different languages and cultures^38–40. The first step toward cross-cultural equivalence is to ensure that rigorous translation procedures are followed. In the field of health outcomes research, recommendations to support the use of self-reported measures in multinational contexts have been provided^41,42. These recommendations apply to measures developed in one language and subsequently translated for use in other countries and cultures. In line with this guidance, the instruments included in the ABOUT™ Toolbox follow a thorough linguistic validation process consisting of two forward translations to the targeted new language, one back translation to the source language, and qualitative cognitive debriefing interviews with participants in the targeted language to ensure that the translations are understood (e.g., 43 and Figure 2)⁴⁴.

Figure 2. Linguistic validation process for the development of an ABOUT™ instrument.

The investigation and demonstration of the cross-cultural equivalence of the ABOUT™ instruments are completed by quantitative steps (e.g., 37,45), such as the evaluation of the psychometric properties of the translations and differential item functioning⁴⁶.

Translatability assessment (TA) is a technique that will be applied to all future ABOUT™ instruments. It is defined as the review of an original measure, preferably during the development stage, prior to its use, in order to determine its suitability for future translations in multilingual studies⁴⁷. TA can be viewed as the very first step toward ensuring measurement equivalence between the original measure and its future translations.

5. Appropriate access and use of the validated instruments (original and translations)

Easy, centralized, and appropriate access to original instruments and their translations is often a prerequisite to efficient research. With a unique point of access endorsed by the developers of instruments, researchers are able to access and use the original instrument and its translations⁴⁸. The centralization and control of access also enables the integrity of the each measurement instrument (original and translations) to be respected⁴⁹. Once the instruments are fully developed and validated (Steps 1 and 2 on Figure 1), they are made publicly available through the Patient-Reported Outcome and Quality of Life Instruments Database (PROQOLID™)⁵⁰, managed by Mapi Research Trust and part of the ePROVIDE™ web platform (https://eprovide.mapi-trust.org/).

To get access to the instruments distributed on ePROVIDE™, each new user has to complete a free registration form at https://eprovide.mapi-trust.org/login (click to the Free Registration button). The following link leads to a tutorial to help navigating through the whole registration process: https://eprovide.mapi-trust.org/tutorials/registering-for-free. Once the registration is completed, the register button will lead the user to his/her free ePROVIDE™ account. From there, any retrieval of instruments can be performed.

Instruments and their respective user manuals can be retrieved through the main search engine of the database by using either the full name or parts of the name (e.g., ABOUT™–Perceived Risk, Perceived, or Risk) or within the “Our Catalog” section of ePROVIDE™ (https://eprovide.mapi-trust.org/catalog) by filtering through the Behavior and Behavior Mechanisms category.

To be able to use any of the ABOUT™ Toolbox instruments, all users (whether they are commercial or non-commercial users) will have to accept the conditions of a license/user agreement and complete the appropriate form. This license, issued by Mapi Research Trust, which is the official distributor of the instruments, specifies the terms under which each instrument should be used: Special terms, i.e., specific to each instrument, and General terms: https://eprovide.mapi-trust.org/user-agreement-general-terms.

Tutorials are available on how to submit requests (commercial users) or download copies directly (non-commercial users) (see https://eprovide.mapi-trust.org/faq). All requests are handled by the PROVIDE team of Mapi Research Trust.

Inventory of the ABOUT™ Toolbox

The ABOUT™ Toolbox currently comprises five measurement instruments, which are at various stages of development and validation. Table 1 presents a summary of the relevant domains, concepts of interest, and context of use to be covered by the instruments. The initiative is to be expanded by additional domains as they are identified. The initial inclusion of these current instruments were informed extensively by existing research and domains that have been prioritized based on public health impact and issues of key importance to tobacco regulatory research.

Fully developed and validated instruments

1. ABOUT™−Perceived Risk

Assessment of risk perception is an important domain of tobacco-related behaviors and influences any tobacco harm reduction strategies aimed at getting individuals to switch to less harmful alternatives to cigarette smoking^1,11.

The Perceived Risk Instrument was developed as a multiscale instrument intended to assess the perceived risks associated with use across a range of different of TNPs, relative to other products, cessation aids, and quitting all tobacco products. The health risk and addiction risk scales have been calibrated in three countries (U.S., Italy, and Japan) for two perception types: risk to the individual respondent (personal risk) and risk to users of the product in general (general risk)^25,37. The validation of further scales for perceived social and practical risk is currently under development.

2. ABOUT™−Use History

One of the key aims underlying tobacco control and harm reduction is to reduce the burden of smoking-related diseases through the implementation of monitoring strategies for tobacco consumption and characterization of key patterns and trends in tobacco use³. The Smoking Questionnaire, included in the ABOUT™ Toolbox, was developed to provide a core set of questions that cover the major dimensions of TNP use and is consistent with criteria used for defining smoking history and status⁵¹. It assesses frequency and intensity of current and past TNP use behavior, initiation, and cessation and demonstrates good test-retest reliability^52,53.

Instruments under development

1. ABOUT™−Product Experience

Product experience encompasses a range of self-reported expressions of an individual’s experience using a TNP and is a key predictive measure of short-term preference and long-term TNP use⁵⁴. The modified Cigarette Evaluation Questionnaire (mCEQ) has been endorsed by regulatory and public health bodies to use in the context of MRTP assessment^9,55. The ABOUT™ Toolbox includes a measure consisting of two multi-item scales and three single-item scales arising from an adaptation and rewording of the mCEQ⁵⁶ and the Product Evaluation Scale⁵⁷. The scales focus on satisfaction, psychological reward, craving reduction, aversion, and enjoyment of respiratory tract sensation. Psychometric testing and validation have been carried out for the use of the measure to assess cigarettes and heat-not-burn products⁴⁵, and further validation is ongoing for a variety of TNPs (e.g., e-cigarettes, cigars/cigarillos, smokeless products) and different recall periods.

2. ABOUT™−Dependence

Nicotine dependence has been shown to be a primary driver of smoking and TNP use behaviors⁵⁸. However, dependence on nicotine has generally focused on cigarette smoking. Evidence suggests that measuring dependence with a common set of symptoms across different TNP use groups is feasible and may better reflect the dynamics of dependence across a range of products^59–61. Work is ongoing to develop such a fit-for-purpose dependence instrument for inclusion in the ABOUT™ Toolbox. The proposed instrument is rooted in a conceptual framework of dependence that identifies lack of control (e.g., urgency to use upon waking up, difficulty to cease using, self-awareness of dependence) as the core construct. Recommended practices in PRO development have been used to generate a draft instrument from a range of qualitative research steps, including literature review, concept elicitation, and cognitive debriefing interviews with different groups of adult tobacco users²⁷. The draft instrument is currently undergoing quantitative field testing to assess psychometric properties and produce a final validated measure.

3. ABOUT™−Health and Functioning

Health and functioning is a relevant dimension for the evaluation of TNP impact on public health and requires further investigation^8,9,58. There are no established measures specific to tobacco-related health outcomes to date. In addition, existing generic health and functioning instruments do not capture the small, but potentially important, concepts that may change when a smoker switches to an MRTP. Currently, efforts are ongoing to develop a new outcome measure for inclusion in the ABOUT™ Toolbox that would accurately reflect the health and functioning status of individuals who use TNPs, with a particular focus on healthy adult smokers who switch to MRTPs. The measure’s development will be underpinned by conceptual frameworks including, but not limited to, the WHO’s International Classification of Functioning, Disability, and Health (ICF)⁶² and the Revised Wilson and Cleary Model for Health-Related Quality of Life⁶³. The ICF is particularly well suited to serve as a guiding framework, as it conceptualizes a person’s level of functioning as a dynamic interaction between body structure and functions, health conditions, social participation, personal factors, and environmental factors. The Wilson and Cleary Model is included to address subjective dimensions of health and functioning, such as well-being and health-related quality of life. Literature review, an expert panel, and qualitative research with a wide range of consumers are planned for the initial phase of the project.