QuickCharmStats rationale and technical information

CharmStats is the name of a line of harmonization software products in development by Martin Friedrichs at GESIS—Leibniz Institute for the Social Sciences. QCS was the first product released and it is available for download on the CharmStats website. It is desktop software with a specific audience in mind: researchers working on their own computers, who use SPSS or Stata to conduct their statistical analyses, and who have fewer than one hundred variables to harmonize for any one research project. The primary goals are to 1) reduce the time it takes to assemble the metadata needed to accurately document the harmonization work; 2) improve data management by housing information used to make coding decisions in the same location as metadata on the variables, questions and studies and the SPSS and Stata recoding syntaxes, and 3) provide researchers with the template to generate a submission form to have their harmonization peer reviewed and published so they may cite it in the bibliography of their research articles.

Rather than impose a theoretical framework on all researchers a CharmStats project starts with the conceptual level, allowing the researchers to define their concepts guided by the most appropriate theory, or to document the creation of competing measures of a concept guided by alternative theories. For this reason QCS guides users through a logical workflow to import and work with all the metadata necessary to fully document variable harmonizations. Once the necessary metadata has been imported and organized users may click a button to generate harmonization syntax in both SPSS and Stata, creating reports that can be saved in a folder as either.html or.pdf, and a graph generator that displays (inter alia) source and target response mapping for visual inspection. Fields are provided to attach extended notes on the coding decision process, e.g. how a researcher dealt with missing cases in a particular nation or why she coded ‘Don’t know’ responses as valid, rather than missing, data.

This article is also an attempt to set out what information is both necessary and sufficient to accurately replicate another’s harmonization work. Drawing on personal experience with ex-post harmonization on an international comparative survey we determined the following metadata were necessary to report in order for someone else to have sufficient information to recreate the harmonized variable: 1) the name of the project and its author(s) so the work is searchable and findable, 2) the syntax for the harmonization coding in both SPSS and Stata for precise replication, 3) information on the purpose of the harmonization, 4) the name of the concept to be operationalized and harmonized, as well as any literature or previous work that informs the recoding, 5) metadata on the target and source variables including name, label, definition, the response options values and their labels as well as the level of measurement and whether it is individual or aggregate level data, 6) metadata on the target and source question wording, including notes on the questions and relevant instructions to interviewers on how the question was to be administered, and 7) metadata on the name of the study and its permanent identifier or online source for the data (Table 1 and Fig 1). This information is provided so future researchers can consult the original study’s documentation for details that may be important (sampling frame, collection dates, mode of data collection) but are not necessary to replicate the harmonization recoding.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Marital Status harmonization mapping.

https://doi.org/10.1371/journal.pone.0147795.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Proposed format for Harmonization Project Documentation using CharmStats.

https://doi.org/10.1371/journal.pone.0147795.t001

QuickCharmStats is a Java® based desktop application. All CharmStats software products work by storing persistent information in a relational database. QCS works on a local database instance (localhost). The MySQL DBMS was chosen as the system to manage the database and its content. MySQL (Structured Query Language) is an open-source relational database management system (RDBMS) owned by Oracle Corporation. The software supports versions of Java 1.6 or higher and users can download MySQL software for free if they do not have it. The installation package for QCS is available from the GESIS website [9]. Users must provide a name and email address, then a link for the download is sent to their email accounts. The QCS software, its code, the user manual [10] and practice datasets come as part of the download. After copying the zip-file to the place of installation and unzipping it a user must connect QCS to the MySQL database. This is done by running a setup batch file in the windows command processor (cmd.exe) once. To start the application, double-click the CStatsApp jar-file symbol in the QuickCharmStats directory. The program is ready for use. QCS was developed for PC; however, Mac-tested versions are planned for the future.

QCS was designed for use by social researchers doing statistical analyses. Given their popularity, syntax for SPSS and Stata were selected as the first languages to be generated in the 1.0 and 1.1 versions. Both software packages were examined to evaluate their performance as a source of metadata. To date we have not found a way to retrieve variable metadata from Stata but we have using SPSS; therefore QCS 1.1 imports data using SPSS datasets (.sav files). It is important to note that a CharmStats user need not own the rights to use SPSS in order to use.sav datasets to import metadata into CharmStats. The.sav is read by CharmStats without using the SPSS software. Stata users can use Stat/Transfer to export their datasets as.sav files for importation into QuickCharmStats. Researchers without access to Stat/Transfer can email their.dta file to charmstats@gesis.org at least 2 weeks in advance of when they require the.sav file, free of charge. The number of syntax languages will be expanded in later versions (e.g. SAS). The Data Documentation Initiative (DDI) is an international effort to create a standard to describe statistical and social science data, and work is underway on the QCS 1.2 version that will allow metadata to be imported using forms of DDI (3.0 and 3.1). However, since more social researchers use SPSS or Stata data files to conduct their analyses we decided to realize the.sav import feature first, to make the most of users existing skill sets and to realize the DDI import functions in future versions of the software.

Harmonization documentation types

In our view, the first step in establishing common harmonization documentation practices within and across scientific disciplines is to determine the necessary and sufficient information required to accurately replicate prior variable harmonization work. Unlike other publications a harmonization project is not a single piece of work. Researchers must combine information from one target variable (the harmonized variable structure) and at least one source variable (the original, unharmonized variable). This could mean a researcher needs to report metadata for one target variable in the target dataset, and at least one but potentially dozens of source variables from different source datasets. Therefore, CharmStats products collect and organize information on many levels of metadata to streamline the process and ensure replicability of the data harmonization. We review how the software organizes metadata and other information below.

Types of information hand-entered by the user

• Project information: Harmonizations in CharmStats are organized into projects. A project contains various levels of metadata information and the response option mapping associated with harmonizing N source variables to a single target variable.
• Concept: At its most abstract, variable harmonization is an attempt to harmonize concepts. A project contains the work associated with harmonizing one concept using one target and at least one source variable. QSC 1.1 facilitates single dimension concept harmonization, such as socio-demographic variables. However, the CharmStats data model allows for the harmonization of composite variables such as scales. This capability will be realized in later versions of the software.

Metadata collected or hand-entered during the data importation process

• Target and source variables: Target and source variable metadata contain similar information. The variable name, label, values and value labels, and measurement level are all imported from the.sav file.
• Question information Metadata on the question can be included as well. To facilitate replication and reuse researchers should include the question wording, interviewer instructions, skip pattern structures, or other information needed to replicate the work.
• Study information: QuickCharmStats allows users to attach the same study-level metadata to a series of variables while they are importing them. The name and version of the study, its persistent identifier (e.g. the DOI), the primary investigator(s), data collection methods, field period and collection mode are typed in once and then linked to each variable from that dataset.

The CSES, the BES and the data gap

The Comparative Study of Electoral Systems (CSES) is a research project that collects and harmonizes post-election studies from around the world. Since 1996, the CSES has harmonized and published more than 180 post-election studies from 55 different countries covering various political topics [11]. It requires that samples must be random and cover the voting population in the participating countries. Data may be collected using: face-to-face interviews, telephone interviews or self-administrated mail-back questionnaires. This is done to ensure comparability as well as similar data quality across election studies. Uniformity in these standards can be traced back to the first CSES data collection in 1996; a time when online-sampling was uncommon, internet penetration was limited as well as highly skewed, and web-based survey were a novel method of data collection. These data collection requirements remain in effect until the third module which is why the CSES databases exclude 2005 and 2010 British Election Study data.

A British Election Study (BES) has been conducted at every general election since 1964. Its purpose is to study the long-term trends in British voting behavior; explain election outcomes, party choice and turnout; and to examine the consequences of elections for the operation of democracy more generally [12]. In 2005 the British Election Study team ran an experiment by collecting both face-to-face and the Internet data for the same questionnaire [13]. The CSES questionnaire was included as part of the BES internet survey. In their article Sanders et al. present the results of their extensive experiments on the main BES questionnaire. Their analyses showed ‘there are few statistically significant differences between coefficients generated using the in-person and Internet data, and the relative explanatory power of rival models is virtually identical for the two types of data’ [13]. However, since the data were not collected in the approved modes the 2005 BES data are not included in the CSES harmonized dataset. Instead the 2005 British data were collected as part of a self-completion post-electoral supplement to the annual British Social Attitudes (BSA) survey. In 2010 the British Election Study team again collected CSES data using the post-election web panel [14]. As a consequence the third module of the CSES does not include the 2010 BES data in its publically released dataset. Rather than requiring each comparative researcher to combine the BES and CSES datasets individually, wasting time by repeating the same harmonization work and possibly making different coding decisions, we used CharmStats to create a publishable harmonization project that includes syntaxes for SPSS and Stata and can be cited [15].

How the CSES harmonizes data

The CSES works to implement a common set of questions for participating countries’ post-election studies. Producing high-quality and transparent data harmonizations for statistical analysis and replication requires familiarity with the source and target dataset collection methods and sampling frames. Simply pooling the variables as measured is often not possible due to individual national standards, e.g. education variables reflect national structures that are not equivalent. Consequently, some CSES variables are ‘output harmonized’. Output harmonization is a necessary and routine process; however, it impacts on the cross-national comparability of harmonized measurements as well as the data’s quality and replicability [16]. The process of output harmonization occurs in three major steps: assessing the data at the collection level, assessing the variables at the measurement level, and documenting the process of concept harmonization by recording the exact transformation of the source variable into the harmonized structure (target variable). We review each of these steps below.

The first task is to ensure that all the variables to be used are available in both the target and source datasets. Then the researcher must familiarize himself with the structures of the source and target datasets to be harmonized. This means having information on the underlying populations, the study’s sampling procedures, whether the entire population of interest is covered by the source data and whether the source population of interest can be matched to the target population of interest. Researchers responsible for variable harmonizations must ensure the comparability and compatibility of the data itself, in addition to checking the harmonized variables for any systematical drop-outs that may occur. The third step is harmonizing the measures. The researcher must know the question wording of the source and target variables and response categories as provided to respondents. Guided by the questions and response options a researcher will most likely start with linking identically-worded response categories in both questions. Next would be to collapse individuated response categories in the source variable–in our example ‘divorced’ and ‘separated’- with its multi-category response option counterpart in the target variable ‘divorced or separated’. Afterwards the researcher will focus on response categories that are available in one variable but not in the other, or have slightly different wording in the response options. For example the 2010 BES marital status response category “living with a partner” must be collapsed into the most conceptually equivalent CSES category “married or living together as married”.

In the final step the researcher must document this harmonization process to ensure transparency and replicability for future use; the researcher must write an overview of the source and target studies, their underlying populations, sampling procedures etc., the question wording and response options for each source and (when relevant) target variable, the conceptual mechanism captured by the target variable, as well as the decision criteria for recoding source variable response options. To ensure a high level of transparency, the researcher must detail all the coding decisions, providing the full information necessary to replicate the research.

Finally to illustrate the practicalities in documenting harmonization, in the example above it is important to note that we harmonize data for a project in which missing values (such as refused, skipped, don’t know or missing) are all coded to ‘system missing’ in the appropriate statistical software package. Likewise, some values mentioned in the CSES or the BES codebooks are not included in the example. For instance, the CSES value 5 (labelled as “see election notes”) allows researchers to code marital statuses not covered by the fixed categories 1 to 4, such as “long-term relationship” in the 2008 Austrian subsample. Since the BES does not include an outstanding category that does not fit into the standard CSES categories, the CSES code 5 remains unused and is thus not shown in the mapping (Fig 1).

Publishing and citing with QuickCharmStats

Using the QCS workflow and features, we propose the following formats as standards for published harmonizations. We call these digital harmonization projects. The QCS digital harmonization project contains metadata on multiple levels. The Project area describes the project itself and its contents. In the Concept area users insert the concept definition(s) and links to relevant literature or related studies and publications. Next, the target and source variable metadata are provided in their own sections. The syntaxes for the harmonization, in SPSS and Stata formats, are included in the report. The last section is for graphs generated by QCS, displaying the harmonization mapping in an easy-to-read visual format. We designed the QCS workflow and report so each section has a level with its own type of metadata. Our example digital harmonization project contains the following information:

Project: Project level data includes the harmonization project name and information on the intent of the project. The first challenge was determining how projects should be named. The information used to construct a project name is important since, in future, finding the right project will depend, in part, on the information included in the name. Naming projects is a challenge because one researcher could be searching for marital status measures, another could be looking for British data, and yet another could be seeking CSES data. In our view, the best way to ensure that future users can find the information is to standardize the naming of harmonization projects. The name of the concept and the target and source datasets used in a project name will be converted into keywords for search engines when deposited for publication.

As a best management practice we recommend individual CharmStats harmonization projects build their names ordered by the following information: (1) the concept to be harmonized, (2) the name of the target variable’s study (or its acronym), and (3) the name of the source variable(s) study/ies or acronym(s). In this example the project name is ‘Marital Status Comparative Study of Electoral Systems Module 3 to British Election Study 2010’. If we were harmonizing two source variables to a single target variable, the project name would be ‘Marital Status Comparative Study of Electoral Systems Module 3 to British Election Study 2010 and British Election Study 2005’. More than three source variables to a single target variable would be named ‘Marital Status Comparative Study of Electoral Systems Module 3 to British Election Study 2010, British Election Study 2005, et al.’ CharmStats’ project notes provide space for short descriptions about the project. Other project level information includes the submission and publication dates for the project and the names of the author(s). Project end use notes provide specific or more technical information on how the variable was used.

Harmonization Syntax: QCS produces syntaxes ready for use either in SPSS or Stata. Including the syntax in the documentation should reduce the time and effort needed to replicate others work or to compare harmonization structures across published results. Future versions of QCS and other CharmStats products will provide the recoding in additional popular proprietary statistical package syntaxes.

Concept: This area records the name of the concept to be harmonized and the definition of the concept (or compatible/competing definitions) that guided variable operationalization. Space is provided to write comments on the concept as used in the research as well as the names of and hyperlinks to relevant literature, publications or studies that informed the recoding. Providing extra space for the concept definition and researcher comments should increase clarity and transparency in replication.

Whether future researchers replicate the concept as operationalized in that particular project or whether they are inspired to create new measures, the first step is to provide detailed documentation. This can be illustrated by examining the conflation of multiple concepts within the ‘marital status’ response options. Although the target and source variables both are named ‘marital status, one very narrow but valid definition of marital status could be ‘the condition of being married or unmarried’ [17] This would limit response categories to ‘single’ or ‘married’. In practice, social surveys collect descriptive information from single people (divorced, widowed or have married) and from married people (living apart but still married). To continue with this necessary pedantry, marital status responses options often include ‘living with a partner’ or ‘living together as married’ options that best represent the concept of cohabitation: ‘a living arrangement in which an unmarried couple lives together in a long-term relationship that resembles a marriage’ [18]. While this is common in practice, it is still conflating two different concepts into one measure and we note this in our concept comments. Further, we note the available response categories cannot capture same-sex civil partnerships or changes in marriage laws as they relate to same-sex couples. Space for precise definitions and the rationale for coding decisions are important for theoretical reasons and secondary reuse. For instance, sociological investigations into marriage and cohabitation might see greater differences between the options “being married”, “living with a partner” and “living together as married” than a political scientist. For researchers using European data, in the Netherlands or France the status of ‘legal partnership’ implies a different legal status than simply ‘living together with a partner’ or than being ‘married’ [19, 20], while cross-national political scientists might aim for an overall comparability of the marital status control variable between different election studies.

Target and source variables: Target and source variable sections in the report contain similar metadata and information. These include the target and source variable names and labels, the sampling and measurement levels, the question wording and response wording options and values. The source variable also lists the name of the ‘instance’ in which it is used within the QCS software. When more than one source variable is harmonized to a target variable, each gets a unique ‘instance’ name to differentiate it from other source variables within the same project. Space is provided so researchers can include the name of study dataset and links to the study’s (or studies’) persistent identifier(s). In our example we provide the DOI for the CSES study, however at the time of writing, the 2010 British Election Study had not yet deposited their data. Instead, the BES data is available on the study’s website. We list the BES data url and date last accessed in lieu of a DOI. This is an attempt to capture the versioning of studies not submitted for preservation with a data archive. Finally, both target and source metadata provide space where researchers can include relevant comments. CharmStats users may import existing variables or they can create their own target variables and document that process within the project; the comments section is where researchers can elaborate on other details or construction of the measure. The researcher needs to document, and thus explain, why she collapsed the BES “living with a partner” category into the CSES category “married or living together as married” as opposed to keeping the BES category separate (e.g. using the CSES category 5 that can be modified to be any response option in the CSES). Depending on the research question, unmarried couples might make a difference in the outcome of the analysis and thus impact the research findings. Providing a rational for the collapse of certain categories can be necessary information to achieve full transparency.

Future versions of QCS and CharmStats products will allow users to import certain forms of DDI so that, in addition to the study metadata, users may import question and survey metadata directly into their projects. In addition to including the question wording, we recommend researchers include additional question-level metadata when it is relevant to how the variable should be used in replication, for instance, noting routed questions, interviewer instructions or explaining the skip pattern structure.

Graph: QCS includes a graph generator. Fig 1 shows the marital status harmonization mapping. Users may inspect these graphs, insert them into presentations and include them in their published harmonization. The graphs allow users to spot coding errors before running the syntax, to clarify the coding of collapsed response options; and to make it easier to compare between different harmonizations structures across (and/or within) datasets. Researchers can evaluate data loss between alternative harmonizations by visually comparing the number and types of collapsed categories.

Peer review and publishing

In her article, Replication as Regulation, Fowler wrote that replication provides the basis for a cumulative body of work using similar methods and data but that ‘the amount of information sharing is likely to be suboptimal without some sort of incentive or requirement to bring it about’ [21]. CharmStats software products are designed to provide a digital publication pathway for harmonization work.

Until the development of CharmStats products, collecting the necessary metadata to fully and transparently document variable harmonization work was simply too time consuming and, although it would be virtuous work, there was no career pay-off for the hours spent assembling the information needed for precise replication. At present the intellectual contributions of data harmonizers go unrecognized and their work is not easily citable. Even if a researcher managed to document their variable harmonizations there is no place for them to publish it. QuickCharmStats organizes a researcher’s work in a format pre-prepared for the peer-review process. If a harmonization documentation project is accepted for publication it will receive a permanent identifier that can be listed as a reference when citing researchers’ original harmonization work in a publication. Clicking the permanent identifier link will take the reader directly to the published harmonization documentation–including the recoding syntax–making it freely available, easily findable as well as replicable and citable by other researchers.

Researchers who wish to submit a digital harmonization project for publication can download the peer-review submission report template from the GESIS CharmStats website. The completed report and the graph should be emailed to: charmstats@gesis.org. If the submission does not meet the standards of peer review the researcher will be given precise instructions on what to include in order for it to be accepted. Once accepted the submitted digital harmonization will be assigned a DOI and published on a GESIS website or deposited into an online harmonization library, free of cost.

Finally, we propose a referencing guideline for CharmStats harmonization citations. As discussed above in the project naming rationale, standardizing the project titles will facilitate finding and accessing harmonizations and will thus promote secondary use. We adapted current citation guidelines for datasets to fit digital harmonization projects, including the project’s hyperlink, date of publication, version or DOI and the date the author last accessed the documentation. We provide an example based upon the marital status harmonization.

Winters K, Netscher S Marital status Comparative Study of Electoral Systems Module 3 to British Election Study 2010. [harmonization] Cologne, Germany: GESIS [distributor] 2015. (Accessed 4 January 2016). http://dx.doi.org/10.7802/1160.