To begin, test development specialists identify the content required to produce new items. Gierl et al (2012) introduced the concept of a cognitive model for AIG. Figure 1 contains a cognitive model for AIG required to treat infection during pregnancy. A cognitive model is a representation that highlights and helps organize the knowledge, skills, and content required to make a medical diagnosis. The model also organizes this cognitive and content-specific information into a coherent whole, thereby presenting a succinct yet structured organization of the content relationships and sources of information used in formulating medical diagnoses.

This cognitive model was created by two medical content specialists who were experienced test developers and practicing physicians. It serves as a representation of how physicians think about and solve problems related to infection during pregnancy. The cognitive structure in Figure 1 is presented in three panels. The top panel identifies the problem and its associated scenarios. In this example, five different drug types could be prescribed to treat specific infections during pregnancy [i.e., penicillin (P), cephalosporin (C), macrolides (M), sulfa (S), furantoin (F)]. The brand names for each drug type are also identified (e.g., penicillin G, amoxicillin, and ampicillin). The middle panel specifies the relevant sources of information. Two sources of information are specified in this example: type of infection and patient characteristics. The bottom panel highlights the features. Five features (i.e., urinary tract infection, pneumonia, cellulitis, gestation period, allergy) are identified across the two sources of information. Each feature contains elements and constraints. Elements identify the content specific to each feature that can be manipulated for item generation. The pneumonia feature in the cognitive model, for example, contains the element “present” (i.e., the pregnant patient with infection has pneumonia). The second component for a feature is the constraint. Each element is constrained by the scenarios specific to this problem. Cephalosporin (C) and macrolides (M) are the drugs “very likely” to be used to treat infection during pregnancy when the type of infection is pneumonia. This constraint also implies that the other drugs are not likely to be used.

Once the content is identified in the cognitive model, it must be placed into the template required for item generation. Item models provide this template-based format. Item models contain the components in an assessment task that require content. These components include the stem, the options, and the auxiliary information. The stem contains context, content, and/or the question the examinee is required to answer. The options include a set of alternative answers, with one correct option and one or more incorrect options. Auxiliary information includes any additional content, in either the stem or option, required to generate an item, including text, images, tables, graphs, and diagrams. A sample parent item and the associate item model for the infection during pregnancy example is presented in Figure 2. This example contains a stem and options but no auxiliary information.

Two types of item models can be created for AIG. The first type is a one-layer item model. The goal of item generation using the one-layer approach is to produce new assessment tasks by manipulating elements at one level. An example of a one-layer item model for infection during pregnancy is presented in Figure 3. The stem contains one integer [AGE] and three strings [TRIMESTER; ALLERGY; TYPE OF INFECTION]. The second type is an n-layer item model. The goal of AIG using the n-layer item model is to produce items by manipulating elements at two or more levels. The n-layer structure can therefore be described as a model with multiple layers of elements, where each element can be varied simultaneously at different levels to produce different items. An example of an n-layer item model for infection during pregnancy is presented in Figure 4. In this example, two layers are used. The first layer is sentence structure. The first sentence is “A [AGE]-year-old pregnant female at [TRIMESTER]; [ALLERGY]; presents with clinical and radiological signs and symptoms consistent with [TYPE OF INFECTION].” The second sentence is “Suppose a pregnant woman [ALLERGY] was admitted with signs consistent with [TYPE OF INFECTION]. She was in her [TRIMESTER] .” The second layer includes the same elements specified in the one-layer model, except that trimester is expressed as both an integer and string and the options are presented as both drug types and brand names to increase the generative capacity of the model. The item model also contains the options which specify the correct alternative and one or more incorrect alternatives or distracters. For the infection-during-pregnancy example, the possible treatment options are Furantoin, Sulfa, Macrolide, Cephalosporin, and Penicillin. The correct option serves as the key and the remaining options serve as the distractors for every generated item.

The n-layer model is a flexible structure for item generation because it allows for many different combinations of embedded elements. Gierl et al. (in press) illustrated how language could be used as a layer within an item model to permit multilingual AIG. They introduced the concept of a linked element as a way to facilitate the computer programming task in Step 3 of the AIG process. Layered elements permit content to be embedded within content in an item model, thereby serving a “vertical” function for item content (i.e., content within content). Layered elements are presented in the n-layer example in Figure 4. Linked elements expand the capabilities of item modeling by permitting content to be transformed within an item model. For multilingual AIG, the transformation is from one language to another. Linked elements, therefore, have a “horizontal” function for item content (i.e., content in source language is transformed to content in target language). Gierl et al. (in press) used linked elements in English, French, and Chinese to create an n-layer model for generating medical test items requiring examinees to treat complications with hernias.

Once the content has been specified and the item models created, this information must be assembled to generate new items. This assembly task is conducted with a computer algorithm because it is often a complex combinatorial problem. Gierl, Zhou, and Alves (2008) created a computer program called IGOR to solve this test assembly problem. IGOR, which stands for Item GeneratOR, is a Java-based program designed to assemble the content specified in the item model, subject to the elements and constraints identified in the cognitive model. While we use IGOR to demonstrate Step 3 in the AIG process, it should also be noted that any linear programming method could be used to solve this type of combinatoric problem (van der Linden, 2005).

To begin, the linked elements are translated directly from the source to the target language in two forms. The first form is at the word level, meaning that each word element in the item model is translated from English to French. The second form is the key phrase level, meaning that each key phrase elements in the item model is translated from English to French. The analysis component is therefore akin to the direct approach, which is the first step in Vauquois’s triangle. The bilingual content specialists directly translated the words and key phrases from the source to the target language for all elements in the n-layer infection-during-pregnancy item model. A summary of the translations is presented in Figure 4.

Next, the linked elements are translated from the source to the target language at the sentence level. For this component, the adequacy of the word and key phrase translation is considered by the bilingual content specialist, as in the analysis component, but now the syntactic structure of the words and key phrases as they form sentences is also considered. The process of reordering occurs in this component to capture important structural differences that differentiate the source and target languages. The reordering process is often called alignment (Jurafsky & Martin, 2009). The transfer component corresponds to the syntactic level in the transfer approach, which is the second step in Vauquois’s triangle. Take, for example, the sentence “Which one of the following antibiotics is the most appropriate?”. The French translation is not a literal word-by-word transformation. Instead, the syntactic structure from English must be adapted to French, as shown in Figure 6, so that the two phrases are aligned. The resulting French item model is given as “Lequel des antibiotiques suivants serait le plus indiqué ?”. In short, rules are used in the transfer component to adjust the structure of the source language so it conforms to the structure of the target language.

Finally, the meaningfulness of the translated items, given the outcomes from the analysis and transfer components, is addressed. The linked elements are translated and aligned from the source to the target language at the multiple sentence level to produce models that can be used to generate translated test items. For this component, the word and key phrase translation is synthesized with the newly aligned sentences so items can be generated from the model. Once the items are generated, a judgment is made about the adequacy of the semantic structure for the translated items. In our multilingual AIG method, this judgment is made by a human translator who evaluates a sample of the generated items to determine if they are meaningful. The synthesis component corresponds to the semantic level in the transfer approach, which is the third step in Vauquois’s triangle. For example, our item model initially generates the following stem: “Supposons qu’une femme enceinte a été hospitalisée parce qu’elle présentait des signes de cellulite. Elle était au 1^er trimestre de sa grossesse. Lequel des antibiotiques suivants serait le plus indiqué ?”. But upon review by the bilingual medical content specialist, the item was edited and revised, as follows, to produce a more meaningful stem: “Supposons qu’une femme enceinte est hospitalisée parce qu’elle présente des signes de cellulite. Elle est dans le 1^er trimestre de sa grossesse. Lequel des antibiotiques suivants serait le plus indiqué ?”.

If the analysis and transfer components are implemented successfully, then the elements across the source and target item models are correctly synthesized, thereby producing meaningful test items. Conversely, if either the analysis or the transfer component is implemented unsuccessfully, then the elements across the source and target item models are incorrectly synthesized, producing problematic test items. The root of the problem can be addressed by scrutinizing the analysis and transfer results. Once the potential problems are identified and corrected, the results can again be synthesized and the meaningfulness of the generated items can be determined. Iterating through the analysis and transfer components should occur until the item models can consistently generate meaningful (i.e., semantically appropriate) test items. A summary of our multilingual AIG approach is presented in Figure 7.

We described our multilingual AIG method and then illustrated the method using two languages, English and French. But this method can also be used with three or more languages using the same logic to permit simultaneous multilingual item generation across all of the languages specified in the item modeling step. Once the analysis and transfer components are completed, a multilingual AIG linking map is produced. The map contains the necessary text and translation required to link the words, key phrases, single sentences, and multiple sentences across the language groups. Then, we implement computer assembly algorithms found in programs like IGOR as part of Step 3 in the AIG process so that multilingual items can be generated. Using our multilingual AIG methodology, a total of 720 infection-during-pregnancy items were generated—360 English and 360 French items using the model in Figure 4. A random-sample of four items in English and French is presented in Tables 1 and 2, respectively.

In their chapter on “Technology and Testing” in the 4^th Edition of the handbook Educational Measurement, Drasgow, Luecht, and Bennett (2006, p. 471) proclaim:

This chapter describes our vision a 21^st-century testing program that capitalizes on modern technology and takes advantage of recent innovations in testing. Using an analogy from engineering, we envision a modern testing program as an integrated system of systems. Thus, there is an item generation system, an item pretesting system, and examinee registration system, and so forth. This chapter discusses each system and illustrates how technology can enhance and facilitate the core processes of each system.

This bold view of educational measurement asserts that integrated technology-enhanced systems will direct all testing processes in the future. This perspective captures a growing consensus among researchers and practitioners that a well-defined science of educational measurement will emerge to guide our development, administration, scoring, and reporting practices. It also provides a more complete picture of how the component parts function together to govern this new testing system. We presented a concrete example of how an “item generation system” could operate by describing a three-step AIG process and then illustrating how this process can be used to generate hundreds of medical test items. We develop this analogy further for multilingual test development by describing how an “item generation system” could also contain an explicit “item translation subsystem” using the components and procedures presented in our multilingual AIG method. This system would not only permit the development and generation of items in multiple languages but also allow developers to create large multilingual item banks. The unit of analysis, however, would no longer be the test item. Rather, the unit of analysis for a multilingual AIG item bank would become the item model. Hence, one more component of this new technology-enhanced testing system would be an item model bank.

Current practices in test development are focused on the item as the unit of analysis, meaning that each item is individually written, reviewed, revised, edited, and banked. If a developer wants to have 360 items in a test bank, then 360 items must be written, reviewed, revised, edited, and banked. An item bank serves as an electronic repository for maintaining and managing information on these 360 items. Because the item is the unit of analysis, maintenance of the bank also focuses on item-level information. For instance, the format of the item must be coded. Item formats and item types can include multiple choice, numeric response, written response, linked items, and passage-based items. The content for each item must be coded. Content fields include learning outcomes, blueprint categories, item identification number, item format, date, source of item, and copyright information. The developer attributes must be coded. These attributes include year the item was written, item writer name, item writer demographics, editor information, development status, and review status. The statistical characteristics for the item must also be coded. Statistical characteristics can include word count, readability, classical item statistics, item response theory parameters, distracter functioning analyses, item history, differential item functioning flags, and history of item use. In short, large amounts of detailed information must be collected to maintain and manage a bank because the item is the unit of analysis. The requirements and work needed to address the bank specifications is merely expanded when items are developed in multiple languages.

To address the problem of collecting and maintaining large amounts of item-level information, we propose that item model banks be created. AIG is a scalable process because it treats the model as the unit of analysis where one model yields many items. Hence, the unit of analysis moves from the item to the model, meaning that each model is individually written, reviewed, revised, edited, and banked (but not necessarily the items generated from the model). If, for instance, a developer wants 720 items, then only one bilingual item model is required, as we illustrated in our study. An item model bank serves as an electronic repository for maintaining and managing information on each item model. The item model bank will contain information on every model, but not necessarily on every item. As a result, the amount of information collected to maintain and manage the model bank may still be considerable but, compared to the item bank, it would be substantially reduced. AIG, therefore, leads to more cost-effective test development because the model is continually re-used to yield many test items, compared with developing each item for a test from scratch. Costly errors commonly found in test item translation (e.g., omissions or additions of words, phrases, or expressions as well as spelling, punctuation, capitalization, item structure, typeface, and formatting problems) are minimized because a generalized method for multilingual AIG exists. The monitoring and maintenance of this information is also streamlined by focusing on the attributes of the model. Thus, an item model bank would be another important requirement for an “item generation system” of the future.