Keywords

1 Introduction

Enterprise modelling consists of different perspectives such as goal modelling, business process modelling, value modelling, etc. Amongst those perspectives, the ‘what’ or data perspective, addressed through domain modelling, is considered one of the crucial aspects of Enterprise Modelling [1]. However, any educator that starts teaching domain modelling faces several challenges. First, domain modelling often means formalizing an ill-structured domain or problem description formulated in natural language. This applies to any field where domain modelling or conceptual data modelling is used: engineering design, software development or enterprise engineering. Ill-structured problems are domain- and context-dependent, so in addition to the knowledge of modelling techniques per se, novice modellers should grasp the context and the specifics of the domain in which they have to work [2].

Second, there is no existing generally accepted framework for modelling pedagogy – educators have to come up with the entire course design, tasks and learning paths on their own, based on their professional experience and views on learning process.

Third, not only the approaches to teaching, but even the terminology significantly varies, bringing the entire field of domain modelling close to a “Babylonian” state, where everyone names the same notion in a different way. Though some modelling methods are similar or even identical across communities (object-oriented modelling community, database modelling community, and others), the fact that they may implement completely different terminology and notation hampers the exchange of knowledge. As an example, in [3], the term “domain object” or “object” for short, is used to address a wide variety of concepts, such as entities, associations, agents and events, all of which are considered subtypes of “Object”. Consequently, the terms “object model” (as in [3, 4]) and “object diagram” have completely different meanings: according to the current version of the UML standard, “object diagram” refers to the concrete instances that exist in the system in a given moment, while in [3] and in many other resources “object model” refers to the equivalent of a UML class diagram describing the model in general. To avoid confusion, in this paper we will stick to the standard terminology of UML [5].

Last, but not least, though there are many insights available on what makes a good model (a large review on that matter was made as early as in 1994 [6]), the portrait of a good modeller is still somewhat blurry. The skill set that he/she is expected to possess is not formalized, and, subsequently, the identification of learning outcomes is complicated, so as the development of assessment criteria.

In this work, we aim to make an initial step towards the systematic educational framework for enterprise modelling. As a first step, we limit our research to the data modelling aspect only, not concerning the aspects such as business processes, goals, or business object behaviour, which will be subjects of future studies. We investigate the current state of practice by identifying and classifying learning outcomes pursued in samples of educational literature [3, 4, 7, 8], massive open online courses [9,10,11] and university level courses exams from KU Leuven, Université catholique de Louvain and University of Namur [12,13,14,15,16]. Inspired by learning outcomes categorization works conducted in different fields of studies, such as biology [17], social sciences [18], computer science [19], and others, we use the revised Bloom’s taxonomy of educational objectives [20] as a classification tool.

The following research questions are to be answered in this work:

  • RQ1: What learning outcomes can be identified in current domain modelling education?

  • RQ2: What is the positioning of the identified learning outcomes in the revised Bloom’s taxonomy of educational objectives?

    • RQ2a: How are the learning outcomes distributed among various knowledge levels?

    • RQ2b: What are the most frequently appearing types of tasks?

  • RQ3: What is the range of domain modelling concepts addressed by the educators?

Apart from providing information on the state of practice, developing the methodology for the analysis of the current practice allowed for additional contributions to the field of teaching domain modelling. The analysis required a field-specific revision of the Bloom’s taxonomy (Sect. 3), an analysis of how the learning of domain modelling concepts is scaffolded, and the categorization of the learning goals provides a set of typical tasks that can be used as inspirational templates by teachers.

The remainder of this paper is structured as follows. Section 2 presents related research. Section 3 presents the methodology, amongst which a revised Bloom’s taxonomy for domain modelling (Sect. 3.1), and a scaffolding of domain modelling concepts (Sect. 3.2). Section 4 then presents the results of classifying 291 exercises and tasks from 12 different sources. A discussion follows in Sect. 5 and Sect. 6 presents the conclusions and topics for further research.

2 Related Research

2.1 Pedagogical Resources on Domain Modelling

As already mentioned in the introduction, there is no general agreement on how to teach domain modelling. Existing standards, such as MSIS 2006 [21] or IEEE SE 2014 curriculum guidelines for software engineering education [22], only give a general perspective on the large educational field, such as software engineering, and only briefly mention the aspect of domain modelling. As for domain modelling as a discipline, though there are plenty of approaches to how to model, there is no comprehensive literature or published guidelines on how to teach modelling.

Several attempts were made to determine the criteria of competence of novice modellers and propose the learning paths for their professional growth. [23] describes teaching practices aimed at building a bridge between a novice conceptual data modeller and an expert. Such practices include a high-level four-step strategic plan for training modellers comprising familiarization with data modelling constructs, adopting an expert’s strategies, gaining exposure to different application domains and reviewing the developed data models. Another attempt of improving the understanding of conceptual modelling by novice modellers was made in [24], based on observations of novice versus expert modellers going through all steps of developing a conceptual modelling project, including data collection and domain modelling. The observations show that the patterns applied by novice modellers differ drastically from those applied by the experts. Thus, the identified experts’ patterns could be used to improve procedural knowledge of novice modellers. In [25], the authors propose to assess students’ understanding of the concept of inheritance based on a five-level scale: from understanding the difference between abstract and concrete classes to understanding of complex models with more than one inheritance hierarchy.

The examples of works above demonstrate that the available sources on modelling education propose very high-level approaches that never go into the details, such as the topics learned on each stage or the type of tasks given to the students. Though the available textbooks propose various learning paths, no study was found that analyses the effectiveness of proposed scaffolding methods.

2.2 Implementations of Bloom’s Taxonomy

Bloom’s taxonomy [26] and the revised Bloom’s taxonomy [20] continue to attract the attention of educators in a wider and wider variety of fields. Since the 1960s, when the first handbook proposing the taxonomy was published, the educational community conducted tens of classification works.

In 2008, Bloom’s taxonomy was applied in the University of Washington for developing the Blooming Biology Tool – an assessment tool to assist the biology educators to align the assessments they use with the teaching activities and help develop classroom materials and exams based on a unified evaluation kit [27]. Later the same year, Science published the report on the application of Bloom’s taxonomy to the major biology-related exams [17], based on the exam questions ratings from [27]. The findings were rather surprising for the biology education community, as the analysis of MCAT (Medical College Admission Tests) showed that its tasks, perceived as heavily based on content knowledge (lower-order thinking), contain a large proportion of tasks requiring higher-order thinking, such as problem-solving ability and critical thinking. Later on, the “Blooming” tools expanded to more narrow fields of biology: consequently, the Blooming Anatomy Tool [28] and the Bloom’s Taxonomy Histology Tool [29] were successfully applied in the corresponding fields. In [30], the revised Bloom’s taxonomy was applied to classify a large collection of biology-related assessment packages into a two-dimensional taxonomy; the study showed the lack of procedural and metacognitive knowledge-related questions.

Biology is not the only field where Bloom’s taxonomy was applied: [31] presents a successfully implemented self-assessment tool based on Bloom’s taxonomy for students of programming classes; in [32], the authors developed a set of core learning objectives for accounting ethics using Bloom’s taxonomy; it was applied for assessing the software engineering curriculum and the IEEE software engineering body of knowledge [19]. However, to this moment none of the existing studies attempted to apply Bloom’s taxonomy to domain modelling or enterprise modelling education.

3 Methodology

3.1 Revised Bloom’s Taxonomy for Domain Modelling

As the revised Bloom’s taxonomy itself was developed as a general one and applicable to a wide variety of fields, the criteria for classification given in the description of the taxonomy should be narrowed and tailored to be implemented in domain modelling education. The revised Bloom’s taxonomy of educational objectives [20] is a matrix identifying 6 cognitive process levels and 4 different knowledge levels. As the result of the tailoring to domain modelling, the following criteria of categorization (with examples) were applied in this work.

Cognitive Process Levels

Remembering tasks imply recalling and reproduction of previously learned material. This includes giving definition to terms, recognition of notation and copying/duplicating existing learning material. In case of domain modelling, such tasks may include giving a definition to a given modelling concept (e.g. inheritance, aggregation, association class), or naming a given modelling notation element.

Understanding tasks refer to the previously learned material, possibly by interpreting it (explaining, translating from one form to another), or by comparison. Typical understanding level tasks would include interpreting a given model and providing its textual description or giving an example (instance) of a given class or association.

Applying tasks imply using the previously learned information in new ways and/or implementing a learned technique. Applying domain modelling knowledge may include modification of a model using an available example or the application of a given technique to create an association between two classes.

Analysis tasks imply deconstructing the material to understand its inner structure and the general principles of relationships between different elements. Examples are the comparison of two models for a domain or generalizing a given model.

Evaluation tasks aim at assessing the ability of students to make judgements based on given criteria, standards or guidelines. Examples in domain modelling education are finding mistakes in a given domain model by comparing it to the given description or choosing the model that describes the given domain best and motivating one’s decision.

Creating tasks imply the use of the learned material to create a new structure or to enhance an existing one. This is the highest cognitive level of the taxonomy. Examples in conceptual modelling education are building a model according to a given description and completing an incomplete model.

Knowledge Levels

Factual knowledge includes the basics of the studied disciplines, such as basic terminology. In domain modelling, factual knowledge refers to the knowledge of modelling notation and definitions of various terms and concepts.

Conceptual knowledge implies understanding of the connections and interrelationships between the elements learned on factual level. For domain modelling, conceptual knowledge is related to the understanding of relationships between modelling concepts and between various elements of a given model or model fragment.

Procedural knowledge refers to the subject-specific methods, procedures and rules. Procedural knowledge means knowledge of modelling techniques and criteria for their implementation; it may include the step-by-step approaches to modelling and knowledge of guidelines and procedures specific to the discipline of domain modelling.

Metacognitive knowledge implies strategic knowledge and the student’s awareness of his/her own knowledge. Metacognitive knowledge in domain modelling is related to knowledge about the (typical) mistakes a student (or a group) tends to make, the most successful strategies for learning, and the knowledge of cognitive processes that would be involved in a given task.

3.2 Scaffolding Levels

At first, we attempted to classify the assessment tasks directly into the Bloom’s taxonomy. We however soon faced the problem that certain tasks require prerequisite knowledge. This scaffolding of knowledge is not adequately captured by either the cognitive process levels or the knowledge levels. Therefore, based on the existing learning paths for domain modelling education, we created the following scaffolding tree (Fig. 1), according to which the modelling levels addressed by learning outcomes were additionally classified.

Fig. 1.
figure 1

Scaffolding tree for domain modelling education

Full size image

This scaffolding tree comprises four major levels:

  • Class level, which includes the concepts of object, class and attribute;

  • Relationships level subdivided into generalization and association sections. The generalization section includes the concept of inheritance, while the association section includes binary, n-ary and recursive associations, aggregation and partly the concept of association class, which was included in both class and relationships levels;

  • Model level subdivided into simple and complex model sections. These two sections are introduced to emphasize the pedagogical difference between models that utilize only a limited amount of modelling concepts and those that use a wide variety of concepts: “simple” model and “complex” model. A simple model implies the use of the whole class level and binary associations, while a complex model may include the whole set of relationships level concepts;

  • General knowledge level includes knowledge of modelling notation languages, general conventions and guidelines for modelling, and other necessary information, which is out of the scope of the above three levels.

The arrows should be read “A is a prerequisite for B”, if the arrow starts in A and points at B.

3.3 Materials

For the assessment of current practice, different sources could be used. As explained in the introduction, general curricula designs identify learning goals at a too high level to be useful for everyday educational practice. Better sources would therefore be the individual courses and how they address domain modelling. Here too, different starting points can be used. The learning goals formulated for a course, often found in online course descriptions, could be a potential source. However, such descriptions are still quite high level. Moreover, they only specify what is planned to be addressed, rather than what is effectively addressed by the course. Better sources are therefore the actual assessment questions used to assess the students’ knowledge of domain modelling. This is also in line with previous research on educational frameworks based on Bloom’s taxonomy, which used the applied assessment items to build evaluation tools and frameworks for their subjects.

Other issues are that domain modelling appears as a sub discipline in different fields (object-oriented modelling, conceptual modelling and database design) and that courses have different formats. In order to achieve a sample as representative as possible, the choice of materials was made such as to cover different forms (books, online courses (MOOCs), and face-to-face courses) and different communities. This resulted in the following sources:

  • Four books were chosen from different modelling communities: object-oriented modelling, conceptual data modelling and database design [3, 4, 7, 8]. For each of the books we chose a seminal work having more than 100 references on ResearchGate and, at the same time, containing exercises. Though some books had higher citation rating than those that were picked, the majority of those books did not contain a set of exercises based on which learning outcomes could be derived.

  • All openly available higher-education level MOOCs. Each from a different platform: edX, Open University and Stanford Lagunita [9,10,11].

  • Face-to-face courses from three universities in Belgium: KU Leuven, Université Catholique de Louvain and University of Namur [12,13,14,15,16].

For the books and the MOOCs, all relevant exercises and assessment tasks were classified. Hence, both intermediate and final assessment tasks were represented. For the face-to-face courses, exam questions (final assessment) were collected directly from the teachers. In this case, material from exercise sessions were not collected, to minimize the burden for the teachers willing to share their material with us. In total, 291 assessment tasks from 12 resources were analysed.

3.4 Classification Process

The classification was conducted by the two authors of the paper. The assessment packages were assessed separately, with a high level of interrater agreement (less than 10% of the learning outcomes had to be discussed due to the disagreement in classification). In case of a disagreement, a thorough discussion and study of materials of the particular educational resource was conducted to determine the exact meaning and logic of the task and when necessary, the domain specific interpretation of the Bloom’s taxonomy was further clarified.

4 Results

4.1 Illustration

As an illustration, we are giving an example of how the classification of learning outcomes (LOs) was done for one task from [8].

LO4.9:

Discuss the conventions for creating a class diagram.

For this task, the following levels were determined:

  • Cognitive process level – understanding. Discussion tasks, by definition, are aimed at understanding a given concept or a set of rules/conventions. The border between brainstorming (creating a new set of conventions) and discussing (trying to understand the meaning and logic of the existing conventions) is crucial in such types of tasks.

  • Knowledge level – procedural. Conventions can be considered a subject-specific procedural knowledge: the knowledge of how the model is designed and why it is designed this way and not any other in order to comply with community standards and be readable by other modellers.

  • Scaffolding level – general. General level includes the general knowledge about modelling; rules and conventions, as well as modelling notation languages, are considered part of this general knowledge.

A similar way of reflection was conducted for every learning outcome analysed in this work. The full list of identified learning outcomes translated into UML terminology is available online [33].

4.2 Bloom’s Taxonomy for Domain Modelling

Table 1 demonstrates the normalized results of the classification. The results are normalized to avoid overrepresentation of data from resources that contain more assessment items than others do (e.g., the average amount of exercises in books exceeds the amount of exercises in MOOCs). The table should be read as follows: the columns represent the assessment packages classified in this study (“B” stands for “Books”, “M” for “MOOCs”, “U” for “University exams”). The rows represent the dimensions of the revised Bloom’s taxonomy: in bold the cognitive dimension, as sublevels the knowledge dimension. If a certain knowledge dimension of the four (factual, conceptual, procedural, metacognitive) does not appear in the table, it means that none of the assessment items fit into the category.

Table 1. Summary of the results per assessment package – normalized
Full size table

As can be seen from the last column (Grand Total) of Table 1, in the cognitive dimension (boldface), the most frequent tasks are related to the “Understand” level, then, the second place is almost equally distributed between “Analyse” and “Create”, while the third is given to “Apply”. The most underrepresented levels are “Remember” and “Evaluate”. In the knowledge dimension, the metacognitive knowledge level is not represented in any of the assessment packages. The most represented level is Conceptual, with much lower amount of Procedural level questions and almost none Factual.

In Fig. 2, a plot for the summary of results per category is presented. As can be seen, the books give the most evenly distributed tasks among the cognitive levels, while Exams and MOOCs tend to concentrate more on a few cognitive levels. The large majority of exam questions we analysed focus on applying conceptual knowledge and on creating and contained no assessment items related to the understanding level.

Fig. 2.
figure 2

Summary of results per category

Full size image

At the same time, MOOCs are more focused on understanding and analyzing. Unlike the books and the exams, MOOCs provide hardly any creative questions.

4.3 Scaffolding Levels

Figure 3 shows the normalized summary of results related to the scaffolding levels addressed by the assessment tasks. It can be seen from the plot that the majority of questions address model and relationships levels (with particular concentration on complex model sublevel and associations sublevel). Books and exams put most focus on complex models, while MOOCs are rather aimed at assessing knowledge related to associations and simple models.

Fig. 3.
figure 3

Scaffolding levels per category – normalized

Full size image

4.4 Frequent Tasks

The following frequent learning outcomes (more than two appearances of a learning outcome of this type in two or more resources) were identified across the assessment packagesFootnote 1:

Type 1: :

Draw a class diagram/create a domain model according to the given requirements.

Type 1a: :

Draw a class diagram describing a given domain, following the given steps/procedure.

Type 2: :

Make changes to a model to correspond to a new system description.

Type 3: :

Elicit all the possible classes from a given requirements document.

Type 4: :

Suggest attributes for the given classes based on the requirements document.

Type 5: :

Define the multiplicity of a given association in a given domain model.

Type 6: :

Analyze the lifecycle of a given object.

Type 7: :

Find structural issues/ways to improve the given domain model.

Type 8: :

Refine a given model by using a given modelling technique.

Type 9: :

Develop an alternative design for a model.

Type 10: :

Draw an object diagram of a given model.

Type 11: :

Propose the improvements for a given modelling notation.

Type 12: :

Write a complete narrative description of a given class diagram/explain a given class diagram.

Table 2 shows the frequent learning outcomes positioned in the revised Bloom’s taxonomy. When grouping learning outcomes into types, we observed slight variations in the formulation of similar learning outcomes, which resulted in slight differences in classifications. In this table, only the derived learning outcome types are classified.

Table 2. Distribution of the most frequent task types in the revised Bloom’s taxonomy
Full size table

5 Discussion

When looking at the above results, the following limitations of the study should be taken into consideration: The amount of studied literature sources cannot be considered fully representative for the entire field of domain modelling. In order to reach a deeper and more accurate understanding of the current state of the field and the learning outcomes pursued by the educators, more assessment packages from different sources should be analysed. Nevertheless, because materials were sourced from different communities and formats, the results can to a large extent be considered as sufficiently representative for obtaining a first indicative image of the field. Also, the classification process may be improved by introducing a more formal rating system and inviting more raters to evaluate the positioning of each learning outcome in the revised Bloom’s taxonomy. Nevertheless, the fact that the current two raters obtained agreement on almost all items without discussion, is an indication of reasonable confidence in the correct classification. All in all, while the list of material could be extended and the classification could be further strengthened by adding more raters, the overall validity of the results is estimated as fair given the spread of the material sources and the high interrater agreement.

Looking at Fig. 2, one can see that the results show a difference between different educational sources in terms of the addressed cognitive and knowledge dimensions. MOOCs have the least amount of creative learning outcomes compared to exams and textbooks. This may be explained by the nature of traditional MOOC questions: to enable the automated assessment of students’ knowledge, the MOOCs apply multiple-choice questions. Though multiple-choice questions can possibly be designed in a way that they would address higher cognitive levels of Bloom’s taxonomy [17], the development of such questions is time-consuming. The “easier to create” multiple choice questions typically address the lower levels of the cognitive dimension. Nevertheless, all three MOOCs still have a relatively high rating of their learning outcomes, with more than a half of the questions addressing higher levels such as “Apply”, “Analyse”, “Evaluate” and “Create”.

Exams on the other hand show a strong focus on the outcomes related to the higher cognitive dimensions applying, analysing and creating, while outcomes focusing on understanding were not present at all. This could possibly be explained by the fact that only (summative) assessment items of the final exams were collected. Teachers may possibly use other types of (ungraded) formative assessments in the course of a semester, but we did not request this material from them to limit the burden of their participation. Books provided the most equally distributed learning outcomes among the three learning material types. This can be explained by the fact that exercises included in books are designed for gradual scaffolding rather than for summative assessment, as in the exams.

All three types have “Remember” and “Evaluate” levels underrepresented, with only a few tasks related to “Remember” and slightly more related to “Evaluate”. The first may be explained by the nature of the discipline itself: unlike some disciplines (e.g. biology with a very high rating of “Remember” [30]), domain modelling is perceived as a skill acquired exclusively through practice rather than through remembering terms and definitions. The lack of evaluation-related tasks is evident not only in the field of domain modelling: a categorization of the main unified exams questions for biology also found that evaluation level was not addressed in any of the five types of assessment packages [17]. Nevertheless, it would make sense to have more “Evaluate” types of assessments as this is the intermediate scaffolding step between “Analyse” and “Create”. The absence of this type of assessments indicates a gap in the scaffolding.

In the knowledge dimension, the most represented level is the conceptual level, while none of the analyzed sources addressed the metacognitive level. The second least represented level is the factual level (which correlates with the underrepresented “Remember” level), and the third least represented is the procedural level. The lack of the procedural level outcomes may be a reflection of the state of the field as such: this could be an indication of an absence of unified (or agreed upon) guidelines, standards and procedures for domain modelling.

Regarding scaffolding levels, the majority of outcomes are related to model- and relationships levels, with books and exams mostly focused on complex models and MOOCs – on simple models. General and class levels are the least represented in the assessment packages. This again could indicate a lack of proper scaffolding: assessments seem to jump immediately to higher levels without proper testing of the lower ones. These gaps are also reflected in the identified frequent tasks: they have a high amount of “create”-related learning outcomes among them, with “understand” and “analyse” in the second place.

Obviously, these results are specific to the discipline of domain modelling and cannot be generalized to a different domain. Even generalisation to other sources from the discipline of domain modelling (books, exams, etc.) should be done with much care: as one can see from Table 1 there are substantial differences between sources, even between sources of the same type.

6 Conclusion and Further Research

In this work, we made the first attempt to classify the domain modelling-related assessment packages into the revised Bloom’s taxonomy and find the positioning of the learning outcomes according to the scaffolding levels indicated in Fig. 1. This first classification exercise leads to the conclusion that, generally speaking, assessment packages of a single source (a book, a face-to-face course, a MOOC) show considerable gaps in scaffolding and overall evaluation of student’s knowledge. Several levels both in terms of cognitive and knowledge dimensions seem to be missing. The most underrepresented is the metacognitive knowledge level, associated with strategies of learning and awareness of student’s own cognition. Factual knowledge level, along with the “Remember” cognitive process are the next least represented in all of the assessment packages, with Procedural knowledge level insufficiently addressed by every source. “Evaluation” cognitive process, which can be considered one of the most important high-level cognitive processes on the way to creation of a model, is also addressed insufficiently in all the analysed materials. Exams and textbooks focus mostly on “Create” cognitive process, while in MOOCs this level is heavily underrepresented.

Similar inequalities are observed in the scaffolding levels addressed by the assessment material: it is highly focused on model and relationships levels, while knowledge about classes and the general knowledge on domain modelling is rarely tested.

Summing the above glimpse into the current state of the domain modelling education, we can conclude that the examined assessment materials are considerably unbalanced, which may cause difficulties both in the teaching and the learning processes. A thorough revision of the existing learning materials and assessment packages and their scaffolding can be suggested to the domain modelling educators.

For the future, we plan to develop a systematic educational framework for domain modelling, based on the revised Blooms taxonomy. This framework will include sample classroom tasks, learning paths and scaffolding approaches, as well as a validated assessment tool for domain modelling. Domain modelling educators could benefit from the identified learning outcomes and use the revised Bloom’s taxonomy as an inspiration for creating their own classroom material and assessment packages. The analysis of learning outcomes could be expanded to other levels of modelling, such as object behaviour and process modelling. In addition, the best scaffolding approach should be identified among those proposed in the educational literature and in the university curricula.