A Continuous Process for Validation, Verification, and Accreditation of Simulation Models

Abstract

A simulation model, and more generically, a model, is founded on its assumptions. Assurance of the model’s correctness and correct use is needed to achieve accreditation. Often the exercise of working with a specific code misunderstands the overall process, focusing the resources on the model coding and forgetting the needed resources to ensure the validation of every step of the model definition and coding. The goal of this work is to present a methodology to help in the definition and use of the assumptions in the modeling process. To do so, we present a process to conduct a simulation project, an assumptions taxonomy, and a method that simplifies working with those assumptions. We propose to extend the traditional Validation, Verification, and Accreditation processes to a process composed of eight Validation, Verification, and Accreditation phases that cover the overall life cycle of a model. Although this paper is focused on a simulation model, we can extend the proposed method to a more general modeling approach.

1. Introduction, the System, and the Model

We want to start with a brief discussion regarding the system and the model. The system is the subject of our analysis; that can be something real, something physical that exists or not. On the other side, we have the model, the abstraction we will use to represent this system.

More specifically, a system is defined in the frame of systems engineering as “a construct or collection of different elements that together produce results not obtainable by the elements alone.” (“Systems INCOSE-Systems Engineering-UALR”) [1], while a model is “(…) a representation and abstraction of something such as an entity, a real system, a proposed system design or an idea” [2].

Therefore, following the proposal of [3], the main properties a model owns are:

• Mapping: The goal of models is to represent something, i.e., mappings from natural or artificial systems or models.
• Reduction: Models, in general, do not describe all the features of the original system, rather, only those features that seem important to modelers or system stakeholders.
• Pragmatism: The models themselves are not explicitly assigned to their originals. They fulfill their surrogate functions (a) for certain cognitive and/or behavioral model-like subjects, (b) over certain time intervals, and (c) restricted to certain mental or practical processes.

These properties must be taken into consideration when one builds a model. Specifically, the process of building a simulation model is iterative, usually involving different personnel. This process starts by describing the system we want to represent, that is, the object of our analysis, and from it, we start a description of the key elements that must be considered to define the model. Once we understand what is “my system” we can go further to describe the problem that we are trying to solve. A good thought is that “all models are wrong (…)”, but some of them are useful [4]. Therefore, to ensure that we are applying a useful model, we follow a validation process since our model, although not completely correct, can be useful to our purposes: “A simulation model should always be developed for a particular set of objectives. In fact, a model that is valid for one objective may not be for another” [5].

The core of the model is its assumptions. They represent how we understand the system and express how we will conceptualize the model and, from this, how we develop the coding. In simulation [6,7], the Validation and Verification of the model’s assumptions become key aspects to ensure the correctness of the conclusions obtained from a model. The model can be viewed as an experimental framework able to conduct new experiments [8], whereas wrong models can provide interesting insights [9] for both modelers and clients.

Validation, Verification, and Accreditation processes in simulation have been widely discussed, defining structures, diagrams, and recommendations that help the modeler ensure that the modeling process will succeed. One can review [10] for a simplified version of the model development process, and a more complex view developed by the same author can be found in [11]. The experimental design for a simulation model is detailed in [12], while [2] proposes a set of rules for the Validation, Verification, and Certification of simulation applications. The progress and challenges of a key element in the process of a description of a simulation model, the conceptual modeling, are discussed in [13].

In this work, we will present a pragmatic approach with practical use for modeling, allowing us to identify guidelines to follow in the process of defining a simulation model that must be accredited to eventually implement some solutions in the system. We discuss the current state of the techniques that can be used for the validation of the assumptions, presenting their integration with an improved cycle of a simulation model and showing their application in an industrial case. We will also discuss how this approach has an impact on the creation of a digital twin in the frame of Industry 4.0.

Also, in this paper, coded means writing: that is, implementing the program that is going to execute the model. Implementation means the modification of the system according to the accepted results. Validation means ensuring that the model is correct, while Verification means ensuring that the model has been correctly coded. Accreditation means that the stakeholders believe in the model and will use it to implement or modify (according to the model results) the system. We also want to mention that assumptions and hypotheses are often considered equivalent but are not. A hypothesis must be tested by an experiment, while an assumption can be accepted tacitly. In any simulation model, all the assumptions will be formulated through a conceptual model or a computerized model in a mathematical framework and must be assessed continuously. Therefore, we will be able to define an explicit method to evaluate the validity of these assumptions; hence, we will talk about hypotheses and not assumptions when explicit processes to test these assumptions exist. However, since we do not know if this process will exist in the development of our simulation model, we will use the term assumptions in this document for the sake of simplicity.

This paper is structured as follows. First, in Section 2, we review the existing approaches. Next, in Section 3, we present the process to define a simulation model, an iterative approach that generates different products. In Section 4, we present the assumptions taxonomy. In Section 5, we discuss how to validate those assumptions using the iterative cycle presented in Section 3. We present a table (W3H) that systematizes the selection of a subset of the tests one can use to perform the validations; moreover, we discuss the utility of work with non-validated assumptions. In this section, we also present the assumptions table, a product that helps in the model conceptualization and assumptions use and maintenance. In Section 6, we present an example based on a real project developed in the context of the definition of a Digital Twin for a factory. Finally, some conclusions are drawn in Section 7.

2. Existing Approaches

The classic Software Development Life Cycle (SDLC) is composed of four stages (i) planning; (ii) analysis; (iii) design, and (iv) implementation [14]. From this, we can note that several different Software Development Models exist, such as Waterfall, V-Model, RAD model Iterative model, Agile, Spiral, and Prototyping models, among others [15,16].

In the case of a simulation model that will end up in the coding of a computer program, we will conduct similar phases for the implementation and verification of the simulator. This process will be guided continuously through the Validation, Verification, and Accreditation (VV&A) processes. This guidance is because of the specific particularities of modeling and due to the experimental nature of the results that must be implemented in the system. Therefore, during the process, it is necessary to ensure that not only the computer program fits well with the conceptual model but also that the conceptual model and the assumptions they represent are correct enough to obtain valid conclusions regarding the system [10]. In the case of a simulation model, the system (our object of the analysis) is not necessarily a piece of software, but a factory, the environment, a society, or any other system. Therefore, the VV&A process guides the complete development of a model to ensure that all the assumptions we will use in the model development are correct.

Several proposals exist to conduct the VV&A cycle in a simulation project; [10] presents an approach to ensure the validity of a model using a simplified cycle that presents the key elements of the simulation model building. A more complete vision is presented in [11], including a distinction between the real world and the simulated world. There is a distinction between the “System”, “System Theories”, “Conceptual Model”, “Simulation Model Specification”, “Simulation Model”, “Simulation Model Data/Results”, and the “System Data/Results”, composing a process that conducts the modeling process. Another proposal for a detailed cycle for modeling and simulation is presented in [17]. It encompasses the different elements that one can find in a simulation model like documents (“Formulated Problem”, “Requirements Specification”, “Conceptual Model”, “Architecture Specification, “Design Specification”, “Executable Submodels”, “Simulation Model”, “Simulation Results” and “Presented Results”). This proposal is categorized, according to [18], with the different phases where a simulation model could be: the Problem and Requirement phase, the Design and Programming phase, the Simulation and Certification phase, and the Storage and Reuse phase.

Coherently in [5], assume that Validation, and specifically Data Validation [11], owns a central role in the VV&A process. Therefore, a proper understanding of the nature of the assumption becomes relevant. In [19], the authors provide a taxonomy for modeling assumptions, hypothesizing them, and proposing a use in simulation experiments. They argue that depending on how the assumptions are constructed, one can classify them as either constructive or non-constructive and phenomenological or mechanistic, depending on the abstraction level. With this initial classification, the proposed taxonomy classifies hypotheses as (i) phenomenological hypotheses, which should make statements about the relationship between the simulated inputs and outputs; (ii) mechanistic hypotheses that define the theoretically grounded model’s fundamental mechanisms that produce and maintain the desired behavior, and (iii) control hypotheses related to the optimization of behavior. In [20], the authors provide a detailed process for conducting hypothesis-driven simulation studies and show how such explicit hypotheses models can facilitate validation processes. Both [21,22] present a more general approach, describing the phases of the life cycle of a simulation study based on the use of assumptions. These studies try to accommodate the structure of the assumptions used in a simulation model but not by outlining how one can perform a validation of the model assumptions.

However, currently, there is no complete view of the overall process that combines the description, classification, and use of the model assumptions and how to manage them in a detailed Validation, Verification, and Accreditation process. This will help to detect what tests can be applied to each assumption to increase the credibility and accountability of the models. Moreover, there is no description of the impact of the Digital Twin concept in the context of this process. In this work, we propose an approach that, from a very pragmatic point of view, allows for an understanding of the nature of the assumptions we use and how to use them in a complete VV&A life cycle that also encompasses the Digital Twin concept.

3. Defining a Simulation Model

The complete VV&A cycle that guides the development of a simulation model by introducing all aspects that must be validated in the cycle of a modeling project is presented in Figure 1. With the emergence of the Digital Twin concept (in the frame of Industry 4.0), some validations, like “Solution Validation”, emerge as a key element to ensure the correctness of the proposed solution and the implementations completed in the system. This allows us to define the concept of model-based discussion that starts from the software engineering concept of Model-Based Systems Engineering [23]. This is in line with the hypothesis-driven simulation studies as proposed in [24], the formal specification hypothesis [21], and also the goal [19]. Notice that due to the cyclic nature of modeling, different versions or prototypes can appear. This will be a source of discussion to understand the validity of the assumptions by performing an analysis between different assumptions used on the different model versions.

The Goals, that is, why we are building the model, determines the definition of the Problem Entity that defines the assumptions that are going to be used on the project and formalized with the Conceptual Model, along with the Scenarios/Configurations definition. Although the goals have a clear impact on the Problem Entity and consequently on the Conceptual Model and the Scenarios/Configurations of the model, they rule the overall development of the project. As an example, the selection of the software (or tools) that must be used to do the model coding and the subsequent experimentation with the model must be guided by a clear definition of the goals that will be translated into a set of metrics that allow this selection following a proper methodology [25,26]. Therefore, depending on the goals, the software to be used to code the model must own some specific features or others.

Each one of these products can end up with different documentation. Here we propose to define a document for each of these products, as is presented in

Table 1. Proposed documents were obtained in each product of the modeling cycle.

PRODUCT	DOCUMENTATION	EXPLANATION
SYSTEM	System documentation	All the documentation regarding the system operations. Tacit and informal information can also be used here to understand the system’s true behavior.
GOALS	Goals document	A document that contains the goals that drive the development of the model.
PROBLEM ENTITY	Assumptions document	A document that contains the assumptions of the model.
CONCEPTUAL MODEL	Conceptualization of the model	The conceptual model constitutes a specification and includes all the assumptions (not the simplifications) detailed in the Hypotheses document.
SCENARIOS/CONFIGURATIONS	Experimental design definition	A document containing the experimental design. A discussion regarding the quality of the RNG used in the experimentation can be added here if needed.
COMPUTERIZED MODEL	Code and technical documentation of the code	Along with the code, the technical documentation explains the code and the needed configurations of the computational infrastructure needed to execute and maintain the model.
SOLUTIONS	Solutions document	Description and data containing the solutions obtained from the model’s execution.
ACCEPTED SOLUTIONS	Executive summary of the project	Conclusions of the modeling project and suggestions to perform the implementation on the system.

. This documentation allows presenting the product, but also it will be present the process of its development; hence, these documents are living documents updated continuously along with the life of the system and the model they represent.

To transform one product into another product, we have the actions. They are (i) Problem Definition, (ii) Analysis and Modelling, (iii) Coding, (iv) Parametrization, (v) Calibration, (vi) Experimentation, (vii) Understanding of the solutions, and (viii) Implementation. All these actions finish with a product that is going to be enough for our purposes depending on the validation processes that must be done continuously.

The different validation processes one must do are the: (i) Conceptual Model validation, which allows detecting if the definition of the model is complete and the model allows to answer the questions proposed by the Goals; (ii) Data validation; (iii) Verification of the model coding; (iv) Operational validation; (v) Experimental validation; (vi) Accreditation, and finally (vii) Solution validation. These validation processes can be done depending on the status of the development of the model. Notice that the definition of the goals also implies a validation, (viii) Goals Validation, related to the system management, which will be focused on assuring that the goals are aligned and detailed completely and correctly, with the main purposes of the system stakeholders.

As modeling is an iterative process, it generates several prototypes during the different iterations. In each of these iterations, the assumptions can be modified according to the goals and the evolution of the several products that answer those goals, driven by the actions and the validation processes. The iterative nature of modeling makes it natural to use an approach to coding like Agile, Prototyping, Iterative, and Incremental or also Rapid Application Development (RAD) methodologies.

Before entering more details regarding the different validation processes to be done, in the next section, we discuss the assumptions of the model and how a taxonomy of those assumptions helps in the different validation processes that must be done to ensure the correctness of the products.

4. Working with the Assumptions

Systems can be complex realities. Even though when we can work with simple systems, we need to use assumptions to describe what the system is. Once it is clear what the system is, we must define a model that represents and connects all the assumptions through the model structure and the behavior.

This model must be conceptualized using a formalism independent of the selected implementation tool. The tool must be selected depending on a clear set of metrics ruled by the goals and never by any other preference [25]. Several books and papers detail the advantages of using a conceptual model. Here, we want to focus on three aspects. First, the model conceptualization can be considered a product by itself [27]. We support this, as represented in Figure 1, because the knowledge representation that rules the different processes in a system can sometimes be more interesting than the simulation by itself. The conceptual model transforms the tacit knowledge of the system into explicit knowledge that can later be used to make decisions. A formal representation of a system, through the model conceptualization, helps to understand how the system behaves.

Second, a formal model representation simplifies its coding and enhances its maintainability. Third, a conceptual model simplifies the model understanding by the different actors that participate in the simulation project. This improves the communication between diverse profiles simplifying the validation processes, a key aspect of the IT/OT convergence (Information Technology and Operations Technologies) in the frame of Industry 4.0.

Since the conceptual model is founded on assumptions, the process of validating a model is the process of validating the different assumptions used to build the formal representation of the simulation model and accepting the needed assumptions as valid. The conceptual model starts with the definition of the assumptions document. This document represents how we understand the behavior and the structure of the System under the lens of the Problem Entity: the reason we build the model. The conceptual model is the formal representation of the assumptions we are going to validate in the process of building the model. However, it is first necessary to detail the nature of the assumptions. An initial proposal of this taxonomy is presented here [28], defining three main types of assumptions: Systemic Data assumptions, Systemic Structural assumptions, and Simplification assumptions.

Assumptions Taxonomy

First are those assumptions that allow us to define how the system behaves. If we have a deeper knowledge of the system, more assumptions describing its behavior can be used. We want to add as many as we can of these assumptions (if they are useful for the model goal) since they help us in the description of the model. We suggest naming these assumptions Systemic since they describe the behavior of the system. These assumptions represent our current knowledge of the system and are assumed to be true (but may not be true). These assumptions can also be divided into two categories.

(i) 

Systemic Data assumptions are those related to the data, for example, those that define the probability distributions that represent the model element’s behavior.

(ii) 

Systemic Structural assumptions can be those that represent the relations between the different elements that compose the model, the model behavior, and the causal relations between the elements.

Second, there is a category composed of assumptions that simplify the model we are going to build.

(iii) 

Simplification assumptions are useful to reduce the complexity of the model. Because of the resources, the time we own to code the model, and the limited knowledge we have of the system, we must always use these kinds of assumptions. They represent our current limitations in two directions: (i) system knowledge or (ii) resource knowledge and are assumed to be false (but may be true, becoming Systemic). We want to keep the model small enough to be useful, keeping the model simple but not simplistic. Notice that in the hypothetical case that we have a complete understanding of the system and infinite resources, there is no need to use simplification assumptions in our models.

This taxonomy has the main goal of simplifying and helping in the VV&A process and informing the understanding of the nature of the decision taken during the modeling process. They also represent the current limitations and understanding of our system; hence, as we will show next, it will be interesting to store this information in a structure named the assumptions document.

As an example, considering a linear model, one can introduce an error (i) when omitting a relevant variable (because we do not know it, in that case, this is a Systemic Structural assumption error, or because we assume that it is not relevant, in which case we are using a Simplification assumption erroneously); (ii) when including an irrelevant variable (using a false Systemic Structural assumption), or (iii) when specifying a linear relation that does not exist (again, using a false Systemic Structural assumption). The meaning of the assumptions is the key to interpreting the causality of the model and performing an accurate Validation. In addition, following the example of a linear model, the hypotheses regarding the data (independency, normality, homoscedasticity…) are Systemic Data assumptions that must be satisfied. As we will see next, we can use a model that does not have all the assumptions validated if we clearly understand our model goal and what is the nature of the assumptions that fail.

5. Validating a Simulation Model

Validated means that the Systemic assumptions are tested (or assumed) as true, and the Simplification assumptions are tolerable by all the parts involved in the modeling project and supported by the results. However, we cannot ensure that a model obtained from a set of assumptions is true (we can only ensure that a model is false [29].) but useful for our goals. Modeling can also be useful for many other reasons, instead of predict [30], we can assume that a model is valid for a specific purpose (but false for others), implying that the underlying assumptions are valid for this purpose.

Hence, what do we mean by Validation? Validation can be seen differently depending on the view used to understand the assumptions [31], with the following being the more common approaches [32]: (i) the rationalist view (where the model becomes a structure that starts from premises that act as axioms); (ii) the empiricist view (where the model must be tested empirically with experimental data), and (iii) positivist view (where a model is going to be considered valid only if accurate predictions are going to be obtained).

To conduct the Validation process in a simulation project, Naylor and Finger [32] proposed combining the three historical methods of rationalism, empiricism, and positivism into a multistage process of validation, the multistage or utilitarian approach. This validation method consists of [32]:

(i)

Developing the model’s assumptions, observations, and general knowledge.

(ii)

Validating the model’s assumptions and testing them empirically if possible.

(iii)

Comparing the input-output relationships of the model with the real system.

Of the different processes we propose in Figure 1 that one must conduct to ensure the correctness of the solutions implemented in the system, the essential steps that must be completed are:

• Data validation to ensure that the data we are going to use on the model, the probability distributions, the data sources that are used to obtain the data, the support structures to keep the data, etc., are accurate and correctly defined. To validate the data, we must focus our efforts on the Systemic Data assumptions. The tests must focus on analyzing the data we are going to use in the model in two directions: (i) assuring that the data fits well with the system process it wants to model (data fitting) and (ii) ensuring that the data expiration constraints are valid, therefore, guarantee that the institutions (structures, enterprises, data warehouses, etc.) needed to obtain the data works and keep this data valid during the life of the model. Defining data expiration constrain is a key aspect to ensuring the validity of the simulation models (that eventually will be part of a Digital Twin in the frame of Industry 4.0) since the model is going to be a representation of a real system, analyzed along with the life of the system that emulates.
• Conceptual model validation is determining (i) the correctness of the conceptual model assumptions and (ii) that the problem entity model’s representation, the model’s structure, and behavior, represented by the logic and the mathematical and causal relationships, are “reasonable” for the goals. To validate the conceptual model, we must focus on the formal representation of the model or on the coding of this formal representation to validate the Systemic Structural and Simplification assumptions. The conceptual model represents the structural relations of the different model elements and the behavior of the different elements that compose the model. This implies that we must define techniques to ensure that (i) the Systemic Structural assumptions are correct for our purposes and (ii) the Simplifications assumptions do not transform the model into a simplistic version of the reality it represents. Several formal languages like Specification and Description Language (SDL) [33,34,35], Petri Nets [36,37,38,39], and DEVS [40,41,42] allow an analysis of its behavior before the coding (reachability analysis, etc.). Moreover, there are methods to transform a conceptual model from one to another, becoming excellent tools to ensure that the underlying assumptions are correctly defined [43,44]. This also simplifies the integration between different conceptual models that express the same system or different parts of a system that must be defined as one following meta-formalism, common formalism or co-simulation approaches [45,46].
• Operational validation is focused on the results that we can obtain from a model implementation. Some of the methodologies that we present next, like Black Box validation, among many others, imply the use of all the model assumptions since the modeler tries to validate the whole behavior of the computer program that codes the model. In that case, we often cannot distinguish if the results (in case the results are wrong) are due to incorrect Systemic assumptions (Data or Structural), due to a wrong Simplification assumption, or an incorrect model coding, introducing the verification phase into account. However, Operational Validation is one of the most widely used approaches to do validations and must always be done to ensure that all the connections between the distinct parts have been correctly defined.
• Verification ensures that the code has been completed correctly. The techniques at this point are not focused on understanding if the assumptions are correct, but if they have been correctly represented in the computational framework, we use them to implement the model.
• Experimental validation analyzes if the experimental procedures used to obtain the results are good enough. A simulation model needs to be executed based on an experimental framework; hence the design of experiments and the execution of those experiments must be validated. One must ensure that the factors needed and the levels to be analyzed are correctly defined for the goals proposed. We must also ensure that the number of replications for each scenario in the experiment has been correctly obtained (along with the methods used to obtain these replications and the length of the replications). Without this validation, the results of a model cannot be considered for its implementation in the system.
• Solution validation focus on the accuracy of the results obtained from the model proposed solution and the data obtained on the implemented solution. This is a key validation in the frame Industry 4.0 Digital Twin concept since it ensures that the model and the system are close enough to be useful for the goals proposed. Solution Validation is a useful validation for the modelers that will eventually learn about the divergences between the model proposed solutions and the implementation of the accepted solutions on the system. This will allow further modeling improvement through the detection of errors. It is also useful for the owners of the system to understand if the different validation processes, specifically those related to the accreditation, are working properly. Notice, however, that because this validation can show flaws in the accreditation process, in the model validation and verification processes, or during the system implementation, this validation is sensible and often forgotten. However, in the frame of cyber-physical systems (in Industry 4.0), this validation becomes a key element since the Digital Twin must accurately represent the current system, which is prone to change. This solution validation traditionally has been used to test if the model can also be used as a product, often to test the behavior of the system in other scenarios, see [47,48]; in the context of Industry 4.0 is a continuous process to ensure that the simulation model is yet valid. The nature of techniques one can use in the Solution Validation process and the assumptions that are going to be involved are similar to the ones used in the Operational Validation process.

To detect the assumptions that are validated depending on the method used, we propose a classification in the next section of some of the well-known validation methods and what the assumptions are that they try to validate. This is not an extensive list of the methods that can be used, but just a starting point to define a taxonomy that systematizes the overall Validation, Verification, and Accreditation process in a model.

5.1. Testing the Model Assumptions

In [2,5,11,49,50,51,52,53], one can review a description of the different tests one can apply for simulation validation. In Appendix A.1 VV Tests, we present a description of those tests following the proposed method of this paper. However, since the resources we have are finite, we must establish some classification to select the more appropriate test to be used. In [54,55], is presented a classification of the tests one can apply to a model according to the formal, informal, static, or dynamic nature of the techniques. We adapt this table using the Informal/Formal and Static/Dynamic categories as dimensions. With this idea, we propose

Table 2. Classification of different validation tests, depending on their nature.

	STATIC	DYNAMIC
INFORMAL	✓ Face Validation ✓ Walkthroughs ✓ Audit ✓ Desk Checking ✓ Documentation consistency checking ✓ Inspections ✓ Reviews ✓ Traceability Assessment ✓ Interface Analysis (model interface; user interface)	✓ Visualization/Animation ✓ Turing Test ✓ Graphical Comparisons ✓ Acceptance Testing ✓ Object-Flow Testing ✓ Compliance Testing (authorization; performance; security; standards) ✓ Product Testing ✓ Alpha Testing ✓ Beta Testing
FORMAL	✓ Induction ✓ Inference ✓ Logical Deduction ✓ Lambda Calculus ✓ Predicate Calculus ✓ Predicate Transformations ✓ Proof of Correctness ✓ Cause-Effect Graphing (State Transition analysis…) ✓ Goodness-of-fit ✓ Data Analysis (data dependency; data flow) ✓ Control Analysis (state transition; calling structure; analysis of concurrent process; control flow) ✓ Semantic Analysis ✓ Structural Analysis ✓ Syntax Analysis ✓ Fault/Failure Analysis	✓ Partition analysis ✓ Path Analysis ✓ Symbolic Execution ✓ Assertion Checking ✓ Inductive Assertions ✓ Fault/Failure Insertion Testing ✓ Interface Testing (data; model; user) ✓ Special Input Testing (boundary value; equivalence partitioning; degenerative; extreme input; invalid input; real-time input; self-driven input; stress; trace-driven input; fixed values)	TEST MODEL LOGIC
		✓ Black-Box Testing ✓ Bottom-Up Testing ✓ Top-Down Testing ✓ Debugging ✓ Execution Testing (monitoring; profiling; tracing) ✓ Field Testing ✓ Predictive Validation (historical data) ✓ Regression Testing ✓ Sensitivity Analysis ✓ Statistical Techniques ✓ Sub-model/Module Testing ✓ Symbolic Debugging ✓ Structural (White-Box) Testing (branch; condition; data flow; loop; path; statement) ✓ Comparison Testing ✓ Event Validity	TEST MODEL OUTPUT DATA

. Informal tests are those tests where a high range of subjectivity exists because they are based on human perception. Informal tests require the intervention of specialists. Hence in this type of test, specialists are a needed resource. Static tests are those tests that do not need the execution of the simulation model to be applied. Static tests will always be focused on the model structure, while dynamic tests can be focused on the model structure but also on the output data we obtain from this execution. Dynamic tests need the coding of the simulation model to be conducted. Formal/Dynamic tests can need real data (focused on testing the model logic) or can use fake data (focused on testing model data, mainly its output). This generates a subcategory, as is presented in Table 2.

Using this table as a starting point, we go further, analyzing what assumptions will usually be tested in each test and when (the validation process) the test can be applied. This classification is intended to be used as an approximation to understand when applying a specific test.

5.2. W3H Testing Table

Table 3. The final classification of the assumptions validated in each test depends on how the team is implementing the tests, but this table helps in the understanding of where to put the effort in the validation process. Finally, all the different validation processes must be done, and ideally, all the assumptions must be tested through the different validation phases.

WHEN	WHAT	HOW		WHY
		Formal (F)/Informal (I)	Static (S)/Dynamic (D)	Systemic Structural	Systemic Data	Simplification
CONCEPTUAL MODEL VALIDATION	Audit	I	S	+		+
	Cause-Effect Graphing	F	S	+		+
	Desk Checking	I	S	+		+
	Documentation consistency checking	I	S	+		+
	Face Validation	I	S	+		+
	Induction	F	S	+		+
	Inference	F	S	+		+
	Inspections	I	S	+		+
	Interface Analysis	I	S	+		+
	Lambda Calculus	F	S	+		+
	Logical Deduction	F	S	+		+
	Predicate Calculus	F	S	+		+
	Predicate Transformations	F	S	+		+
	Proof of Correctness	F	S	+		+
	Reviews	I	S	+		+
	Traceability Assessment	I	S	+		+
	Walkthroughs	I	S	+		+
DATA VALIDATION	Events	F	D		+
	Goodness-of-fit	F	S		+
	Statistical Techniques	F	D		+
VERIFICATION	Control analysis	F	S	+
	Data Analysis	F	S	+
	Debugging	F	D	+
	Execution Testing	F	D	+
	Fault/Failure Analysis	F	S	+
	Interface Testing	F	D	+
	Semantic Analysis	F	S	+
	Structural Analysis	F	S	+
	Symbolic Debugging	F	D	+
	Syntax Analysis	F	S	+
OPERATIONAL VALIDATION	Assertion Checking	F	D	+
	Black-box Testing	F	D	+	+	+
	Bottom-Up Testing	F	D	+	+	+
	Comparison to other models	F	D	+	+	+
	Compliance Testing	I	D	+
	Degenerative	F	D	+
	Extreme conditions	F	D	+
	Fault/Failure Insertion Testing	F	D	+
	Fixed values	F	D	+
	Graphical Comparisons	I	D	+	+	+
	Historical data	F	D	+	+	+
	Inductive Assertions	F	D	+
	Object-Flow Testing	I	D	+	+	+
	Partition analysis	F	D	+	+	+
	Path Analysis	F	D	+
	Sub-model/Module Testing	F	D	+	+	+
	Symbolic Execution	F	D	+
	Top-Down Testing	F	D	+	+	+
	Turing tests	I	D	+	+	+
	Visualization/Animation	I	D	+	+	+
	White-Box Testing	F	D	+
EXPERIMENTAL VALIDATION	Sensitivity analysis	F	D	+	+
ACCREDITATION	Acceptance Testing	I	D	+	+	+
	Alpha Testing	I	D	+	+	+
	Beta Testing	I	D	+	+	+
	Product Testing	I	D	+	+	+
SOLUTION VALIDATION	Field Testing	F	D	+	+	+
	Regression Testing	F	D	+	+	+

details when to do the test, what test to do, how to do the test (if the test is Dynamic, needs the model execution, and if it is Informal, needs the experts on the system participation), and why, that is the nature of the assumptions that will be tested. The final goal is to test whether all the assumptions are valid or useful for our model. Following this table, if we want to validate our simulation model, we need to apply at least one test to each assumption. This implies the selection of the tests that allow doing this better if we can select tests that are strongly focused on specific assumptions rather than all assumptions to allow us to detect possible sources of error.

Notice that simplification assumptions are not validated in the strict sense. Since they are not introduced in the model, they represent the limits of our knowledge or the resources we own. This is the reason we do not detect a single validation method that is focused only on the Simplification assumptions; doing so implies the modification of the model, usually increasing its complexity and adding new Systemic assumptions to discard the simplification.

5.3. Working with Not Validated Assumptions

Validate a simulation model is a time-demanding and iterative task: see Figure 1. Achieving a fully validated model also needs a growing (exponential) number of resources [56]. In our path to find a useful model, we must often accept working with non-validated assumptions at some point to advance the model development.

Table 4. Effects of using no validated assumptions.

	Systemic Structural Assumptions	Systemic Data Assumptions	Simplification Assumptions
Validated	Wanted	Wanted	Useful
Non-validated	Unwanted	Useful	Unwanted

shows if this situation is wanted, unwanted or useful because of the expected model results. The desired situation for an assumption is represented by “wanted”; “useful” can be transitory of a final situation for an assumption (consider it at the end of the completion of one of the sub-products following our SDLC) that can be needed for the model construction, and “unwanted” are those states that are not desirable for any purpose of the simulation model, hence are just transitory states. We want to mention that the use of Simplification assumptions can be useful but never wanted. The final objective of a model is to work with as few simplifications as possible (or without simplifications). As we noted previously, simplification assumptions will always be false; we know that reality is more complex than the model.

Systemic Structural assumptions represent the relations between the different elements that compose the model. If these relations are not well-defined, the model is not correct: the causality is the crux of a simulation model [57]. For that, using no validated Systemic Structural assumptions is an unwanted state.

For Systemic Data assumptions, using no validated data can be useful for testing purposes and to allow advance in the model definition without waiting for a dataset that needs time to be defined, prepared, etc. In some cases, it is necessary to use fake data to validate the behavior of the model (as an example in extreme conditions tests). Therefore, in these contexts, the use of no validated Systemic Data assumptions can be useful.

Finally, if we have validated Simplification assumptions in our model, we suppose their utility to accomplish our expected result considering the project constraints (technology level, budget, time, computational resources, knowledge, etc.). If these simplification assumptions are not accepted, that implies that we are using some simplifications in our model that the Stakeholders cannot assume. This situation is dangerous for the project, and often reflects bad communication with the client. Communication with the client from the beginning of the project and the definition of a good assumptions document are key elements for the success of a simulation project [5]. Again, note that working with all the assumptions in the “wanted” state is (often) unrealistic since this implies avoiding the use of simplification assumptions.

From this table, we can understand that the status of an assumption can be true, valid (when it is not true but is useful), and false.

5.4. Working with this Taxonomy, Writing the Assumptions Table, and Building the Conceptual Model

For the success of a simulation project, the assumptions document is a key element [5]. This document can start with some initial meetings between the Stakeholders and modelers and describes the model assumptions in natural language. This document must be simple and clear and, as we will show next, defined as a structure. In the frame of Industry 4.0, we propose to transform this document into a structured table that can be stored in a database, with an identifier that allows us to automatically detect whether a specific assumption is valid or not and where these assumptions are used on the model. We will also refer to a condition for the assumptions that allow understanding if these assumptions need revision, for example, the Systemic Data assumptions that have their own expiration date.

However, to accurately describe and use the Systemic assumptions, we must use formal language. This allows us to describe the structure and the behavior of the model completely and unambiguously. A formal language like SDL, DEVS, or Petri Nets, among others, becomes an excellent tool to represent the Systemic Structural assumptions and perform the conceptual model Validations. Hence along with the assumptions table, we must take care to keep the conceptual model representation of the system up to date to reflect our model.

The structure of the assumptions table must include the following:

• Type of the assumption: Systemic Structural assumptions, Systemic Data assumptions, or Simplification assumption.
• Identifier, which uniquely describes the nature of the assumption: as an example, SD_01 refers to the Systemic Data Assumption number “1”.
• Description of the assumption.
• Location on the conceptual model, list of the diagrams that are affected by the assumption.
• The validation status is true if the assumption is assumed correct by the Stakeholders or false otherwise.
• Review condition, a function that modifies the Validation Status to false when this assumption must be reviewed: never (implying that the function is true), by time, or by a more complex set of rules. Systemic Data assumptions often have a data expiration value and hence must be reviewed at this time. In addition, Systemic Structural assumptions can change (because of changes in the system). For the Simplification assumptions, the introduction of a new technique means that we can now consider adding more detail to the model (or eliminating detail).

A Review condition is a necessary element when we model a digital twin (in the frame of Industry 4.0). In that case, the system and the model can follow separate paths, which leads to the need for the revision of the assumptions to keep the model useful. Notice that the assumptions table is a product by itself, and a company may be interested in keeping a digital version of this table to understand the assumptions that rule the behavior of the main company processes, to be used in conjunction with the model conceptualization and the code of the simulation model (that eventually will be used as a part of a digital twin).

As the construction of a simulation model is an iterative task, once we achieve an iteration where all the assumptions of the document have been accepted, we have a simulation model that eventually can be used to make decisions and can be prepared for Accreditation.

6. A Real Case Application Example

Now we are going to show how we will apply this to a real case: see [58]. The project was developed for a real industry that wants to optimize the behavior of a specific area of a production plant. Specifically, we want to perform a detailed simulation of the conveyor that moves some material from one point to another point of the factory. Note that this is not a short conveyor, and its behavior is critical for the good performance of the factory; since it is composed of several sections, it is a complex element that must be accurately represented. We want to configure the conveyor behavior, speed, and capacity to properly configure the machines located at the beginning and the end of the conveyor. The conveyor is connected with other elements, specifically in this example, with a machine named ADP, which provides the input material that the conveyor transports to a robot that will process them to other areas in the factory.

To do so, we talk with the managers to understand the main assumptions of the system, perform the classification, and define the assumptions document. From this, once the different Stakeholders agree on the correctness of the Structural assumptions and the suitability of the Simplification assumptions (to avoid unwanted states), we define a conceptual model based on Specification and Description Language (SDL) and ITU-T formal and graphical language. Note that the initial stages include Systemic data assumptions that have not been validated and some Simplification assumptions that may or may not be making the model too simplistic.

Following the cycle proposed in Figure 1, we must start with a clear definition of the goals. Once this is completed, the assumptions table must be defined, and the different validations are going to drive the achievement of the goals. If validation fails (see Figure 1), we go back to the previous product, revise it, and build again an updated version of the product that fails. The process will drive us back until a product supports the validations that must be done to test the assumptions, and we can complete the actions to build a new product that eventually is going to pass the Validations. The assumptions taxonomy helps to understand the assumptions to be tested in each phase (see Table 3), but sometimes, not all the assumptions can be tested due to the incremental behavior of the modeling process. In our approach, we always present the assumptions table, and on this table, those in red are assumptions that are not yet validated but that we are going to use (considering them valid) to continue with the prototyping modeling process. In order to do so, an identifier for each assumption is proposed as an example. SD_01 is a Systemic Data assumption, SS_01 a Systemic Structural assumption, and SM_01 a simplification assumption.

This identifier, used for each assumption, allows us to later connect the assumptions table with the formal representation of the assumptions done in a formal language (SDL in our case). As an example, Figure 2 shows the first page of an SDL conceptual model for a conveyor element: see the details of the model [58]. The red squares represent those assumptions that are described in the assumptions document. See

Table 5. Assumptions table. The TimeToRoll variable is not yet validated on this iteration; hence all the assumptions that are related to it are considered valid but not true. In this iteration, we also detect that the maximum number of elements that can be on a conveyor is not 10; hence SD_02 is false; we must go to the factory and measure the new number of elements that can be on the conveyor.

TYPE	IDENTIFIER	DESCRIPTION	LOCATION	STATUS	REVIEW CONDITION
STRUCTURAL	SS_01	The previous element of the conveyor is the machine named “BADP_PADP”.	PCinta	true	If BADP_PADP is modified
STRUCTURAL	SS_02	The next element to the conveyor will be the robot “BRobot_PRobot”.	PCinta	true	If BRobot_PRobot is modified
STRUCTURAL	SS_03	When the conveyor is full, it sends a message to the previous element to stop this machine.	PCinta	true	never
STRUCTURAL	SS_04	When the conveyor has empty spaces, the previous element puts a new box in the conveyor and starts its movement to the position that lasts a time defined in SD_03.	PCinta	true	never
DATA	SD_01	The time needed to cross the conveyor is 6 s, defined by the TimeToRoll variable.	PCinta	valid	It TimeToRoll is modified
DATA	SD_02	The maximum number of elements on the conveyor is 10, defined by the MaxElems variable.	PCinta	false	If MaxElems is modified
DATA	SD_03	The time needed for the element to reach its position on the conveyor is represented by the expression $\frac{TimeToRoll}{\left(MaxElems-1\right)}$.	PCinta	valid	never

. The table contains the TYPE of the assumption, the IDENTIFIER of the assumption to simplify its use, the DESCRIPTION that describes the assumption, the LOCATION that details the element (or elements) of the conceptual model that receives the assumption, the STATUS of the assumption, that can be true, false or valid, and the REVIEW CONDITION that expresses when the assumption must be reviewed again to ensure that its STATUS did not change. Note that the conceptual model formally represents all the assumptions. Hence if all of the organization members are familiar with the language selected, this can be the communication channel to be used in the organization.

The textual representation of the assumption in the conceptual models owns ambiguity, but this document needs to be kept until not all the parts involved in the project feel confident with the formal language used to define the conceptual model that finally replaces this table, except for the simplification assumptions that may not be represented on the conceptual model.

Following the proposed approach, once we have the conceptual model and the assumptions from Table 3, we select a set of tests that covers all the assumptions we want to validate in all the Validation Processes we follow. This allows us to define a table such as

Table 6. Validations to be completed in the project.

PHASE	VALIDATION METHOD	SYSTEMIC STRUCTURAL	SYSTEMIC DATA	SIMPLIFICATIONS
CONCEPTUAL MODEL VALIDATION	Face validation	+		+
DATA VALIDATION	Goodness-of-fit		+
OPERATIONAL VALIDATION	Black Box	+	+	+
EXPERIMENTAL VALIDATION	Sensitivity analysis	+	+
VERIFICATION	Symbolic debugging	+
ACCREDITATION	Acceptance testing	+	+	+
SOLUTION VALIDATION	Field testing	+	+	+

.

The selection of the validations methods in each phase must be made to validate. Before the Accreditation and Solution Validation phases, all the assumptions must consider the resources (time, personnel) we have on the project. In each phase, we apply the test and analyze different assumptions from the table. If some assumptions remain not validated, we can add more tests to complete the validation. In short, we select this subset of validations intending to ensure that all the assumptions are tested by a validation method and that all the phases are also covered. Figure 1 helps to understand the flow we will follow in the iterative modeling process, while the use of the assumptions table and Table 3 help to select the more appropriate tests. Table 6 is the one we follow in this project. However, other validation approaches can be made to achieve the complete validation of the assumptions and the different phases.

7. Conclusions

The development of a model must be guided by the needed Validation, Verification, and Accreditation processes. This defines an iterative approach until the achievement of the Accepted Solutions, a product that will be implemented in the system. The approach presented in this paper encompasses all the needed actions, products, and validation processes that must be done to achieve a valid model. Moreover, considering the possibility that the model belongs to a Digital Twin, the validation of the model continues while it is in use; this iterative approach systematizes the Solution Validation, allowing understanding and detection of the assumptions that become false due to incorrect understanding or modifications of the analyzed system. Therefore, the prototyping approach, often used in Industry 4.0 to implement different testbeds, is supported by this approach. In a Digital Twin, composed of the Digital Master, which represents the models, and the Digital Shadows, representing the data, obtained from the model and the system, the continuous validation process done in the Solution Validation can cause the reevaluation of the different products obtained through the modeling process.

The definition of a model through its assumptions is the key element that rules its behavior, depicting the suggested causality rules. However, not all the assumptions used in a model have the same effect on its definition. The classification of the assumptions simplifies the model’s reevaluation and maintenance. Furthermore, the tests we can use to prove the validity of the model are not working with any typology of assumptions; in our taxonomy, the three classes of assumptions that exist are Systemic Structural assumptions, Systemic Data assumptions, and Simplification assumptions. The use of a taxonomy allows focusing our efforts on a selected subset of the validation tests; the W3H table is the first step to understanding and systematizing the validation test selection. We show that during the iterative process, it is not necessary to ensure that all the assumptions are valid. Sometimes it could be desirable to work with no validated Systemic Data assumptions to validate the Systemic Structural assumptions, or to obtain data related to extreme conditions situations to perform the validations. Working with the proposed approach clarifies the need to increase the Systemic assumptions to represent better and with more detail the structure and the data. On the other hand, we want to decrease the number of Simplification assumptions used to represent a deeper understanding of the system.

The definition of an assumptions table can also be a valuable tool for a company that wants to perform a digitalization of its processes, not only to drive the modeling process. We also note that we must take care of expiration conditions for the assumptions, like the data expiration for the Systemic Data assumptions, to ensure that a model is valid during the life span of the model and take care of any modification in the system that will affect the Systemic (Data and Structural) assumptions. Modifications in the technology, or improvements to the scientific framework used to develop the model, will eventually trigger some simplification assumptions to be discarded.

The life cycle of the modeling process we present depicts all the actions, processes, and products one can obtain in an iterative validation and verification process. This diagram understands the specific features of the new Industry 4.0 requirements and the continuous Validation Verification and Accreditation process that must be done in the context of a Digital Twin creation. The validation and Verification process are the key aspects that must drive any simulation as proposed on the iterative cycle. No model can be considered true but useful, and this depends strongly on how this validation process has been done. The review and the classification of the different Validation techniques one can use will serve precisely to simplify, orientate, and accelerate this Validation process.

This work proposes a complete life cycle guided through the Validation and Verification processes that simplifies the assumptions used and clarifies the products obtained in each phase. Following the proposed process guides simplifies the Validation Verification and Accreditation actions, increasing the confidence of both modelers and Stakeholders in the final solution.

Finally, we want to mention that, at the beginning of this work, we presented a juxtaposition between the model and the system. Notice that with this approach and understanding the possibility of the interaction between model and system through the Solution Validation, a new entity emerges, including both elements, the system and the Digital Twin.