1 Introduction

Manufacturers lack a concrete methodology to choose and prioritize emerging technologies that aid in the creation of smart manufacturing systems and factories. On top of this, manufacturers may need to implement organizational and process improvements to realize the full benefits from these technologies. A survey indicate that most manufacturers have trouble making such improvements [1]. While larger companies can bring in consultants to assist with such issues, small and medium size manufacturers typically do not have the funds to do the same.

Existing methods such as the Supply Chain Readiness Level [2] and MESA Manufacturing Transformation Strategy [3] exist, but they largely ignore the use of information and communication technologies (ICT) as a primary foundation for making those improvements. There are existing works, which study the impact of Information Technology (IT) adoption to businesses (also known as business & IT alignment). However, each study typically focuses on evaluating a single technology such as the Enterprise Resource Planning (ERP) system or Manufacturing Execution System (MES). These studies have not taken into account other aspects of the organization that can affect the impact of the respective technology adoption.

This paper describes our initial work to develop a method for assessing a factory’s readiness for incorporate emerging ICT technologies to become a smart factory. In our view, a smart factory uses ICT to maintain and improve its operational performance in response to its changing environment. Consequently, the method breaks down the assessment into four maturity components including the IT, the information connectivity, organization, performance management program maturities. Our method then combines these assessments into a single Smart Manufacturing System Readiness Level (SMSRL) index, which can be used for benchmarking individual factories or as criteria for selecting a supplier among several factories. In addition to describing our method, we discuss a correlation analysis that we performed. That analysis shows that the SMSRL index has a positive correlation with the operational performance.

Next we discuss related work in more detail before describing the SMSRL assessment method. Then, the result from the validation study is presented and a conclusion and remarks are given.

2 Related Work

Several types of readiness levels are used within the manufacturing sector. Technology Readiness Level (TRL) indicates the maturity of a technology for commercial adoption [4]. Similarly, Manufacturing Readiness Level (MRL) indicates the maturity of a manufacturing process technology [5]. An organization can use the same scale to indicate the maturity/capability of its respective technology as well. These methods do not evaluate a particular company for its readiness to adopt a particular technology.

Supply Chain Readiness Level (SCRL) [2] provides a method to assess the ability of the supply chain to operate and to achieve specific operational performance targets. The readiness levels are associated with characteristics within fifteen (15) categories that discretely provide an improvement roadmap for a supply chain design and the operation. Similar to the TRL and MRL, SCRL does not provide a methodology for a particular organization to assess its readiness to adopt a particular technology, which may correspond to some categories of the characteristics.

The MESA manufacturing transformation strategy (MTS) provides a framework based on the ISA-95 standard [3] to prepare an organization for Manufacturing-Operation Management (MOM) technologies adoption in four (4) business domains including Business Processes, Organization Structure, Personnel Skill Sets, and Manufacturing System Technology.

The SMSRL assessment objective is similar to that of the MESA-MTS albeit with the scope going beyond MOM technologies. Technically, the SMSRL index provides an indication of the current state with respect to a reference model. Both the SMSRL and the MESA-MTS allow the reference model to evolve as new technologies emerge and become available. Because of this, the assessment piece of the method, by design, is kept independent of the reference model.

3 Method

Figure 1 provides an overall architecture of the readiness level assessment. It summarizes the steps, the inputs, and the outputs involved in the assessment followed by improvement plan development. The process is iterated after the plan has been implemented. The primary purpose of the proposed assessment based on the SMSRL index is to (1) help manufacturers determine their current level and (2) develop a customized improvement plan.

Fig. 1.
figure 1

Overall assessment framework

Full size image

3.1 Profiling the Current State

The SMSRL proposed in this paper is based on the Factory Design and Improvement (FDI) reference-activity model defined in [6]. That model provides a set of reference activities; information entities for input, output, and constraints on each activity; and relevant software functions using the IDEF0 functional requirement modeling method [7]. On the basis of this model, we developed a questionnaire for profiling [8]. It is to be answered by relevant factory personnel. The questionnaire is organized into four measurement dimensions (C1 to C4) each of which consists of measurement items (process, designated personnel, etc.) as shown in Fig. 2. See the citations in Table 1. for the sources of these dimensions. Next we discuss each dimension in more detail.

Fig. 2.
figure 2

SMSRL measurements

Full size image
Table 1. Computation Methods for SMSRL
Full size table

The Organizational maturity dimension (C1) is conceptually defined as the comprehensiveness of the activities in the reference activity model performed by the manufacturers. It is measured by (1) whether there is a process that formally manages each activity; and (2) whether there is a responsible human resource assigned to the activities.

The IT maturity dimension (C2) is conceptually defined as the degree to which IT resources are available and working. The IT resources refer to computerized tools and methods. For example, a paper-based analysis method for layout design would not be qualified as an IT resource.

The Performance Management maturity dimension (C3) is conceptually defined as the degree to which the performance measures are used and monitored. This dimension also takes into account the connectivity between different operational performance measures where appropriate.

Lastly, the Information connectivity maturity dimension (C4) is conceptually defined as the maturity of the method to exchange the required information and the degree to which the information is shared/exchanged.

Profiling the current state consists of three operations: scope determination, information collection and consolidation. The scope is represented by relevant activities and stakeholders. The FDI model also indicates stakeholders relevant to each activity based on the ISA-88 manufacturing control architecture [9]. Information collection and consolidation are performed collaboratively among the group of stakeholders that are relevant to the scope.

3.2 Evaluate Current State

The evaluation of the current state compares the profile to the reference activity model. Computation methods, as shown in Table 1, are applied to each measurement dimension resulting in quantitative measures that can be used for comparison and benchmark.

The counting measure is the ratio between the number of elements employed in the current practice and those suggested in the reference activity model. For example, the number of software functions (per the reference model) available in the factory divided by the number of all the software functions identified in the reference model gives a C2 measure.

The Activity maturity scoring scheme is based on the Capability Maturity Model Integration (CMMI) [14]. The stakeholder of each activity scores the maturity of the activity based on the scale shown in Table 2.

Table 2. Activity maturing scoring scale
Full size table

The incidence matrix is commonly used to represent and analyze interactions between entities in a complex system. Here, an incidence matrix is used to represent the information entities connected between activities; and as such can be used to quantify the information connectivity maturity (C4). It is an n x n matrix where n is the number of activities under evaluation, the row is the activity that provides the information entity (from-activity or sender activity), and the column is the activity taking the information entity as an input (to-activity or receiver activity). The cell, the incidence, indicates the maturity of the information flow from the row to the column. Table 3 shows a schematic view of an incidence matrix. The maturity of the information flow is marked by the scoring scheme shown in Table 4 with the highest score (1) being connected by a standard data exchange. All reference incidence matrices assume the highest score where there is the information connectivity between the from- and to-activity. A score of 0.7 represents that software capability for data exchange among activities exists, but information is not currently exchanged. When the data is exchanged manually between activities, the score is 0.3. A score of zero indicates that there is no data exchange between the activities.

Table 3. Activity incidence matrix
Full size table
Table 4. Incidence scoring scheme
Full size table

The evaluation result can be visualized as shown in Fig. 3. Each indicator can be used individually or combined into a single SMSRL index. For simplicity, a single SMSRL index was computed using an average of C1, C2, C3 and C4. The overall index and/or individual construct can be used to prioritize the factory improvements or to evaluate potential suppliers.

Fig. 3.
figure 3

An exemplary assessment result

Full size image

3.3 Develop Improvement Plan

In the last step, the evaluation result is used to develop and prioritize an improvement plan. A classification analysis shown in the next section provides a high-level improvement recommendation. Our future work lies in developing a method to provide a more detailed recommendation.

4 Validation Study

This section investigates the validity of the proposed assessment using a similar approach to [13]. First, data about the relationships between the SMSRL and operational performance was collected. Then, hypothesis tests for the statistical significance of the relationships were performed. Lastly, we analyzed patterns of the SMSRL that can guide an improvement plan development. These activities are explained below.

Data Used for the Validation.

Existing studies in the domain of business and IT alignment were used for the validation. A detailed analysis on the existing studies can be found in [8]. Different alignment constructs (i.e., measurement items) from these studies were mapped to performance categories (e.g., operational, financial) and were statistically correlated using empirical data.

Validation Method.

To establish the relationship between the SMSRL assessment and the performance categories, the measurement items of the SMSRL assessment are mapped to those considered in the studies (operational, financial, value-based, and overall). A similarity value between the SMSRL assessment and the target study is then calculated using the n-gram measure (intersection divided by union). This gives the basis for the correlation analysis shown in the next subsection.

Hypothesis Test.

Four hypothesis tests were performed. Statistically significant, positive-correlations with the SMSRL index were found on the operational performance, overall performance, and value-based performance as shown in Table 5. The financial performance was not found (hence not shown) to have a statistically significant positive-correlation.

Table 5. Hypothesis test results
Full size table

High-level Recommendation.

A k-means-clustering analysis on the simulated SMSRL results has been performed (k = 3). Based on its result shown in Table 5, a high-level recommendation can be made for each SMSRL cluster. The cells with bold-font values show the category of improvement a factory should focus on to have the largest impact on a respective performance category. For example, the first row indicates that improvements in the information connectivity (C4) is likely to have the best impact on the operational performance (Table 6).

Table 6. High-level recommendation
Full size table

5 Conclusion and Remark

We introduced a new, smart manufacturing system readiness assessment (SMSRL). SMSRL measures the readiness using maturity scoring of four dimensions: Organizational, IT, Performance management, and Information connectivity maturities. The core of the smart manufacturing concept is the ability to use information effectively. The SMSRL assessment provides a quantitative measure of this ability. Such measure, which is in the form of an index, can be used for benchmarking. The statistical analysis shows that the index has a positive correlation with three types of performance: operational, overall, and value-based.

The SMSRL index provides a real number as its readiness measure. The SCRL, on the other hand, provides discrete readiness levels. Each type of measure has its advantages. Discrete measures lend themselves readily to definitional levels. Real-numbered levels do not; however, they can be used in other quantitative analysis – such as the ones shown in Sect. 4. Discrete measures cannot

In our future work, we will (1) develop a method to provide more detailed improvement recommendations (2) extending and/or experimenting with other models used as a reference for the assessment.