Fuzzy Soft Skills Assessment through Active Learning Sessions

Abstract

This research aims to present a Fuzzy Expert System with psychologist expertise that seeks to assist professors, researchers and educational institutions in assessing the level of incorporation of students’ Soft Skills while attending Active Learning sessions. The difficulties encountered by higher education institutions, researchers and professors in evaluating subjective and behavioral components, such as Soft Skills, was one of the problems that motivated the undertaking of this research. The theoretical framework on which this work is based includes the development and evaluation of Soft Skills in students, some concepts and characteristics about Active Learning and the main attributes and properties of Fuzzy Logic. This research is of an exploratory applied nature, a qualitative and quantitative approach is proposed, in which the methodological triangulation between the bibliographic analysis, the case study and the modeling and implementation of the expert system called Fuzzy Soft Skills Assessment was used to achieve the objective proposed.

Introduction

Developing skills and Competencies needed by students lies far beyond the mastery of theories and practices of science process skills (Hard Skills), and the job market has increasingly sought professionals possessing management, communication, innovation and social skills (Soft Skills) (Mulcahy-Dunn et al., 2018; Levant et al., 2016; Kyllonen, 2013). Experts argue that higher education should not only focus on professional training composed of Hard Skills complemented with Soft Skills in order to overcome this challenge, and thus teaching strategies related to Active Learning (Levant et al., 2016; Conchado et al., 2015; Devadason et al., 2010) have often been adopted.

Although the challenge of developing Soft Skills is being faced by higher education institutions (Chan et al., 2017; Conchado et al., 2015; Achcaoucaou et al., 2014; Marques, 2013; Devadason et al., 2010; Chamorro-Premuzic et al., 2010; Andrews and Higson, 2008), how to assess the incorporation of students’ Soft Skills is yet to be found. Written examinations have been mainly used for Hard Skills assessment. However, given the fact that Soft Skills assessment comprise behavioral and subjective skills, such a tool is not recommended for such a purpose (Heckman and Kautz, 2012).

Researchers have suggested that a combination of self-assessment, peer-assessment and examination by teachers is the best tool to assess Soft Skills incorporation (Verbic et al., 2017; Kyllonen, 2013; Allen and Van Der Velden, 2005; Shuman et al., 2005)¹. Nonetheless, current instruments present the following drawbacks: difficulty to provide constant feedback to students (Verbic et al., 2017) as the assessment is carried out only at the beginning and/or end of the course (Deep et al., 2019; Levant et al., 2016; Conchado et al., 2015; Rodríguez et al., 2015); templates for forms to be used in longitudinal studies involving many variables ( Chan et al., 2017; Bohlouli et al., 2017; Levant et al., 2016), which are not feasible to be used at the end of all classes and; sole use of self-assessment (Chan et al., 2017; Levant et al., 2016; Conchado et al., 2015; Achcaoucaou et al., 2014). In addition, teachers and students lack experience in this type of assessment. The effort of this work is to minimize these gaps.

An alternative to overcome parts of these drawbacks is the use of an Expert System, based on the Fuzzy Logic that simulates the expertise of a human being at making decisions regarding Soft Skills assessment. Fuzzy Logic has been referred to as an inference mechanism to make decisions that involve a degree of uncertainty and imprecision (Saido et al., 2018; Gomathi and Rajamani, 2017; Ozdemir and Tekin, 2016; Dias and Diniz, 2013). A Expert System with Fuzzy Logic can be applied to improve and classify student behaviors (Lin et al., 2016). In the area of education, we have the following examples: Assessment of students’ difficulties to suggest strategies for learning (Gomathi and Rajamani, 2017; Lin et al., 2016; Novak and Oreski, 2016), assessment of cognitive skills (Saido et al., 2018; Chrysafiadi and Virvou, 2015), assessment of learning styles (Aissaoui et al., 2019; Alian and Shaout, 2017; Ozdemir et al., 2016), assessment of the Soft Skills communication (Ozdemir and Tekin, 2016) and creativity (Susnea and Vasiliu, 2016).

Why choose Fuzzy Logic? To exemplify the choice, suppose two classes using Problem Based Learning, in which a group of students denoted by A, B, C, D, and E is working collaboratively. In the first class, the teacher proposes a problem X that has different characteristics from problem Y proposed in the second class. In the first class, student A can present skills such as Initiative, Time Management, and Flexibility at a high level, in the second class, which has different characteristics from the first, student A can present these same skills at a low level or not present. Thus, the assessment of A’s skills carried out by peers will have different results in each of the classes, characterizing the uncertainty and imprecision of this assessment process. In addition, beliefs between the assessor and the assessed can change the understanding of skills, such as good communication or teamwork, which vary according to situations and contexts, further characterizing the Soft Skills assessment with fuzziness (Gibb, 2014). The membership sets implemented by the Fuzzy Logic have an important role in this evaluation process, as they guarantee decision-making in phenomena characterized by uncertainty and imprecision. The possibility of describing crisp variables in linguistic variables was also one of the attributes that motivated us to choose Fuzzy Logic.

Therefore, this article aims to present an expert system for Fuzzy Soft Skills Assessment - FSSA as an innovative tool to assess the Soft Skills acquired by students during Active Learning sessions. To overcome some of the aforementioned drawbacks, the FSSA system consists in students’ self-assessment and peer-assessment; it is simple and easy to be filled out and it can be applied at the end of all Active Learning sessions; its knowledge base was acquired through a behavioral analysis expert; and it ensures constant feedback through an individual profile report about Soft Skills.

In addition to assisting teachers who are keen evaluators of skills and competencies acquired by their students, it can assist researchers in the field of Active Learning to validate their conjectures about the fact that active learning methods are more prone to develop skills and Competencies (Deep et al., 2019; Brilingaite et al., 2018; Lutsenko, 2018; Vogler et al., 2018; Yehia and Gunn, 2018; Baytiyeh, 2017;Verbic et al., 2017; Bautista, 2016; Conchado et al., 2015; Rodríguez et al., 2015). Furthermore, assessing skills and competencies is rather important (Verbic et al., 2017; Bautista, 2016; García et al., 2016), as it can assist universities in developing their curricula (Conchado et al., 2015; Devadason et al., 2010) and improving the teaching/learning process (Achcaoucaou et al., 2014). However, there are few instruments that have actually been validated to assess the incorporation of skills and competencies (Conchado et al., 2015; Shuman et al., 2005).

The second section presents a literature review highlighting the theoretical framework and works related to this study. In the third section, a brief description of the Fuzzy inference process is presented. The FSSA modeling process is shown in the fourth section, and its results are discussed in the fifth one. Finally, its conclusions are presented in the sixth section.

Theoretical Framework and Related Works

In this section, some background knowledge about Skills and Competencies are presented, followed by Active Learning approaches and finally a few works related to Soft Skills.

Skills and Competencies

For professional success, a student must possess more than just technical abilities (Hard Skills). Effective communication, persuasion, leadership, teamwork, interpersonal relationships, among other Soft Skills (Deep et al., 2019; Vogler et al., 2018; Baytiyeh, 2017; Chan et al., 2017; Levant et al., 2016; Sánchez et al., 2016; Conchado et al., 2015) have been pointed out by employers as essential for a thorough training of a student, moreover, Active Learning has been pointed out as ideal teaching strategy in this context (Shuman et al., 2005).

The term “competencies” can be understood as a set of traits, skills, experiences, values and attitudes, that an individual must have in order to successfully perform an activity in a given area (García et al., 2016; Achcaoucaou et al., 2014). Hard Skills can be understood as technical and cognitive skills (solving mathematical equations, designing a floor plan, etc.), while Soft Skills are those developed through more intangible Competencies, such as motivation, empathy, interpersonal relationships, negotiation, teamwork, ethics, etc.

Researchers, professors, educational institutions and employers are in agreement with the importance of Hard Skills in the education of students, but their overlap with Soft Skills is not recommended, since they have been the topic of discussion in the academic community (Mulcahy-Dunn et al., 2018; Levant et al., 2016) which considers Soft Skills as strategic elements in any organization. Thus, their development has been encouraged in university curricula and, although numerous initiatives have been adopted internationally in order to ensure the teaching of Hard Skills complemented by Soft Skills (Chan et al., 2017; Conchado et al., 2015; Achcaoucaou et al., 2014; Marques, 2013; Devadason et al., 2010; Chamorro-Premuzic et al., 2010; Andrews and Higson, 2008), the available instruments to evaluate Soft Skills are still incipient (Gibb, 2014).

Even in the face of such difficulty, identifying students’ skill levels can assist in improving curricula (Devadason et al., 2010), as well as the teaching/learning process (Achcaoucaou et al., 2014), and it ensures that organizations should have the knowledge management necessary for a successful performance (Conchado et al., 2015).

Conchado et al. (2015) highlight the need to develop and validate instruments for measuring skills and Mulcahy-Dunn et al. (2018) explain that relatively little is known about Soft Skills assessment.

Active Learning

In order to ensure efficient learning, students should be encouraged to understand the basic concepts, aspects of a given theory and learn how to use them to solve problems (Deep et al., 2019; Tawfik et al., 2016). Some challenges yet to be faced are learning how to interact with others, expressing and accepting criticism, listening and understanding distinct points of view, which are also some of the requirements for future professionals. In addition, a major challenge posed to universities is not to provide students with in-depth knowledge, but to help them acquire it through their own experiences (Foldnes, 2016). In this context, Active Learning emerges as a methodology based on student-centered education and skills development. It is generally defined as any type of instructional approach that involves students in their own learning, unlike traditional teaching methods in which students end up being passive spectators of information and the teacher is the central and sole holder of knowledge (Roberts, 2019; de Novais et al., 2017). Among Active Learning activities, reading, writing, group work, peer discussion, interdisciplinary projects, seminars, discussions, case studies, simulations, among others, stand out.

During the FSSA implementation experiment, 19 students divided into groups of 4 or 5 students who participated in Active Learning sessions. The main approaches used in these sessions were Problem Based Learning, Project Based Learning and Flipped Classroom. In Problem Based Learning, the problem is considered as a starting point of the learning process (Lutsenko, 2018; Tawfik et al., 2016), i.e. a new theory or concept must be initially supported by problem solving, preferably involving real-world contextualization. Project Based Learning involves students in a more extreme problem-solving instance in which countless interdisciplinary concepts must be used, thus they must develop a tangible project so as to ascertain their development and specialization (Vogler et al., 2018; Rodríguez et al., 2015). Flipped Classroom is an interactive teaching method that flips the traditional classroom, therefore transferring the introduction of concepts and features to the home and the assimilation of information to the classroom (Burke and Fedorek, 2017; Bergmann and Sams  2016; Foldnes, 2016; Seery, 2015).

Conchado et al. (2015) argue that traditional teacher-oriented methods were less prone to develop skills if compared to student-oriented active learning methods. Deep et al. (2019) point out that numerous researches show that teaching through oral and written expression does not promote a deep understanding of concepts and fundamentals, and that a solution would therefore be Problem Based Learning - based courses. Yehia and Gunn (2018) emphasize that the introduction of Active Learning encourages student involvement, therefore relinquishing them from the impartation of information and urging them to develop social skills. Brilingaite et al. (2018) state that, in addition to increasing generic skills (Soft Skills), Problem Based Learning is suitable for the development of specific skills (Hard Skills).

As explained earlier, Soft Skills are acquired in practice through human relationships, socialization and interaction between individuals involving constant feedback. Therefore, no significant changes must be made to the curricula to meet the expectations of companies. Soft Skills can be developed while teaching Hard Skills using Active Learning as methodology. Vogler et al. (2018) highlight that Project Based Learning can be used to develop technical skills together with Soft Skills, while Baytiyeh (2017) argues that Flipped Classroom improves reasoning skills, teamwork, problem solving and student-teacher and student-student interactions. Deep et al. (2019) state that a large amount of studies indicates an improvement in the incorporation of various Soft Skills by adopting Problem Based Learning practices.

Soft Skills Assessment and Related Works

Regarding the state-of-the-art in methods to assess the Soft Skills of individuals, Table 1 presents nine references that have used some physical or virtual means to reach this objective.

Table 1 Related Works

Most authors assess students’ Soft Skills in longitudinal studies and in a timely manner, i.e. at the beginning and/or end of a course. This procedure makes a formative assessment of students unfeasible which is generally characterized by continuous feedback and improvement in learning experiences (Chai et al., 2015; Lan et al., 2011). To ensure a formative assessment of Soft Skills, students must be evaluated on a regular basis, who consequently will receive continuous feedback on their competencies.

Seven of the nine surveys presented prioritize the use of self-assessment only. The main advantages that justify this attitude are: the instrument is relatively easy to be employed in large samples; it can be used simultaneously in different locations; and it produces responses that can be easily measured (Allen and Van Der Velden, 2005). The main disadvantage revolves around a greater chance of measurement errors (Allen and Van Der Velden, 2005). Some studies claim that peer assessment is more accurate than self-assessment, which may be due to the fact that individuals are more prone to identify other people’s behaviors than their own behaviors (Allen and Van Der Velden, 2005). 360-degree assessment has been suggested as the best instrument for evaluating Soft Skills (Verbic et al., 2017; Shuman et al., 2005), however, on account of its potential complexity, its application in the classroom may be unfeasible.

One of the authors opted to use 360-degree assessment (Bohlouli et al., 2017) through an Expert System, but it was used in a corporate system with the aim of assessing employee skills. In addition, it involves the use of several questionnaires, one for each of the Competencies under assessment. Although it is not an objective of authors, to use this Expert System in order to assess the Soft Skills of students in the classroom, but it becomes impracticable due to the degree of complexity of the process.

Four authors (Deep et al., 2019; Verbic et al., 2017; García et al., 2016; Levant et al., 2016) attempt to analyze the cause and effect relationships between Active Learning and Soft Skills. A particular case presented by García et al. (2016) highlight a system with the purpose of promoting learning and assessing generic skills in an e-learning or b-learning scenario. Despite being applied in a virtual learning environment, the built-in self-assessment questionnaire template to assess teamwork is no different from others.

Chandrasekhar and Khare (2021) used a fuzzy soft set-based hybrid optimization algorithm to reduce intelligent tutoring system parameters. The authors’ aim is to analyze students’ bio-psychological potential by identifying cognitive and personality traits using psychometric inventories. Data collected from the self-assessment inventories refer to thinking styles, multiple intelligences, and the big five personality attributes. To support human resources managers in the evaluation of Soft Skills profiles of PhD candidates, Azzini et al. (2019) propose an evolving fuzzy model that evaluates the Soft Skills which used questionnaires completed by the participants and interviews carried out by a human resources operator. Here, the candidate’s Soft Skills profile focuses on self-assessment and the assessment of a Human Resources Operator, dispensing with peer assessment.

Fuzzy Inference and Related Works

Fuzzy Logic, proposed by Zadeh (1965), is generally selected to make inferences about problem-solving situations that involve a degree of uncertainty and imprecision (Gomathi and Rajamani, 2017; Ozdemir and Tekin, 2016; Dias and Diniz, 2013). This logic is used in computers with the purpose of making decisions based on inaccurate quantities, as the human brain similarly does (Saido et al., 2018). In addition, the use of linguistic variables is one of the attributes of Fuzzy Logic (Saido et al., 2018; Dias and Diniz, 2013) which facilitates the understanding of results by users who are less prone to interpret numerical variables. In a very superficial manner, it is worth mentioning that, according to Fuzzy Logic, the expert’s knowledge is represented as “if A, then B”, which is transformed into analytics data using the Fuzzy set theory, thus allowing an ambiguity in the natural language. In this case, a pair of numbers 0, 1 based on the Boolean Logic (where zero means that it does not belong and 1 means that it belongs) is generalized to numbers in the interval [0, 1]. A membership function maps all elements of the universe of discourse to numbers in that interval, otherwise known as called Fuzzy numbers (where zero means it does not belong, 1 means it fully belongs and any other number in this interval indicates partial membership) (Capuano et al., 2017; Dias and Diniz, 2013).

Artificial Intelligence in Education with Fuzzy Inference

The area of education involves many human and social decisions, which in some cases have a high degree of uncertainty and fuzziness. In addition, with the advancement of Virtual learning environments, Intelligent tutoring systems, and Game-based learning, decision-making in uncertain scenarios becomes increasingly necessary. Thus, some knowledge engineers have chosen to use Fuzzy Logic in the development of their AIED. Table 2 highlights some works in this area.

Table 2 AIED with Fuzzy inference

Among the references presented in Table 2, Chrysafiadi et al. (2022) propose an educational game that offers adaptability to teaching dynamically and individually. The adaptation uses fuzzy inference that detects the player’s current cognitive state and adapts the game scenario. Still, in the Game-based learning group, the work of Troussas et al. (2020) stand out here, who presents Quiz Time, an intelligent learning application based on mobile games to assess and advance students’ knowledge of the C# programming language. The game has a Dynamic Advice Generator (DAG) based on fuzzy logic. A similarity in the authors’ proposal is the use of statically defined membership functions, that is, the intervals of the domains of the trapezoidal functions are fixed real constants.

Alian and Shaout (2017) propose a Fuzzy Inference model to predict the student’s learning style in an e-learning environment based on the VARK questionnaire domains (Visual, Aural, Read/Write, Kinesthetic). To find the best learning style for teachers, Ozdemir et al. (2016) present a Fuzzy inference system based on the learning styles of Honey &Mumford and McCarthy. Gomathi and Rajamani (2017) propose to investigate the individual skills of students in an e-learning platform that, through a system with Fuzzy inference, suggest courses and curriculum adapted for the student. Here some Hard Skills and Soft Skills are part of the system’s input variables. Novak and Oreski (2016) propose a knowledge-based system with a fuzzy inference that assesses students’ main difficulties and produces appropriate learning strategies. The system’s knowledge base was obtained from interviews with graduating students. As in the previous cases, these four authors use static membership functions, and this characteristic is what differentiates our work. In our construction strategy, the functions are not static (the ranges of the domains of the trapezoidal functions are not fixed constants), they are flexible and adjust to the evaluated group. Section 3.3 describes the construction of the FSSA membership functions, where the ranges of the domains of the functions depend on the z parameter. The z parameter is the normalization of the skill evaluated when its score is null (see section 3.3).

Ozdemir and Tekin (2016) highlighted a fuzzy inference system to evaluate the oral presentations of teachers in training. The authors compare the model results with the Presentation Evaluation Scale (PES). To measure the quality of the interaction of b-Learning users of Learning Management Systems, Dias and Diniz (2013) use Fuzzy logic (FuzzyQoI) constructs. In these two cases, the authors use a structure of cascaded fuzzy systems². Similar to the structure used in our research. With this type of structure, in addition to reducing the number of parameters, it avoids a combinatorial explosion of fuzzy rules caused by too many input variables (see section 3.1 figure 3).

Lin et al. (2016) present a fuzzy expert system to determine the number of materials for Remedial Learning. Here, the authors used flexible membership functions, that is, the construction parameters depend on the mean m and standard deviation sd of the tests performed, a strategy analogous to that used in the current research. However, our parameter is the value of z.

Development and Structure of FSSA

In this section, the FSSA knowledge acquisition and representation process is going to be discussed, as well as the process of representation and implementation of the proposed model.

Knowledge Acquisition for FSSA

Before acquiring knowledge and modeling for the FSSA, it is necessary to determine the main skills and Competencies that:

• the job market requires for new professionals and;

• higher education institutions intend to teach their students.

For such a purpose, in an initial analysis, 2305 publications were collected from journals between 2009 and 2019 whose topics contained terms related to ways of to evaluate and to develop of the Soft Skills (Soft Skills analogous terms was used here³ and words⁴). The analysis was made in the Web of Science categories linked to education, engineering and computer. All keywords in these publications were grouped into VOSviewer clusters (Type of analyzes: Co-occurrence; Counting method: Full counting; Unit of analyzes: All keywords; Using thesaurus file; and Minimum number as occurrences of a keywords: 10) so that the relationship between items was determined by the number of documents in which they occur together (figure 1).

Fig. 1

Bibliometric mapping of main keywords related to Soft Skills

The strength of items, which is indicated by the diameter of circles, highlights the keywords Soft Skills (ss), competencies, higher education (he), employability, communication, assessment, design, performance and active learning (al), which eventually emphasizes the importance of the Soft Skills assessment in a relation between Higher Education and employability, in addition to associating it to Active Learning. Other skills highlights too in this figure: leadership, team-work, technical, teaching, attitudes, feedback, motivation, collaboration, self-regulation, mind, interpersonal, innovation/creativity, ethics and critical thinking. After previous analysis, this same parameters was used in the title in the papers, finding 236 publications. A initial read of the titles and abstracts was realized, restricting the 28 publications that contained some classification or categorization of the Soft Skills. A content analysis was performed to identify the main categorizations between skills and competencies. Of this stage, 26 skills was selected to be used on an observation form and as FSSA’s system inputs.

Through the systematic review, it was noticed that there were no detailed studies on the development and assessment of Soft Skills in Active Learning environments. That way was realized a exploratory study through a case study. The case study involved observation of 19 students (observees) who participated in Active Learning sessions for 7 weeks in 2019. The question to be answered in this case study was: “How do observers, teachers and students perceive the incorporation of Soft Skills during Active Learning classes?”

For developing the FSSA knowledge base, monthly semi-structured interviews with a psychologist (Expert in skill assessment⁵) were carried out in the period of 1 year (2019-2020). Audio recordings and description of reports were collected. The objectives of these interviews were:

• to understand how skills are interconnected to form competencies;

• to interpret the reports of observers, and verify the cause and effect relationships between actions and the behavior of those under observees;

• to validate the results found by the FSSA.

The first author of this paper (acting as knowledge engineer) and the expert trained four students in the last period of the undergraduate psychology course (observers) to participate in the case study stage. In this training, we studied: observation techniques, research objectives, selective attention, reactivity, attention deficit, construct validity, selective data entry, selective memory, decisions about how to record, consistency, judgments, skills to be analyzed, and filling of the semi-structured observation⁶. Each observer $O_i$ to observed the teamwork of the members of the group $E_i$ (with $1\le i \le 4$) in each of the Active Learning sessions, thus recording the perceptions about the Soft Skills of each student. Each observer was responsible for collecting data on the skills of five students (observees) in the seven sessions, except observer $O_2$ who was responsible for four students (observees). Figure 2 shows one of the moments of observation, with emphasis on two of the observers analyzing the work of two teams of observees. Biweekly, the knowledge engineer and the expert interviewed the observers to analyze the data and align the observation criteria.

Fig. 2

Arrangement of observers and observees

In the last Active Learning session, the class teacher answered a questionnaire about his perceptions regarding the incorporation of Soft Skills by each student.

Students were assigned to assess their own skills and those of their peers at the end of each Active Learning session by using forms comprising a list of 26 skills in which they should check those that stand out and those that need improvement.

The knowledge engineer and expert assembled with the observers during the observation period so as to analyze the data collected, generate group reports and individual reports. After analyzing the data, some cases of reactivity were noticed (changes in the behavior of the observees due to the observation). But this information was adjusted by triangulating the evidence collected in observation; interview with the class teacher; self-evaluation; and peer review.

Through a theoretical/conceptual analysis and interviews with the Expert, the structural diagram shown in figure 3 was built. The left side highlights the 26 skills evaluated which comprise crisp input variables in each FSSA’s Fuzzy subsystem (S01, S02, ..., S17). Rectangles are the competencies assessed by each of the FSSA subset.

Fig. 3

FSSA structural diagram

Among the advantages of this type of structure is the improved computational performance of the system and an executable number of rules to be verified and validated. Without this clustering, there would be a combinatorial explosion in the number of rules for the 26 skills. Similar structures to this can be found in Alian and Shaout (2017); Ozdemir and Tekin (2016); Veltri et al. (2014); and Dias and Diniz (2013). Although Azzini et al. (2019) reveal the lack of a widely accepted Soft Skills taxonomy, Bohlouli et al. (2017) present a similar structure to figure 3 to manage competencies in a human resources system, based on a competency model called the Professional, Innovative and Social competency tree. In this tree, the authors highlight the assessment of managerial and communication competencies (such as flexibility, commitment, proactivity, autonomy, and oral/written communication), social competencies (such as teamwork, interpersonal relationships, and motivation to learn), among other skills. Andrews-Todd and Forsyth (2020) propose a hierarchical structure to evaluate Collaborative Problem Solving. In this ontology, the authors describe the analysis of social, negotiation, and cognitive skills through online team tasks. Other authors who used some type of classification to assess the Soft Skills, and served as a basis for the construction of the structure proposed in figure 3 were Deep et al. (2019); Vogler et al. (2018); Chan et al. (2017); Levant et al. (2016), and Conchado et al. (2015).

Standardized Parameterized Skill

In the FSSA process, students must assess themselves and their peers using at least three outstanding skills and three skills that require improvement at the end of each Active Learning session. Thus, for each of the 26 skills presented, there was a counter-skill (for example, a student can be assessed as being Organized or Disorganized).

During at least p sessions in which n students assess themselves and their peers, each element $a_{ij}$ of the matrix (Eq. 1)

$$\begin{aligned} H_h^p=(a_{ij})_{nxn}= \left( \begin{array}{cccc} a_{11} &{} a_{12} &{} \cdots &{} a_{1n} \\ a_{12} &{} a_{22} &{} \cdots &{} a_{2n} \\ \vdots &{} \vdots &{} \ddots &{} a_{1n} \\ a_{n1} &{} a_{n2} &{} \cdots &{} a_{nn} \end{array} \right) \end{aligned}$$

(1)

is the absolute frequency at which skill h was assigned by student i to describe student j in p Active Learning sessions. For example, when $h=04$ (Motivation, see figure 3) and $p=6$, the value of element $a_{32}=5$ which indicates that assessing student 3 described student 2 as Motivated in 5 out of 6 Active Learning sessions.

Analogously to the definition given by Eq. 1, the Eq. 2 is defined as the matrix

$$\begin{aligned} \bar{H}_h^p=(b_{ij})_{nxn}= \left( \begin{array}{cccc} b_{11} &{} b_{12} &{} \cdots &{} b_{1n} \\ b_{12} &{} b_{22} &{} \cdots &{} b_{2n} \\ \vdots &{} \vdots &{} \ddots &{} b_{1n} \\ b_{n1} &{} b_{n2} &{} \cdots &{} b_{nn} \end{array} \right) \end{aligned}$$

(2)

where $\bar{H}$ indicates the counter-skill associated with skill h. In the example where $h=04$ (Motivation), Eq. 2 describes absolute frequencies related to the counter-skill “Lack of Motivation”.

The values of $a_{ij}$ in which $i=j$ represent the results of students’ self-assessment, while $a_{ij}$ in which $i\ne j$ are the results of the peer assessment.

Taking $j=u$, the function $SH_h^p:\mathbb {N} \rightarrow \mathbb {R}$ (Eq.3) associated to Eq. 1 is given by

$$\begin{aligned} SH_h^p(u)=q\cdot \sum _{i=1}^{n} a_{iu} \end{aligned}$$

(3)

indicates the absolute frequency accumulated in p sessions in which skill h was checked by evaluators 1, 2, ..., n to describe assessed student u. Weight $q \in \{a,b,c\} \subset [1,2]$ with $c \le b \le a$ increases the value of this summation under the following conditions:

• 1. $q=a, \; \text {se} \; a_{uu}>0 \; \text {e} \; \sum _{i=1}^{n} a_{iu}>0 \; \forall \; i \ne u$;

• 2. $q=b, \; \text {se} \; a_{uu}=0 \; \text {e} \; \sum _{i=1}^{n} a_{iu}>0 \; \forall \; i \ne u$;

• 3. $q=c, \; \text {se} \; a_{uu}>0 \; \text {e} \; \sum _{i=1}^{n} a_{iu}=0 \; \forall i \ne u$.

In condition i, the summation of Eq. 3 is enhanced, once skill h was perceived in student u, both during the self-assessment and the peer assessment. In condition ii, it less pronounced than in i, since skill h was identified in student u only by their peers. In condition iii, it would be lesser than in previous cases or null, since only student u identified that he possesses skill h through self assessment.

Analogously to the definition of Eq. 3, the function $S\bar{H}_h^p:\mathbb {N} \rightarrow \mathbb {R}$ (Eq. 4), was defined, but it was only to assess the counter-skill associated with h. Conditions r for Eq. 4 are the same as conditions q for Eq. 3.

$$\begin{aligned} S\bar{H}_h^p(u)=r\cdot \sum _{i=1}^{n} b_{iu} \end{aligned}$$

(4)

The Parameterized Skill h of student u in p Active Learning sessions is the term used to define function $PS_h^p:\mathbb {R} \rightarrow \mathbb {R}$ by the difference between Eq. 3 and Eq. 4, as shown by Eq. 5.

$$\begin{aligned} PS_h^p(u)=SH_h^p-S\bar{H}_h^p=q\cdot \sum _{i=1}^{n} a_{iu}-r\cdot \sum _{i=1}^{n} b_{iu} \end{aligned}$$

(5)

The value of Eq. 5 indicates how often the assessing students identify student u as possessing skill h, which enabled to infer the following cases:

• If $PS_h^p(u)>0$ then student u presented the assessed skill h during p Active Learning sessions;

• If $PS_h^p(u)=0$ then student u has not presented the assessed skill h during p Active Learning sessions;

• If $PS_h^p(u)<0$ then student u has not presented the assessed skill h during p Active Learning sessions, in addition to presenting characteristics of a counter-skill associated with h.

In view of the uncertainty of the total range of the set (Eq. 6)

$$\begin{aligned} A_h=\underset{u=1}{\overset{n}{\cup }} \{PS_h^p(u)\} \end{aligned}$$

(6)

that describes Parameterized Skills of all n students assessed as possessing skill h during p Active Learning sessions, it was necessary to standardize $PS_h^p(u)$ in order to establish the universe of discourse of 26 variables (skills) assessed through the FSSA.

Thus, the Standardized Parameterized Skill of student u, concerning skill h during p Active Learning sessions is defined according to Eq. 7 as:

$$\begin{aligned} SPS_h^p(u)=\dfrac{PS_h^p(u)-\mu _h}{\sigma _h} \end{aligned}$$

(7)

where $\mu _h$ and $\sigma _h$ are the mean and standard deviation of $A_h$.

In normal distribution, approximately 99.99% of the data set presents values ranging between $\mu \pm 4\sigma$ with the standardization of the 26 variables selected in this study, it was possible to establish a universe of discourse for all crisp input variables in the interval [-5, 5], i.e. between $\mu \pm 5\sigma$. With the standardization of $PS_h^p(u)$, it is unlikely that a given student u will present $|(SPS_h^p(u))|>5$. This process enabled the establishment of a unique universe of discourse for all 26 crisp input variables of FSSA subsets S01, S02, ..., S10.

FSSA Membership Functions and its Rule Base

For the 26 crisp input variables of fuzzy subsets S01, S02, ..., S10 of the FSSA, five membership functions were defined and denoted by $\mathcal {G}_{-2}$, $\mathcal {G}_{-1}$, $\mathcal {G}_{0}$, $\mathcal {G}_{+1}$ e $\mathcal {G}_{+2}$. Table 3 presents a description of their interpretation, whose defining parameter is $SPS_h^p(u)$ when $PS_h^p(u)=0$, denoted by z:

• If $SPS_h^p(u)\le z$ then assessed skill h of student u will be $\mathcal {G}_{-2}$ or $\mathcal {G}_{-1}$ or $\mathcal {G}_{0}$;

• If $SPS_h^p(u)\ge z$ then assessed skill h of student u will be $\mathcal {G}_{0}$ or $\mathcal {G}_{+1}$ or $\mathcal {G}_{+2}$.

Table 3 Notation of membership sets

Based on parameter z, the membership functions of Fuzzy subsets S1, S2,..., S10 of the FSSA are defined according to Eq. 8:

$$\begin{aligned} \begin{array}{r} \mathcal {G}_{k}: \; [-5, 5] \rightarrow [0, 1] \\ x \;\; \mapsto \mathcal {G}_{k}(x) \\ \end{array} \end{aligned}$$

(8)

where $k \in \{ -2, -1, 0, +1, +2\}$, x is the Standardized Parameterized Skill of a given student u concerning skill h during p Active Learning sessions and $\mathcal {G}_{k}(x)$ is the degree of membership of x in set $\mathcal {G}_{k}$.

Figure 4 shows the membership functions. It is worth mentioning that the domains of each function depend on the value of z and $\sigma$ with a strict membership between these functions and the standardized normal distribution.

Fig. 4

Membership Functions

In the case of output variables of all FSSA subsets, the same notation $\mathcal {G}_{-2}$, $\mathcal {G}_{-1}$, $\mathcal {G}_{0}$, $\mathcal {G}_{+1}$ and $\mathcal {G}_{+2}$ was used. However, as regards these variables, the universe of discourse comprised values of $y \in [-1.3,1.3]$, so that the Mamdani inference engine would ensure a centroid-like defuzzyfication in the interval $[-1, 1]$.

This type of output seeks to facilitate the interpretation of results which are in percentage terms. Although this indicator has no percentage relationship with any type of quantity, it can serve as a historical parameter of the student’s Soft Skills level in order to analyze their evolution over time.

The FSSA structure shown in figure 3 ensures that output variables from subsets S01, S02, ..., S10 were input variables from subsets S11, S12 and S13. Therefore, the universe of discourse of these inputs/outputs is the same. The same occurs with the S14, S15, S16 and S17 systems.

The FSSA structure consists of 1225 Fuzzy rules, with 125 rules for each of the S01, S02, S04, S07, S08, S09, S16 and S17 subsets; and 25 rules for each of the S03, S05, S06, S10, S11, S12, S13, S14 and S15 subsets. The rules were built and validated with biweekly monitoring by the expert⁷. Table 4 presents 15 of the 125 rules to assess competence Adaptability (AD), system S08 in figure 3. The input variables are Receptiveness (RE), Flexibility (FL), and Conflict Management (CM). One such example would be ““If AO=$\mathcal {G}_{0}$ and FL=$\mathcal {G}_{+1}$ and GC=$\mathcal {G}_{+ 2}$ then AD=$\mathcal {G}_{+1}$” (highlighted), that is, if a student has a Null or Low level of Receptiveness and has a Moderate level of Flexibility, and a High level of Conflict Management, then he has a level of Moderate Adaptability.

Table 4 Example to assess Adaptability

For the verification process of the knowledge base, 17 truth tables were compiled as regards subsets S01 to S17. For each of the Competencies to be assessed, expected values (proposed by the expert) and those found by the FSSA have been compared. This analysis was performed for crisp inputs at the extreme regions of the hypersurface/surface generated by the assessed Competency in median regions of the hypersurface/surface and for crisp inputs generated by the self-assessment and peer assessment of the 19 sampled students. For example, in the case of S14, the surface represented in figure 5 highlights Management Skills in terms of Management and Commitment. For verification purposes, the following inputs were considered:

• Extremes: (1, 1), $(1, -1)$, $(-1, 1)$ and $(-1, -1)$. Input (1, 1) corresponds to a student assessed as regards Management skills at a high level $\mathcal {G}_{+2}$ and Commitment skills also at high level $\mathcal {G}_{+2}$;

• Medians: (1, 0), (0, 1), $(-1, 0)$, $(0,-1)$ e (0, 0). Input $(-1, 0)$ corresponds to a student assessed as regards Lack of Management skills at high level $\mathcal {G}_{-2}$ and Commitment skills at low level $\mathcal {G}_{0}$;

• Students assessed in the case study were: (0.44; 0.50); $(0.19; -0.40)$ and $(-0.50; -0.55)$. There are three cases among the 19 sampled students. Both Management and Commitment skills were assessed for the first input at a moderate level $\mathcal {G}_{+1}$.

Fig. 5

Surface generated by subset S14

The results of assessment carried out by students in the case study were used to implement and validate the model. This process is discussed in the next section.

Analysis of Results

In addition to providing us with rich material to assist in the development of the FSSA knowledge base, the data collected in the case study served to simulate real inputs into the system, which facilitated the process of model verification and validation.

Participants

19 students of the Industrial Automation Technical Course at the Federal Institute of Rio de Janeiro, Volta Redonda campus, participated in the case study. They were aged 16.6 years on average for a total of 14 boys and 5 girls.

During the 7 weeks of the case study, students participated in Active Learning sessions involving Flipped Classroom, Problem Based Learning and Project Based Learning techniques and have been divided into 4 groups of 4 to 5 students. At the end of each Active Learning session, students assessed themselves and assessed their peers. For analyzing the data used as input in the FSSA, the students were identified as A01, A02, ..., A19.

Statistical Distribution of Parameterized Skills

Once having obtained each of the sets $A_1$, $A_2$, ..., $A_{26}$ given by $A_h=\underset{u=1}{\overset{19}{\cup }} \{PS_h^7(u)\}$ some measures were described regarding the 26 input variables (figure 6).

Fig. 6

Statistical distribution of Parameterized Skills

Thus, it was possible to collect some outliers, and analyze the dispersion of the data set and its asymmetry. For example, as regards the skill $h=05$ (Initiative) we have: outlier -36.2; minimum -18.0, first quartile -3.6, median 2.6, third quartile 12.0 and maximum 22.0.

Through the Q-Q plot, it was found that the data collected showed strong signs of adequacy to normal distribution, as suggested by the FSSA model. In figure 7, four out of 26 variables under assessment are exemplified by Q-Q plot. It was opted to use the Shapiro-Wilk test to confirm this normality.

Fig. 7

Q-Q plot $PS_h^p(u)$

Normality was confirmed for the skills at $5\%$ significance by the w of Shapiro-Wilk test with p (normal)$>\alpha$ (table 5). This table describes the sample size of each skill under assessment, the value of mean $\mu$, standard error $\sigma _{\bar{x}}$, w statistic and the calculated p(normal) value. From this confirmation, the transformation process for crisp $SPS_h^p(u)$ variables and membership functions was initiated.

Table 5 Normality test of variables

Analyzing the Membership Functions

Table 6 presents the values of Parameterized Skills of sampled students and the respective values of Standardized Parameterized Skills for the skill Planning ($h=1$).

Table 6 Transformation $PS_1^7(u)$ em $SPS_1^7(u)$ para $h=1$

It is observed that students A01, A09 and A11 achieved $PS_1^7(u)$ which is approximately the same as $\mu =1.77$, and therefore presented $SPS_1^7(u)\approx 0$, while students A07 and A16 had $PS_1^7(u)=0$ and thus presented $SPS_1^7(u)=z=-0.3828$. As previously discussed, students with $SPS_1^7(u)<z=-0,3828$ were found as possessing no planning skills in 7 Active Learning sessions and those with $SPS_1^7(u)>z=-0,3828$ were assessed as possessing planning skills in these classes.

Fig. 8

Relation between membership functions and Normal Distribution

By superimposing Gaussian over the membership functions of planning skill (figure 8), it is possible to observe a strict relationship between these functions and normal distribution, which is ensured by using parameter z. It is important to note that:

• The value of $\mu \approx 1.8$ which corresponds to $SPS_1^7(u)=0$ indicates that, on average, students realize that they possess planning skills themselves;

• Approximately $68\%$ of sampled students ($\mu \pm \sigma$) obtained $\mathcal {G}_{-1}$, $\mathcal {G}_{0}$ and $\mathcal {G}_{+1}$ (with greater concentration between $\mathcal {G}_{0}$ and $\mathcal {G}_{+1}$) regarding planning skills;

• Approximately $95\%$ of sampled students ($\mu \pm 2\sigma$) were assessed as achieving levels ranging between $\mathcal {G}_{-1}$, $\mathcal {G}_{0}$, $\mathcal {G}_{+1}$, $\mathcal {G}_{+2}$ concerning this skill (an insignificant amount in $\mathcal {G}_{-2}$);

• In the range between $-2\sigma$ and $-1\sigma$, students were assessed as possessing no planning skills at levels $\mathcal {G}_{-2}$, $\mathcal {G}_{-1}$ and $\mathcal {G}_{0}$ (which was higher in $\mathcal {G}_{-1}$);

• In the range between $1\sigma$ and $2\sigma$ students were assessed at levels $\mathcal {G}_{+1}$ and $\mathcal {G}_{+2}$.

Similarly to that exemplified in the variable Planning skill, the membership functions of all other input variables of subsets S01, S02, ..., S10 depend on parameter z, thus ensuring their strict relationship with the standardized normal distribution.

Feedback Reports and FSSA Validation

The diagram in Figure 9 highlights the proposed assessment process. Note that the beginning takes place with Active Learning sessions in teamwork. At the end of each class, students self-assess and assess their team members, signaling at least three outstanding skills and three skills to improve in that day’s session. This dataset is recorded in a skills database. After at least 5 Active Learning sessions, the reports can be processed, for that, the absolute frequency of the skills is parameterized and standardized, generating the crisp input variables. These variables are fuzzified through the Mamdani inference engine using the rule base, the fuzzy outputs that are defuzzified generate numerical and linguistic variables. For example, the vector $v_P=(5.4, 2, 4)$ indicates the parameterized variables of student A09 in 7 sessions of AL in the skills Motivation, Initiative, and Focus, respectively (S02 Figure 3). The standardization for this vector is $v_{SP}=(1.25, -0.02, 0.19)$, which is crisp input. Through the membership functions, the first component of the vector (Motivation) assumes the linguistic variables $\mathcal {G}_{+1}$ and $\mathcal {G}_{+2}$, the second (Initiative) assumes $\mathcal {G}_{0}$ and $\mathcal {G}_{+1}$, and the third (Focus) assumes $\mathcal {G}_{0}$ and $\mathcal {G}_{+1}$. After fuzzification, the output has a value of 0.5078 with linguistic variable (Interest) between $\mathcal {G}_{+1}$ and with $\mathcal {G}_{+2}$.

Fig. 9

Assessment process

Through the FSSA, it was possible to generate individual reports of 19 students sampled in the case study. Figure 10 presents only an excerpt from the Language and Social Skills Report of the individual profile of student A09 (a description of Communication and Innovation Competencies and Social Skills that are also part of this report were omitted from this figure). As this language report, the FSSA generates other indicators in percentage terms and Soft Skills incorporation levels.

Fig. 10

Report of outstanding Soft Skills and those that need improvement regarding student A09

Despite being evaluated by the expert and indicated as a possible instrument for providing continuous feedback, this report was not evaluated by teachers and students. Gibb (2014) highlights some strengths and challenges regarding the operationalization of feedback: the feedback makes it possible to close gaps between current and desired performance; the people tend to be more accepting of positive feedback; some people associate negative feedback with external causes; in order not to hurt formative assessment, feedback needs to be well managed; and it is peculiarly complex and challenging to operationalize feedback. Considering these elements, this research did not analyze the effects of feedback on the development of students’ Soft Skills.

These reports aim to provide professors, students, educational institutions and researchers with substantial feedback on the incorporation of students’ Soft Skills in an Active Learning environment. According to Lan et al. (2011), students can reflect and try to acquire the knowledge they are lacking in by receiving feedback.

Once in possession of reports and interviews generated by observers, teachers and students in Active Learning sessions, the knowledge engineer and the expert compared the results of Soft Skills profiles of 19 students sampled in the case study with those found by the FSSA. The validation procedure took place as follows:

• It commenced by assessing system S01 (Managerial competency assessment);

• The expert analyzed the results produced by the FSSA, indicating agreement and disagreement in relation to Managerial competency of the 19 sampled students (in addition to some arbitrary inputs determined by them);

• In case of agreements, part of the S01 system was considered validated;

• In the case of disagreements, the system was reconfigured by the knowledge engineer;

• Cyclically, the system provided the expert with feedback on assessment and the knowledge engineer on readjustments to be made in such a way as there would be no more disagreements between received results and students Managerial competency according to their expertise and case study reports;

• Thus, the S01 system was considered validated;

• Then, the S02 system assessment (assessing the Interest) started out in a similar way to that of the S01 system;

• This process was carried out consecutively until all systems from S1 to S17 have been validated by the expert.

Although there was an initial interest in using the system validated by the expert to assess the Soft Skills of other groups of students participating in Active Learning sessions in order to ensure a more robust validation of the FSSA, the health crisis imposed by the Coronavirus pandemic made this proposal unfeasible. Thus, this proposal is suggested for further works.

In summary, the individual Soft Skills profile of sampled students generated by the FSSA is described in table 7. In it, the terms “Posse.” was used to indicate that student possesses the skill at a moderate or high level, “Poor” to indicate that the student possesses the skill at a low level and “Lacks” to indicate that the student does not possess the skill. Table 7 shows that student A13 “Posse” Leadership (S17, figure 3), Social Skills (S16), Managerial (S01) and Commitment skills (S11); has “Poor” Effective Communication skills (S12) and; “Lacks” Creation Power (S13).

Table 7 Individual students profile provided by the FSSA

Three tables similar to Table 7 were build, the first with the observers’ perceptions of each student’s Soft Skills profile, the second with the teacher’s perceptions, and the third with the students’ perceptions based on the expert’s analysis. The data collected by the observers during the case study, and the interview with the teacher used the semi-structured form mentioned in section 3.1.

The results obtained by the FSSA were compared with those found in the case study involving observers, teachers and students. Figure 11 shows the percentage of coincident, approximate and opposite perceptions between the results of FSSA $\times$ Observer, FSSA $\times$ Teacher and FSSA $\times$ Students. For clarification purposes, as regards a comparison between FSSA $\times$ Observers, it should be considered as Coincident perceptions those that both the Observer and the FSSA presented the same result for the student (example: both the Observer and the FSSA found that student A13 has “Posse.” leadership skills). Approximate perceptions are those in which one of the actors concluded that the student has “Poor” skill and the other actor found that the student has “Posse.” or “Lacks” skill (for example: the Observer perceived that student A07 has “Poor” Interpersonal Relationship skills, while the FSSA concluded that this student has “Posse.” this skill). Finally, Opposite perceptions are those in which one of the actors concluded that the student has “Posse.” the skill and the other one concluded that the student “Lacks” skill (example: for the Observer, the student A17 has “Posse.” Commitment skills, but the FSSA proved that he/she “Lacks”). Similarly, such an interpretation applies to a comparison between Teacher $\times$ FSSA and Student $\times$ FSSA.

Fig. 11

Percentage of coincident, similar and opposite perceptions in the comparison between FSSA results and the case study

With respect to figure 11, it can be observed that the percentage of coincidence between FSSA and students ($76.3\%$) is higher when compared to the perceptions of observers ($44.7\%$) and the teacher ($63.1\%$). In addition, the case for opposing assessments is $11.4\%$ between the FSSA and the observer and $4.4\%$ between the FSSA and the teacher, and it has reached zero percent between the FSSA and students. These results are supported by the $\chi ^2$ with $\alpha =0.05$ significance level $p<0$ ensures the rejection of a null hypothesis ($H_0$: the coincident, approximate and opposite perceptions compared to the FSSA do not vary according to the type of evaluator, and $H_1$: the coincident, approximate and opposite perceptions regarding the FSSA depend on who is assessing skills). That is, the frequencies of coincident, similar and opposite perceptions under assessment are different from the expected frequencies (theoretical).

We use data from self-assessment and peer assessment as input to the FSSA. Thus, these differences in percentages can be explained by the fact that the same data used to draw the students’ table (similar to table 7), were used as input for the FSSA.

Fig. 12

Percentage of coincident, similar and opposite perceptions in the comparison between the FSSA results and Students by competency

A comparison between FSSA and students regarding the six Competencies highlighted in table 7 (Managerial, Commitment, Effective Communication, Power of Creation, Social Skills, and Leadership) shows the following results: $89,5\%$ of perceptions coincide in Managerial, $84,2\%$ in Commitment, $63.2\%$ in Effective Communication, $63,2\%$ in Power of Creation, $73,7\%$ in Social Skills, and $84.2\%$ in Leadership, (figure 12).

In all cases in which there was a difference between the perception of students and the FSSA, they were on the borderline between “Posse. And Poor” or “Poor and Lacks”. For example, the difference between “Having Analytical Skills or Having little Analytical Skills” and the difference between “Having no Analytical Skills or Having little Analytical Skills” is very subjective in a behavioral analysis. Therefore, despite being different perceptions, they were considered as approximate perceptions.

Discussion

Among the main gaps discussed in the introduction, the proposed assessment system contributes to

• providing a simple and quick Soft Skills assessment strategy, allowing use in all team classes;

• ensuring assessment of the student’s individual Soft Skills profile based on self-assessment and peers’ assessment;

• presenting a Fuzzy inference model with a knowledge base and ontology defined from a literature review, case study, and expert interviews, dismissing the need for an expert “psychologist” during the assessment process;

• and indicating an evaluation process that can help students, teachers, educational institutions, and researchers in the development of Soft Skills for future professionals.

Some limitations in this research can be minimized with the perspective of future work. One of the main limitations was the use of only one expert in the knowledge acquisition stage, due to two factors: difficulty in finding experts with availability to be present in fortnightly meetings with the knowledge engineer for a year and; difficulty managing the number of tasks to be performed with more experts. The selected expert participated in: semi-structured interviews; consolidation of observation forms; the training of psychology students who carried out the observations; the interpretation of student profiles based on observers’ reports; the analysis of individual student profile reports issued by the FSSA in comparison with the data collected in the case study and; the system validation. The interaction of other experts could guarantee a greater degree of consensus. However, in possession of the knowledge base acquired in this research, future work with expert panels can be used to optimize the proposed system.

Another limitation was the validation of the FSSA was restricted to the expert’s analysis, as there are difficulties in finding a base case for assessment the Soft Skills profile of students using AIED. Despite this difficulty, a proposal for future work is to compare the results obtained by the FSSA with other Soft Skills assessment instruments, such as questionnaires (Deep et al., 2019; Chan et al., 2017; Levant et al., 2016; Conchado et al., 2015), self-report questionnaires based on scenarios (Mulcahy-Dunn et al., 2018), Situational Test of Emotional Understanding (STEU) and Situational Test of Emotion Management (STEM) (Maccann et al., 2011; MacCann and Robert, 2008).

It was also considered as a limitation, the non-use of teacher assessment in the FSSA. Although some authors (Verbic et al., 2017; Shuman et al., 2005) have suggested the 360-degree assessment as the best instrument to assess the Soft Skills of individuals, to enable the use of the FSSA in all Active Learning sessions, we used as input to the system only the data collected from the self-assessment and peer assessment. Although we collected, in the last session of Active Learning, the class teacher’s perception of the students’ Soft Skills, this data was used only during the development of the knowledge base and system validation. Given the results found, the exclusive use of data collected from the student’s self-assessment and peer assessment proved to be adequate to describe the Soft Skills profile of the students. However, as a proposal for future work, it is intended to incorporate the teacher’s assessment of the class, increasing the accuracy of the expected results.

Not implementing the FSSA in regular classes after its validation can also be considered a limitation since in possession of the results of the application in more classes, possible effects (positive and negative) (Gibb, 2014) of the feedback reports in the incorporation of the Soft Skills of the students can be evaluated and adjusted. Thus, a proposal for future work is the longitudinal analysis of the effects of continuous feedback promoted by the FSSA, on the development of students’ Soft Skills.

Finally, the assessments were collected on paper forms and used as input in the system. The lack of an interactive and user-friendly platform was also considered a limitation. Ensuring an application for smartphones and computers is also a proposal for future work.

Conclusion

Despite counting on the expertise of a single expert, the knowledge acquisition process through FSSA involved a case study and a theoretical/conceptual analysis, thus ensuring a substantial knowledge base. Thus, with inputs self-assessed by students and their peers, the system can infer the level of incorporation of Soft Skills of a given student.

The selection of Fuzzy Logic for FSSA inference showed to be adequate, since the degree of uncertainty arising from decisions involving the subjectivity of behavioral analysis has been taken into account.

From the theoretical/conceptual analysis and interviews with the expert, 26 skills were used to compile the 1225 FSSA rules in a hierarchical structure that avoids combinatorial explosion, enhances computational processing and facilitates the system validation process.

The absolute frequencies of skills were converted into $PS_h^p$ which enhances the individual assessment of a given skill when perceived by more than one actor.

A conversion of $PS_h^p$ into $SPS_h^p$ through standardization enabled to treat variables in a universe of discourse of $[-5, 5]$ by associating the membership sets to a standardized normal distribution.

The definition of membership sets $\mathcal {G}_{-2}$, $\mathcal {G}_{-1}$, $\mathcal {G}_{0}$, $\mathcal {G}_{1}$ and $\mathcal {G}_{2}$ was performed using parameter z as a reference, in addition to ensuring a strict relationship between the input data with normal distribution, which means that membership sets are not displaced based on mean $\mu$ of $PS_h^p$ but based on the z value that is the limit between possessing or not possessing a certain skill. With linguistic output variables and crisp output interval between -1 and 1, the interpretation of results is facilitated for professionals who are not very familiar with numerical quantities.

The Soft Skills profile of students found by the FSSA in comparison with the Soft Skills profile perceived by students in self-assessment and peer assessment (case study) showed good agreement.

As a contribution, this research proposes an expert system that describes the Soft Skills profile of students, using an expert’s knowledge base. Inputs to the system are student self-assessment and peer assessment. The process is simple and fast and can be used in all Active Learning sessions. In the discussion section, we present some proposals for future work, aiming to minimize the limitations: the use of only one specialist during the acquisition of knowledge and validation of the system; the non-use of teacher evaluation; the non-implementation of the system in classes after validation and; the lack of an interactive and friendly platform.

As there are few expert systems for the domain of Soft Skills in AIED, this work helps to advance the discussion about this field of study.