Challenges to Engineering Design Teamwork in a Remote Learning Environment

Abstract

Remote team collaboration was not familiar to many engineering students before COVID-19. The rapid shift from in-person to remote during the pandemic caused dramatic challenges, especially for freshmen and sophomore students in engineering design classes, where teamwork is typically needed to explore both the problem and solution spaces for ill-defined problems and students have had little previous design project experience. This study aims to explore challenges revealed by students in remote design collaboration through the lens of a sophomore-level class about early-stage engineering design. The authors closely observed team members’ struggles through three datasets collected in one semester: (1) team performance and survey responses in an in-class idea generation activity; (2) individual student final reflection essays about their semester-long team project at the end of the semester; and (3) bi-weekly individual reflections on the discussion board throughout the entire semester. Unlike the classic findings that sketches improve performance, we found significant positive correlations between teamwork experience (e.g., communication, efficiency, perceived contribution) and the number of ideas expressed in text, and significant negative correlations between teamwork experience and number of ideas expressed in a combination of sketches and text. Therefore, we propose educators should also work on improving students’ ability to express design ideas with text descriptions, on top of traditionally emphasized visual representations. In addition, we found the remote environment exacerbated existing team challenges more than it created new challenges. The remote-related challenges also dropped dramatically after the first few weeks and then remained steady. The remote-related challenges and their changing patterns indicate large potential to improve remote design collaboration.

1. Introduction

The COVID-19 pandemic has changed higher education and introduced new challenges and hidden benefits. To best fit student needs and address safety concerns, remote asynchronous and synchronous learning are the most widely used modalities [1]. Remote learning involves more flexibility in time and location [2]. For example, students no longer commute to classes and can access more learning resources. On the other hand, the challenges caused by remote learning cannot be ignored. These include lack of engagement, more opportunities for academic dishonesty, and lack of experience in self-regulation [1]. Some students struggle in their home environment to remain focused [3]. Technology can also be a source of frustration, even though most students have grown up with high exposure to technology and have a strong understanding of how to solve technological difficulties [3,4]. Additionally, the reliance on technology has become a major barrier to education for students (and some educators) who do not have consistent access to digital devices or internet [5]. These challenges are particularly prominent for the classes that are traditionally taught in-person and require significant amount of various activities, such as engineering design.

It is acknowledged that online education is not new, and it has gained more popularity since the late 20th century [6], in such areas as computer science, education, statistics, and architecture virtual design studios [7]. This is seldom emphasized, however, in traditional engineering education (e.g., mechanical engineering, electric engineering). The COVID-19 pandemic forced engineering educators to abruptly jump into remote education. This presented challenges, but also provided opportunities for educators to seriously consider how to establish successful remote collaborations, and how to improve students’ ability to do so. Implementing design collaboration into a remote environment has been a particular challenge, even for companies that have offices worldwide. The remote environment is not conducive to teamwork and can be a cause of frustration for team members in terms of communication, disproportionate engagement, and scheduling. Online engineering design presents the challenge of creating designs without access to labs, physical materials or equipment [8], and physically showing and discussing design sketches. A common approach to understand and address such challenges is through a continuous improvement program that surveys the subjects and provides feedback [9]. Rather than using surveys in a post-hoc manner, this study employs an anthropological approach, which observed and captured student learning behaviors and team design performance with multiple approaches (see Section 2) to organically track challenges that students faced in a mechanical engineering design class during a 14-week period. The findings suggest areas to investigate more to improve teaching and learning experiences in remote design collaboration. In the long term, such investigations will help engineering educators purposefully integrate both remote and in-person components into engineering design, and thus better prepare students to perform well in a distributed workforce.

1.1. Engineering Design Process and Education

Engineering design is a process in which open ended problems are analyzed and solutions are generated and assessed within a set of constraints. The process requires an amalgamation of skills including design thinking, communication, creativity, problem solving, and technical knowledge. In a typical engineering design class, students are expected to learn design methods, theories, and tools in lectures, and practice to use them through laboratory activities. To connect what students learn with how to apply them to solve real-world problems, team projects are common where students spend several months with their teammates to define, design, and iterate solutions. There are multitudes of methods that break the engineering design process into steps. Howard’s survey of 24 papers identified 4 major phases: analysis of tasks, conceptual design, embodiment design, and detailed design [10]. Pahl and Beitz break these down further into the following seven steps: defining the task and requirements, identifying essential problems, establishing function structures, searching for solutions, selecting solutions, firming up principle solutions, and evaluating principal solutions [11]. Searching for and selecting solutions are also commonly referred to as concept generation/ brainstorming and alternative analysis respectively [10,12]. Additionally, engineering design is iterative, i.e., each individual step requires planning, monitoring, and evaluation, and as designs are refined, the designers must frequently refer back to previous steps [13,14]. In the class examined in this study, the design process treated is most similar to Pahl and Beitz. The only exception was that students did not have the opportunity to physically build or evaluate designs.

The engineering design process is frequently taught with the idea that experience is the best teacher, where students work on longer term projects that are broken down into smaller pieces. According to Libii [15], there are four main ways to integrate design into engineering education. There are first year design experiences that are typically focused more on a conceptual understanding of the design process, rather than the application of technical knowledge, which is often found in capstone courses [15,16]. These first year experiences are particularly valuable because they promote soft skills that are essential for teamwork, like communication, that employers value greatly [16]. The other three ways to incorporate design are design focused classes, integration into core courses (e.g., fluid mechanics), and senior capstone projects [15]. Usually, these types of projects require students to work in teams. The formation of these teams may differ at each university and course. Some common methods are student and instructor selection based on criteria like strengths, weaknesses, GPA, project interests, and random assignment [13,17,18]. Project selection also plays a major role in the outcomes of these experiences. These projects should have achievable objectives, promote creativity, provide external feedback and have defined intermediate milestones in order to provide suitable, but feasible challenges and encourage motivation and engagement [17]. These design problems are often derived from industry partnerships, service learning projects, external design competitions, and faculty research [19]. Another important aspect to consider in engineering design education is how students externalize and communicate design concepts. Sketching and text description are often-used tools in in-person studies by which ideas can be quickly generated and communicated [20]. The importance of sketching skills are undeniable due to its effectiveness in conveying large amount of information, enhancing creativity, and improving communication [21,22,23,24]. Representing ideas in written or verbal texts are used naturally in design activities [25], but less purposefully emphasized. One possible reason is that most of the design studies and design collaboration activities in engineering happened in face-to-face communication in the past. We speculate that, with the increasing needs for remote collaboration, the importance of textual representations will rise.

1.2. Team Collaboration

Team-based learning is found to be very effective in engineering design [26], especially when it emphasizes concept application with scaffolded processes through which students can learn both the concepts and the application in self-managed learning teams. This experience also helps students improve team collaboration and communication skills (which are the mostly mentioned characteristics of top-notch professionals).

With the inclusion of teamwork-related skills into the Accrediting Board for Engineering and Technology (ABET) requirements, universities have sought to incorporate teamwork into their curriculum, often through design projects. Historically, it has been assumed that experience is the best way to teach team skills. However, with the hands-off approach instructors often take, students can find group projects frustrating and threatening to their grades.

Chowdhury identified communication as one of eleven key attributes for effective teams and claims team members should practice open dialogue, active listening, and regular and prompt communication [27]. Communication is often a leading problem in student teams. Lingard found that students perceived themselves as being poorly skilled in communication [28], meanwhile Mostafapour’s study contradicts this, as students felt themselves highly skilled in communication [18]. However, it is additionally noted that the most frequently reported conflicts are commonly caused by poor communication and that students potentially “don’t know what they don’t know” [18]. While those studies were completed with traditional face-to-face classes and little research had been completed on the effects of remote work due to COVID-19, Waisenegger noted a decrease in spontaneous interactions and an increase in scheduled and focused communication among adults working from home [29]. Unfortunately, the lack of spontaneous information has decreased knowledge sharing and made people hesitant to ask for help, which is a theme Chowdhury labels as interdependence [27,29]. According to Lingard’s survey, students who had already scored themselves low in both asking and providing help within their teams without the negative effects of remote collaboration, thus this could be an urgent issue that instructors must address [28]. In addition, the remote learning environment is not friendly to quick sketching and sharing ideas with the current technology, while sketching is traditionally emphasized in engineering design as an important communication medium [22,23,30]. It is interesting to observe teams’ natural choices of how they communicate ideas and the related challenges.

Besides communication issues, negative experiences and conflicts with teamwork are common among students, to the point where negative emotions and stress make students wary and underestimate the importance of teamwork. Instructors in Mostafapour’s study determined that differences in expectations are a major theme amongst team conflict [18]. This was also validated by Chowdhury’s inclusion of shared goals and values as a common attribute among effective teams [27]. Mostafapour’s research showed that teams with shared expectations perceived higher effectiveness and enjoyment [18]. While students in a study of global design teams claimed to easily resolve or not experience conflict at all, several groups reported unresolved conflict at the end of the project, resulting in a recommendation to provide formal conflict resolution to the teams [31]. The students in Mostafapour’s study also perceived themselves as poorly skilled and in need of more training in conflict resolution and dealing with difficult team members, who were found to produce the worst effects on enjoyment [18]. With these conclusions, it is surprising that formal conflict resolution has not become a standard topic in the education of student teams.

1.3. Research Questions

To examine team behaviors and challenges in the context of remote collaboration in early-stage design, we ask the following questions:

RQ1: How are the usages of sketches and text descriptions in idea generation related to team performance in an online environment?

Sketches are typically encouraged to represent ideas, especially in early stages, as it can improve communication and design creativity in in-person teamwork. Remote collaborations, however, might hinder most students from easily sharing and explaining ideas to peers with supporting sketches. This question seeks to understand students’ natural choices of representing design ideas in sketches, texts, or a combination in remote team collaboration and whether there are any relationships among the representation modalities and the team performance. It is common to give students or study participants freedom of using sketches, texts, or their combinations in idea generation at early stage [32,33,34]. One associated question is whether high performance teams more likely to use sketches. Another is how the number of ideas generated in different modalities related to team members’ perceived team contribution, communication, and meeting efficiency.

RQ2: What are the challenges in early design processes that remote teams have? Are there any changes to these over a semester?

Considering students’ limited design project experiences and the abrupt change from in-person to remote, there should be distinct challenges that engineering students encounter. This question seeks to understand the challenges in the context of remote team collaboration in early- stage design, and examine changes in the challenges over time. We expect the observed challenges and their changes will be instrumental for educators to pinpoint potential areas for interventions to improve remote design collaboration. Additionally, we look into the emotions of students throughout the semester to understand the general feelings of students while they face these challenges.

2. Materials and Methods

2.1. Context

We collected data from a second-year engineering design course at a public university in the United States in Fall 2020. Due to pandemic, the entire course was instructed remotely. A total of 42 students in 11 teams learned and practiced the human-centered design process for 14 weeks. Students who participated in the study were offered bonus points on their team project grade. For students who did not want to participate in the study, an alternative assignment was provided to gain the same bonus points. Student participation was completely voluntary, and all students agreed to participate. Use of students’ coursework data was approved by the university institutional review board.

Students were strategically organized into teams for the entire semester on the first day of the class. Teams were purposefully diversified by considering students’ gender, CAD (Computer-aided Design) skills, writing and presentation skills, leadership skills, and internship experience. The course content was organized into major units based on the design processes (e.g., define, design, select, analyze, etc.). Teams selected their own project in response to an open-ended design problem, “Design for the coronavirus pandemic: Design a way to help people stay healthy physically and/or mentally”. The selected projects need to meet basic requirements: (1) involve mechanical components; (2) require basic engineering calculations and analyses; and (3) that the design can be modeled in CAD. At the beginning of the semester, teams were required to define their project charters to guide their remote collaboration (i.e., objectives, scopes, expected deliverables, responsibilities, etc.). Then, every one or two weeks, teams submitted an assignment to report their progress.

2.2. Datasets

To observe student remote design collaboration challenges, three datasets were used: (1) a 45 minute class activity on team design idea generation in Week 7; (2) student final reflection essays about their semester-long team project written at the end of the semester; and (3) bi-weekly reflections on the discussion board throughout the entire semester. See Figure 1 for a summary of the data sources. The following paragraphs provide details about dataset (1), and a brief overview about datasets (2) and (3) (see Ref [35] for details). All these activities were online due to the pandemic.

Dataset (1). In the brainstorming activity, students were randomly assigned to a team, different from their semester-long project groups. During class, each team was assigned with an individual breakout room on Zoom. A template to capture their design ideas was provided. This was mainly used to answer RQ1. All teams performed brainstorming on the same design problem: “Your team aims to design a cash cleaner to reduce disease spreading during cash transactions. It accepts cash into a covered machine, then the cash is moved through a UV cleansing area. Finally, the cash is dispensed on the other side of the machine. One of the key sub-functions is move the cash. Generate as many ideas as possible for moving cash in the cash cleaner. The ideas can be sketched and/or written as text.” Teams were asked to follow the instructions on the template. First, individuals generated ideas for 8 minutes (Stage 1), then uploaded their ideas to the shared template on Google Slides. Next, teams together reviewed each other’s design ideas to get inspiration, then continued to refine and generate ideas (Stage 2) for 13 minutes. Ideas from the two stages were documented separately. Lastly, the teams discussed their design ideas and each filled out a post-brainstorming survey individually, which collected information such as perceptions of their team work efficiency, communication, their own comfort levels in expressing ideas with sketches and text descriptions, challenges during the team design (see Appendix A for the survey questions). Aside from the problem statement and meeting timeline, no other limitations were imposed on the students. They were free to use paper and pencil, digital sketching tools, or any other medium to complete the activity. This activity served as a low stakes practice round for iterating on designs based on functional analysis as a group before they were required to do the same on their semester long projects. As a result, the students did not present designs to the professor for feedback and the activity was graded based on completion.

Dataset (2). Upon finishing the project at the end of the semester, each student was asked to write a final reflection essay specifically about their own semester-long team design project, such as the process their team used, what they did well, what they struggled with, and how they could improve (see Appendix B for the detailed prompts in the essay assignment). The design problem each team chose for their semester-long project was different from the simplified problem in the in-class activity of Dataset (1), which was used for an idea generation practice.

Dataset (3). Individual students were required to post regular reflections biweekly on a discussion board, and were encouraged to respond to their peers on the board. They were prompted to write down their challenges, excitement, and adjustment strategies during the reflection window (see Appendix C for the detailed prompts in the bi-weekly reflections). Answering RQ2 mainly relied on the results from Datasets (2) and (3).

2.3. Data Processing

2.3.1. Dataset (1)

Dataset (1) consisted of PowerPoint slides with teams’ design ideas in sketches, text, or combinations of sketches and texts from the two stages, and individual student post-activity survey responses. The goal of the first stage was to generate possible methods of moving money for a sanitation machine in 8 min while the second stage was aimed to refine and expand upon those ideas in 13 min. The number of ideas in each stage, representation format (text, sketch, or a combination) was recorded for each team. Teams were separated into high and low based on a combination of the quantity of ideas generated per person in each stage and their team’s average efficiency, communication, and contribution scores from the post activity survey. Teams with above average values in most categories were considered high performance while teams with below average values in most categories were considered low performance. After these initial distinctions were made, three groups that performed poorly in concepts generated per person, but high in team performance were assessed using sketch complexity. The deciding factor for teams on the border of high/low was the complexity of their sketches to make up for the fact that more detailed sketches tend to take more time and could possibly result in a lower number of ideas. Two groups had high complexity sketches similar to the one shown in Figure 2 and were classified as high performing. The groups with lower complexity were classified as low performing. It is important to note that sketch complexity alone cannot determine a team’s performance nor the quality of a concept, but rather among groups that produced a similar quantity of sketches per person, including more details adds to the time it takes to sketch a single idea in a timed setting.

Due to technical issues, the data from one team was corrupted, thus only 10 teams were analyzed. Five teams were classified as low while the other five were high from the processes described above. An average rating for each survey question was calculated by averaging the ratings from all members in a team. The responses for the open-ended question about the challenges in this activity was reviewed. The frequency of each challenge was counted across the teams. We divided the teams into only two groups (high/low) in order to examine if there are any obvious difference patterns among high and low performing groups. They were merged when calculating correlations among factors.

2.3.2. Datasets (2) and (3)

We used grounded theory [36,37] to process Datasets (2) and (3) because the data are in text format. The process involved the following basic steps: coding, categorizing, and theorizing. First, an initial broad set of categories was developed based on the research questions reported in [35]: challenge, strategy, and emotion. Then all the data were separated into segments and tabulated against the main categories. Next, a bottom-up approach was used to identify possible codes for each category, i.e., placing the keywords from a data segment into the corresponding columns that make sense, and then further combining and refining the keywords so that they are abstract enough to be categorized while still easily understood without requiring the audience to look into the details of the data. In this round, we sought to keep a small number of keywords so that a big picture could be perceived at the first glance. We dove deep into detailed coding in the next step. For the same reason, we kept only two levels for emotions (positive and negative) to track the emotional dynamics of the teams along the way, instead of going into detailed emotions, e.g., proud, anxious. It is possible that a data segment might only have keywords identified for one category, such as emotion, while not related to challenge or strategy at all.

Table 1. Example Coding Generated.

Data Segments	Challenge	Strategy	Emotion
Seeing the design process from start to finish has inspired tremendous confidence in the possibility that I could do this for my future profession.			Positive
One of the biggest troubles that our team encountered was setting up a meeting ... As I am staying in a country with a different time zone, this even poses a bigger challenge.	Team, remote
Every assignment we submitted, I asked everyone the question of, “Are you proud to have your name on this” and we did not want to submit anything until the answer was “Yes”.		Shared goals/expectations

provides example coding. Strategy-related findings are not reported in this paper. See Ref [35] for more complete findings.

3. Results

3.1. Observations from Dataset (1)—RQ1

Observed differences among high and low performing teams.

Due to small sample size, statistical tests on the data to compare high and low performing groups was not performed. Instead, descriptive statistics of means were used to show if there are any different patterns. As show in Figure 3, the average number of ideas represented in text, sketches, and a combination of text and sketches in low performance teams in Stage 1 are 1.8, 1.2, and 5.6, respectively. The corresponding numbers in the high performance teams are 3.2, 0.2, and 6.2. It can be observed that, in Stage 1: (1) both groups were more likely to represent ideas in a combination of text and sketches, and (2) the high performing teams used text more and sketches less than low performing teams.

Additionally, the average number of ideas in text, sketches, and a combination in Stage 2 are 0.2, 0.8, and 5.2 for low values, and 4.4, 0, and 1.8 for high. The numbers indicate the high performing teams made a noticeable shift from predominantly using combinations in Stage 1 to predominantly using text in Stage 2, while the low performance teams used combinations the most in both stages, but also increased the numbers of sketches in Stage 2.

Figure 4 shows the frequency of different challenges reported by both high and low performing teams about the in-class activity. Overall, more challenges were reported in the low performance teams (high vs. low: 7 times vs. 17 times). Both high and low teams struggled with understanding other’s design ideas, and limited ability to draw ideas. Low performing teams were more blocked by their low confidence in their own ideas, technical difficulties with online tools, and unfamiliar teammates. When representing ideas in sketches, most students drew on paper, then took pictures and uploaded them to the shared document. Lack of tools to support easy sketching and sharing online might have exaggerated the difficulty feelings of technical difficulties, and other challenges.

Correlations among various team factors and propositions to improve remote team design.

In response to the second part of RQ1, pairwise correlations among number of ideas generated in different modalities, team members survey data, such as perceived team contribution, communication, and efficiency of the meetings, were calculated by using the MATLAB Econometrics Toolbox. Given the sample size of 10, a two-sided test with power 80% is able to detect an approximate simple correlation of least 0.8 at 5% significance level, and a simple correlation of least 0.7 at 10% significance level [38].

Table 2. Spearman correlation coefficients between pairs of variables in matrix of data (sample size:10 teams). Red text indicates significance at a 0.05 level, and the blue text indicates significance at a 0.10 level. A coefficient value indicates the strength of a relationship, i.e., the larger the coefficient, the stronger the relationship.

	Commu	Contri	Efficiency	Enthusiasm	CSketching	Ctext	Text1	Sket1	TeSk1	Text2	Sket2
Contri	0.97
Efficiency	0.90	0.83
Enthusiasm	0.78	0.81	0.81
Csketching	−0.08	−0.17	−0.01	0.10
Ctext	0.56	0.43	0.51	0.55	0.43
Text1	0.04	−0.03	0.07	0.10	−0.08	0.15
Sket1	−0.22	−0.10	−0.26	−0.02	0.26	−0.26	−0.06
TeSk1	−0.27	−0.26	−0.25	−0.33	−0.02	0.17	−0.57	−0.43
Text2	0.54	0.56	0.50	0.76	0.14	0.65	0.51	−0.09	−0.32
Sket2	−0.39	−0.43	−0.25	−0.22	0.28	−0.49	0.14	0.38	−0.57	−0.39
TeSk2	−0.70	−0.70	−0.61	−0.74	−0.35	−0.45	−0.22	−0.12	0.45	−0.78	0.21

KEY: “Commu”—Team communication, “Contri”—Team contribution, “Efficiency”—Team meeting efficiency, “Enthusiasm”—Enthusiasm in design, “Csketching”—Comfortable level with sketching to express ideas, “Ctext”—Comfortable level with text descriptions to express ideas, “Text1”—Number of ideas from a team represented in text in Stage 1, Sket2”—Number of ideas from a team represented in sketches in Stage 1, “TeSk1”—Number of ideas from a team represented in a combination of sketches and text descriptions in Stage 1, etc. See Appendix A for the detailed survey questions about the first 6 factors.

summarizes the pairwise Spearman correlation coefficients. Significant correlations are highlighted using bolded red font (p-value < 0.05). Coefficients in italic blue fonts with underlines are marginally significant (p-value < 0.1). Shaded means significant positive correlations. Careful interpretations are needed when drawing insights due to the limited sample size. The following summary helps to provide some preliminary data for the hypotheses about team factors in remote design idea generation.

From Table 2, two propositions are derived for future research to improve remote design collaboration. The supporting evidence is provided after each proposition.

Proposition 1.

Tools, strategies, or logistics that enhance team online communication, meeting efficiency, or team members’ design enthusiasm would positively contribute to team performance during collaboration (e.g., team members contribute more ideas and provide more constructive feedback).

This proposition is supported by observations that team communication, meeting efficiency, perceived team contributions, and team members’ enthusiasm in design are significantly (p-value < 0.05) and positively (r > 0.75) correlated with each other as measured by the pairwise Spearman correlations.

Proposition 2.

Improving students’ ability to express ideas with text descriptions might improve team performance and enhance communication.

This proposition is supported by six observations: (1) Comfort level to express ideas with text descriptions is marginally and positively correlated to team communication (r = 0.56, p-value < 0.1). (2) Comfort level to express ideas with text descriptions is significantly and positively correlated to number of ideas expressed in text in Stage 2 (r = 0.65, p-value < 0.05), but comfortable level with sketches is not correlated with any items. (3) Design idea representation format in Stage 1 is marginally and positively correlated with that in Stage 2 when text format is used (text: r = 0.51, p-values < 0.10). i.e., higher number of ideas in text in stage 1, higher number of ideas in text in Stage 2. (4) The number of design ideas represented in text in Stage 2 is marginally and positively correlated to team communication (r = 0.54, p-value < 0.10) and team contribution (r = 0.56, p-value < 0.10), and significantly and positively correlated to enthusiasm in design (r = 0.76, p-value < 0.05), and comfort level to express ideas with text descriptions (r = 0.65, p-value < 0.05). (5) The number of design ideas represented in a combination of text and sketches in Stage 2 is significantly and negatively correlated to team communication (r = −0.7, p-value < 0.05), enthusiasm in design (r = −0.74, p-value < 0.05), team contribution (r = −0.70, p-value < 0.05), and marginally and negatively with meeting efficiency (r = −0.61, p-value < 0.10). (6) On a 5-point scale, the average comfort level with hand sketching was a 3.91, while the average comfort level with text was a 3.88. There was not a significant difference in comfort level between sketching and text.

3.2. Observations from Dataset (2)—RQ2

In total, Dataset (2) consisted of 41 essays that specifically discussed students’ design projects. R Studio was used to calculate the frequency of each keyword in each category.

Each member reflected on their team project individually in the final essays, regarding the process their team used, what the team did well, how they could improve, how they evaluate and justify each team member’s overall contribution. The data from the essays serves as a more comprehensive view to learning pains in the team’s design process specifically. In total, 62 individual data were identified as challenges. Some were coded with more than one keyword. For example, the quote below from a student’s response was coded with both remote and understanding course content. Figure 5 indicates the frequency of each type of challenge mentioned in the final essays.

“In addition to the online environment and differences in time zones, although I’d been working with other team-based engineering projects prior to this class, this was the first time I tried a professional approach in regards to identifying and solving engineering problems. At the start, it was extremely confusing. Most of the aspects that we all used to tackle qualitatively in our subconscious minds were brought out to evaluate quantitatively on paper in this project, therefore, none of the team members, including myself, knew what to do or how to complete the provided forms and procedures.”

At a higher level, it was found that the most frequently mentioned challenges were team-related (27 times), which included, for example, divergent mindsets and personalities, decision making, scheduling and commitment to meetings, unequal team contributions. 22% of the team-related challenges were explicitly attributed to being remote, such as different time zones and technology limitations for remote collaboration.

The second category was understanding course content (25 times). It included conceptual understanding and applications, as well as confusion about the expectations of specific tasks, such as systematic ways of identifying customer requirements, engineering specifications, and generating design concepts. Only 4% of these challenges were explicitly attributed to the online learning environment.

The third category included all the challenges that explicitly mentioned being remote (12 times). Some were coded into multiple categories (e.g., team, understanding course contents, including remote). Therefore, only 7 were coded as remote in general. These include missing the face-to-face communication, difficulty adjusting to online classes, being stuck in the house, spending a long time in front of the computer, and staying mentally motivated.

Lastly, specific products that teams worked on also created new challenges (8 times). They were mostly about difficulties in finding competitors and information on competitors, because design for COVID-19 was such a new topic at that moment. A personal situation was mentioned by one student, which delayed her study progress and team work.

3.3. Observations from Dataset (3)—RQ2

Across the 6 biweekly reflections in the first 12 weeks, an average of 37 students participated in each reflection discussion. R Studio was used to calculate the frequency of each keyword in each category.

Students were required to reflect biweekly about their gains, difficulties, and adjustments in this class. Data from biweekly reflections were expected to be more specific because they captured difficulties at a given moment and reduced dependency on a student’s long-term memory about what happened in the past. It is also possible that students focused more on their personal experiences in the class overall instead of strictly the design project, due to the nature of the bi-weekly reflections. The data showed students discussed pains related to exams and CAD modeling. However, these were not of interest in this investigation. After removing them, there were 166 individual challenges left. Some of them might repeat, but they were counted individually, as the total frequency of a challenge indicates its breadth across students and persistence across weeks.

Figure 6 depicts the frequency of each type of challenge mentioned across weeks. First, personal and remote challenges showed similar trends: they were most frequently reported at the beginning, and then dropped significantly after a few weeks, with small fluctuations afterwards. Specifically, students had the most challenges about their personal situations at the beginning of the design process. The frequency of personal challenges dropped for about two thirds after the first four weeks, and fluctuated in a small range. Top personal challenges based on frequency mentioned are: lack of organizational skills, time management, procrastination, hesitation to ask questions, low motivation, and personal communication skills. Similarly, challenges related to remote learning and collaboration had a significant drop by about 75% after the first four weeks, and then flattened. This indicates an adjustment to the online modality during the beginning of the semester.

Secondly, challenges about teams and understanding course content did not have a clear trend. Instead, they seemed to be more correlated with the amount of work and difficulty of the course content. In Weeks 3, 4, and 5, students were assigned with the tasks to define their product, i.e., customer requirements, engineering specifications, and benchmarking. Weeks 9 and 10 were about concept synthesis, selection, and analysis. The course contents of Weeks 9 and 10 were the areas that generated the most questions from students in the process, and required extensive team work. It makes sense that the two types of challenges briefly increased in these weeks. We did not observe notable findings about product-related challenges.

3.4. Detailed Challenges across Datasets (2) and (3)—RQ(2)

Lastly, we are interested in looking at the challenges at a detailed level, and identifying some specific top challenges for team collaboration over a longer period, rather than in just one short activity (such as for Dataset (1)). Datasets (2) and (3) were both about the semester-long team design projects. Therefore, they were combined and keywords for each challenge were further coded in detail. Figure 7 sorts the keywords mentioned from high to low frequency. It shows that “remote” was far ahead of all other keywords, for a total close to 50 times. The second tier included challenges that were mentioned at least 10 times, including “conceptual understanding” and “application”, “unclear communication”, “team collaboration” in general, “personal schedule”, “time management”, “team schedule”, “commitment to schedule”, “procrastination”, “distraction, and “motivation. Low frequency of the remaining keywords of challenges did not mean that they were not important. Instead, the ranking provides a general heuristic about priority to mitigate the challenges when there are limited resources. Comparing Figure 7 to Figure 5 and Figure 6, “remote” was the only keyword that was not further decomposed. Therefore, decomposition of other challenge categories into more detailed levels increased the relative ranking of “remote”. For example, the following quotations were all labeled as “remote,” while they were also further decomposed into more specific categories, e.g., “motivation”, “team schedule”, “asking questions”, and “lack of organization skills”.

Example responses are the following.

Motivation: “the hardest part for me is getting my self-motivated to pay attention in class especially since we are online. It is very easy to get distracted by my phone or other things in my house. I feel I just need to learn how to focus more on the lectures.”

Team schedule: “the most challenging part of learning was scheduling a team meeting for my group to meet alongside of COVID restrictions and members being in other remote locations.”

Asking questions: “the biggest challenge in learning online for me is not having as many opportunities to talk with classmates and talk to instructors to discuss problems and work together to understand”

Lack of organization skills: “the biggest challenge I faced so far is keep track of assignment due dates. Since at the moment all of my classes are online it has been a struggle to keep track of all the due dates of assignment and to make sure that I do not forget about a deadline of an assignment, quiz, or a discussion post.”

3.5. Emotion Dynamics

Emotion dynamics over the entire semester were also checked as supplementary information because emotions could be an indicator of a person’s overall psychological state. Elements that revealed positive and negative emotions in Datasets (s) and (3) were identified and coded. Figure 8 is a plot of the total frequency of negative and positive emotions across weeks in biweekly reflections. It was clear that negative emotions dominated at the beginning, and then positive emotions increased. The process indicated an adjustment at the beginning, and it was highly likely that this could be partially attributed to the new online modality from averse to acceptance. Emotions decoded in the final essays showed 86% positive compared to 14% negative. It signaled the students’ positive experience with the remote team collaboration in the design projects, despite of various challenges they have encountered. The team project was a way for students to develop their problem-solving skills, collaboration, and proficiency with applying the design process. Several students commented that the team projects created more social connections while people had to be physically isolated from each other.

4. Discussion

4.1. RQ1: How Are the Usages of Sketches and Text Descriptions in Idea Generation Related to Team Performance in an Online Environment?

Dataset (1) showed that the high performing teams overall relied more on text instead of sketches in representing design ideas compared to the low performing teams. This was more true in Stage 2, where team members refined and generated more ideas after reviewing others’ ideas. In addition, by examining pair-wise correlations between various factors of teams, we derived two propositions for further investigation: (1) Tools, strategies, or logistics that enhance team online communication, meeting efficiency, or team members’ design enthusiasm would positively contribute to team performance during the collaboration (e.g., team members contribute more ideas and provide more constructive feedback); (2) Improving students’ ability to express ideas with text descriptions (rather than sketches as emphasized traditionally in design education) improve team performance and enhance communication in remote collaboration.

Even though sketching is a fundamental part of engineering design, and has been tested to be correlated with design outcomes [22], its positive effect in online team collaboration is still in doubt with the current technology limitations in education. Our qualitative analysis did show that the teams, which relied more on sketching did not produce more ideas than other groups and reported lower levels of team contribution and meeting efficiency. One explanation is that ambiguity of sketches causes difficulty in communicating design [39], despite that, it leaves space open to build and iterate on designs. In addition, sketching takes longer to describe an idea than words do. Although sketching can be quick and efficient for experienced designers, novices often experience sketch hesitancy [40]. It is possible that specific individuals used sketches and produced fewer ideas resulting in the team’s perception of lack of contribution. Furthermore, the remote environment makes text descriptions quicker and easier regardless of sketching ability. Students using text descriptions would have been able to simply type their idea into the document compared to the more steps needed when adding a picture of a sketch.

Students in the current study were not limited to paper and pencil when sketching ideas. No explicit direction was given for the sketching medium, so most students used paper and pencil, while a few used electronic devices. It would be interesting to explore whether training students using touchscreen devices instead of paper and pencil would change the results. Students using paper and pencil have more steps such as taking a picture of the sketch and uploading it to the computer, which might lead them to choose text more often. Ambiguity in sketches in this design stage also does not lend itself to communication easily. Team members might have struggled to understand the ideas or did not value them as much. Participants are likely creating more “thinking sketches” which helps individuals process and synthesize ideas rather than “talking sketches” that are helpful for communicating ideas to groups [41]. This is not to say that students shouldn’t be sketching their ideas, but that students need more support in learning how to communicate through sketching, for example how to incorporate text or elements of “talking sketches”. Further research on the types of sketches made, how they communicate information to a group compared to text, and the discussions that occur during idea generation is necessary to pinpoint the exact challenges related to sharing design concepts.

As mentioned, the comfort level using text explanations was positively correlated with team communication. This makes sense because most communication among team members was done in a written or verbal format and relied solely on visual communication very little. Verbal and written communication both require words rather than symbols or pictures, therefore if a student can describe an idea in text, the idea is more easily translated into conversation with other group members. Similarly, comfort with text was positively correlated with text explanations in Stage 2. The ambiguity in sketching is thought to be beneficial for iterating designs, but it also might be possible that text lends itself to building off each other’s ideas better. It may be quicker to refer to previous ideas using text than redrawing sketches. Additionally, the number of ideas represented in text in Stage 2 was positively correlated with team communication. One possibility is that the number of text explanations is a result of good communication after the first round of brainstorming, or the clarity in written text relative to sketches allows group members to understand ideas more easily. Sketches are often accompanied by a verbal explanation or discussion about what the image shows, while text or combinations might be more standalone. Discussions about ideas can be productive and positive communication, but participants could have perceived this lack of immediate understanding from sketches as poor communication. In Proposition 2, we suggest to improve student skills in expressing ideas in text. Note that improving skills on text does not dismiss the importance of sketching skills. Both are important. But in current engineering education, we have curriculums focusing on teaching students how to sketch and how to represent ideas visually with CAD tools. Text is less focused, or emphasized as a learning outcome, especially in ideation. Text representations exhibit advantages much more when students are collaborating remotely in idea generation. Thus we propose that educators should also focus on skill improvement of expressing ideas in texts in order to support easy remote collaboration.

4.2. RQ2: What Are the Challenges in Early Design Processes That Remote Teams Have? Are There Any Changes over Time in a Semester?

Datasets (2) and (3) provided rich information about teams’ pains in the remote design collaboration processes. At a higher level, it was found that the most frequently mentioned challenges were team-related, and 22% of them were explicitly attributed to being remote, such as different time zones and technology limitations for remote collaboration. The second category was understanding course content, and 4% out of these challenges were explicitly attributed to the online learning environment. The third category was explicitly about being remote, such as being stuck in the house, long time in front of the computer as reported in Section 3.2. In addition, the challenges showed a dynamic change over time, especially that personal and remote challenges had a significant drop after the first few weeks, together with the shift of emotions from negative dominated to more positive. The emotion changes echoed the challenges’ change, and it signaled the students’ positive experience with the remote team collaboration in the design projects, despite of various challenges they have encountered. Since some students felt that the team project helped them create social connections, more frequent groupwork could be used to address the lack of social opportunity in a remote environment.

The low frequency of remote-only challenges combined with the drop in remote challenges across the semester suggests that, beyond the initial adjustment to online learning (in the first four weeks), there were few new challenges created by the online environment, which instead might exacerbate existing team problems. This assessment of team problems is inconsistent with the work of Rooney and Scott, who reported that peer evaluations of teamwork outcomes during emergency remote learning did not differ from traditional face-to-face learning [42]. These differences could be due to that Rooney and Scott observed a course that began in person and transitioned online, providing students with the opportunity to build camaraderie and address some team challenges in person [42]. This could mean that a hybrid format (combined face-to-face and remote) might maintain the flexibility of remote group work but mitigate some of the team related issues exacerbated by the remote conditions. Or additionally, incorporating designated team building early on, in person or online, could provide some more support and enhance team collaboration.

There is an obvious difference between Datasets (2) and (3) regarding the frequency of personal challenges reported. This disparity is likely due to the format of the reflection. Dataset (3) from the biweekly reflections were meant to facilitate discussions between peers and allowed students to be much more open as opposed to the formality of Dataset (2) from the essay, which was read and graded by the instructor. The differences between the frequency of personal challenges in each type of reflection support Krutka’s assertion that collaborative reflection builds a sense of community and support among classmates, eventually leading to more meaningful and personal interactions [43]. This is also the purpose of the biweekly reflections on the discussion board: cultivate an open and supportive online environment for communication. While students built a community through their discussion posts, it might be worth looking deeper into how those interactions impact the inner workings of each team or how to use discussion board as a support tool to establish a better sense of openness and rapport within teams. The evolution of personal challenges throughout the semester is of interest as it, along with the remote challenges, reflects the transition into the online modality and the semester as a whole. Students had to get reacclimated to the school environment after the long summer and reported problems like procrastination, poor sleep schedules, planning, and motivation. These challenges overlap with Gillis’s work, which identified difficulty sleeping, lack of motivation, distractions, and fewer opportunities for peer collaboration or asking questions as academic barriers during the COVID-19 pandemic [44]. This suggests students need additional assistance learning how to deal with personal challenges and establishing good habits at the beginning of the semester, while more assistance is needed with course content and team collaboration throughout difficult periods of the semester. In a hybrid format, this could be improved by teaching difficult course content in person, where students self-report being more focused and emphasizing study or time management strategies and other student resources available at the university like tutoring or academic coaching at the start of the semester.

Unclear communication was the most frequently reported problem relating to teamwork. This commonality matches Lingard’s study, in which students perceived themselves to be poor at communication [28]. Mostafapour’s claim that students might not be aware of how poor communication initiates challenges further on may still hold true even though this data contradicts her findings that students perceived themselves as highly effective at communication [18]. Communication has proved to be a major problem for students remotely and in person, although written communication is similar in both modalities, verbal communication takes different forms for each class format. It is worth more research to determine which specific areas cause problems for remote teams compared to face-to-face teams, so instructors can address these issues more precisely. Taking a deeper dive into the general team collaboration category, challenges included several scenarios where students were unable to reconcile different ideas about the project through discussion. Other less frequent challenges such as organization, time management, and scheduling could also be worsened due to inadequate communication. Time management and communication were among the second tier in terms of frequency along with motivation. Gillis also identified motivation as a barrier during the COVID-19 pandemic, but the combination of these three factors might reflect Demirdag’s assertion that students’ communication and time management skills are predictors for motivation [44,45].

4.3. Limitations

While the study helps understanding students’ challenges in remote design collaboration in the background of a global pandemic, there are limitations when interpreting the findings. First, the sample is limited in terms of its size and representativeness. The data collected is from a second-year engineering class in one university in the U.S. It is possible that students in other regions or at other levels have a different pattern of challenges and strategies. Secondly, the three datasets are all self-reported data, except for the design ideas generated. Biases are unavoidable. It is possible that students were conservative in reporting their challenges, rating their team communication, contribution, or lack of motivation to be comprehensive in their reflections. To address these limitations, it is suggested to involve students at different levels and different regions in future. In addition, appropriate incentives could be designed to help improve students’ motivation and openness in their responses. Bias due to the priming effect could be reduced by restricting the students viewing others’ posts until they post their own. In Dataset (1), The educational goal of the in-class activity was not to generate high-quality designs but rather practice generating ideas and iterating on them as a group. The aim was to get comfortable and understand the process before they were required to do it as a part of the semester long project. Future work could focus on multiple metrics such as quantity, quality, feasibility, and novelty [46] when determining team design performance. Third, the datasets only include deliverables, no information is known about the conversations that occurred during the activities or semester. Future studies could record video and audio of the team working process, and dive deeper to understand the team collaboration behaviors with protocol analysis [47,48]. In addition, the familiarity between the team members are not captured in the study. It is possible that familiarity variances exist across teams. But student teams in Dataset (1) were randomly assigned, and teams in Datasets (2) and (3) were assigned by the instructor at the beginning with the diversity criteria. There should be minimum systematic bias due to familiarity. Similarly, dataset one includes a post-activity survey, but it only captures students’ perceptions of team performance rather than an evaluation of their behavior during the activity or a comparison between groups. Future studies could also be expanded to address potential opportunities that were brought up by the pandemic. For example, the determination of what components of remote collaboration in design can be translated into a post-pandemic teamwork experience and how to best leverage the advantages of both remote and in-person collaboration modes in post-pandemic engineering education could be explored. This could be informed by a direct comparison of challenges that are specifically related to remote challenges versus those are related to in-person challenges.

4.4. Implications for Remote Design Collaborations

Several implications will be highlighted based on the observations and discussions above. Firstly, it is important to consider how remote teams are communicating during design idea generation. For example, if representing ideas in text or sketches leads to good communication through discussion, or if teams perceive this lack of immediate understanding as unclear or poor idea communication. Besides improving engineering students’ sketching abilities as emphasized traditionally, educators should also purposefully train students’ ability to express design ideas clearly in words, which is especially beneficial in remote collaboration environments. The reported negatives and challenges in using sketches in remote team ideation call for new technologies and tools that support easy digital sketching and sharing of ideas.

Secondly, the trend of remote-related challenges suggests a potential to enhance remote team collaboration in design, such as more support in early stages by adding team building activities, or adding some in-person communication if possible, instead of jumping into design projects directly. This would reduce the challenges related to unfamiliar teammates as reported in Dataset (1). Demonstrations of how to use online tools and extra time for practicing the usage could also help team communication and provide a good starting point for team collaboration.

Thirdly, students may need additional support at the beginning of the semester to deal with their personal challenges and build good habits, whereas during difficult content areas throughout the semester, they might need additional support regarding team collaboration and course content. Different data trends between the biweekly reflection and final essay reflections also suggests that regular reflections in a shared space helps student talk more about their personal issues, and it is an effective way to capture their learning dynamics while building a supportive community among students.

Fourthly, unclear communication was the most frequently reported problem relating to teamwork. The observations on communication-related team challenges suggest the urgency of building a mechanism to cultivate better communication among team members from multiple folds for successful remote collaboration, such as team decision making, meeting schedules, efficient use of time, and organization.

Lastly, as the weeks passed, there were trends within the reflections of repetition of challenges and adjustments. Students repeating the same adjustments that they planned to make implies that they were not able to make the desired changes within the two-week period, which indicates external help might be needed in that area. More systematic support is needed to help students build a habit for planning and enhance accountability to their plan. In the meantime, students were aware of the importance of asking questions, while remote communication is a barrier. This issue raises a valuable question regarding how educators, as well as the technology development industry, can leverage or improve technology to reduce the communication barriers, both technically and psychologically.

5. Conclusions

Overall, this paper explores challenges that students encountered during the remote collaborations on projects of early-stage engineering design and identified possible future research propositions and implications for improving remote collaboration in engineering design education. Findings were extracted from three datasets, with one particularly focusing on idea generation (which is a big part of early-stage design) and also the most concerning aspect of remote collaboration. The other two are regular reflections at different time points over the semester. Unlike the classic findings that sketches improve performance, we found significant positive correlations between teamwork experience (e.g., communication, efficiency, perceived contribution) and number of ideas expressed in text, and the significant negative correlations between teamwork experience and number of ideas expressed in a combination of sketches and texts. In other words, heavily relying on sketches in remote team ideation was related with lower team contribution, less efficient meetings, lower design enthusiasm, worse team communication, and more reported challenges. Therefore, we propose that educators should also work on improving students’ ability to express design ideas with words, not only traditionally emphasized visual representations. Written text descriptions have their own merit (e.g., easy to be implemented, externalized, and shared) in online environment considering the limitations of the technologies that facilitate remote collaboration. In addition, the significant positive pairwise correlations among team contribution, design enthusiasm, communication, meeting efficiency suggest that it is valuable to further investigate tools, strategies, or logistics that enhance team remote communication, design enthusiasm, and meeting efficiency, and to test how these affect team performances. On the other hand, the highly reported challenges were about working in teams and understanding course content, but only a lower percentage of the challenges in these two categories were directly attributed to being remote, such as different time zones and technology limitations for remote collaboration. Challenges that were explicitly mentioned being remote were not specific to the design class, include missing the face-to-face communication, difficulty adjusting to online classes, being stuck in the house, long time in front of the computer, and staying mentally motivated. In addition, the remote-related challenges overall dropped dramatically after the first few weeks and then remained steady. The challenges about remote and their changing patterns indicate large potentials to improve remote design collaboration.