Keywords

1 Introduction

In organizations, feedback can have a major impact on the quality of the employees’ performance. Therefore, it is important that accurate and relevant feedback is provided to the organization’s leaders. Because of the hierarchical structure of organizations, feedback is usually provided only by the leaders’ own managers, which is, of course, a limited perspective of the leaders’ effectiveness. A valuable tool for delivering such feedback is multisource feedback (MSF; also known as 360-degree feedback). In recent times, the growth of MSF has been a significant trend in the leadership development field. Since its inception in the late 1980s, MSF has gained increasing acceptance and importance in organizations (Silzer & Church, 2009).

In the organizational context, feedback is defined as information provided to employees related to their behavior on the job and the impact of that behavior on others (Fleenor & Taylor, 2019). Feedback is intended to strengthen desired behaviors and to recommend changes for undesired behaviors. Under the correct conditions, feedback can be a catalyst for change (Fleenor et al., 2020).

Most employees want to know how well they are doing their jobs. If they do not receive sufficient feedback, they often seek it on their own (Fleenor et al., 2020). Receiving useful feedback is an important motivational factor that can lead to increased job satisfaction (Bracken & Rose, 2011). Feedback can enhance self-awareness by identifying strengths and can facilitate growth by highlighting areas for improvement (Nowack, 2019).

The impact of multisource feedback can be significant if it is embedded in a larger leadership development process. That is, if it is fully integrated into the human resource management (HRM) system of the organization. Research has found that MSF can improve performance and lead to behavior change over time (Smither et al., 2005; Walker & Smither, 1999). The implementation of MSF has been shown to improve the financial performance of organizations through increased knowledge sharing and employee effectiveness (Kim et al., 2016).

For student-to-teacher feedback in schools, the most relevant counterpart to MSF in organizations is upward feedback. In upward feedback, ratings are solicited from the direct reports of the leader being assessed. This is a relatively common practice in organizations because direct reports are thought to be in the best position to judge a leader’s effectiveness. The same could be said for the students of the teachers in schools. However, it would also be possible to conduct the “full circle” of feedback for teachers by including self-ratings and ratings from colleagues and headteachers or principals.

There are parallels between being a leader in an organization and a teacher in a school. For much of the discussion in this chapter, “teacher” could be substituted for “leader” and “student” could be substituted for “rater.”

2 Multisource Feedback

The purpose of multisource feedback is to provide accurate and useful feedback related to the effectiveness of leaders in their organizations (Fleenor & Brutus, 2001). This process includes collecting and reporting coworkers’ ratings of a leader’s effectiveness and providing feedback and coaching for each leader. Traditionally in organizations, feedback has come from a single source, the manager, which provides only a limited perspective of a leader’s effectiveness. With MSF, the assessment of a leader’s strengths and development needs is more reliable and valid. Because it uses multiple raters, MSF provides different perspectives of performance, making the feedback more accurate and useful to the leader. Additionally, the collection of feedback from several raters with different relationships to the leader will decrease the effects of the biases of the individual raters on the ratings.

There is little agreement in the literature on the terminology used in multisource feedback (Fleenor et al., 2020). In this chapter, the individual being assessed is referred to as the leader. Coworkers who provide the feedback are called raters, and usually include peers and direct reports. The leader’s direct boss is referred to as the manager, who also provides feedback. The MSF survey that is completed by the raters is called the assessment. The scales on an MSF assessment represent leadership competencies that are important for success in the organization. MSF is sometimes used with employees who are not leaders; however, in that case, there are no direct-report raters.

2.1 The Multisource Feedback Process

Most MSF processes have the following features (Fleenor & Taylor, 2019):

  • Multiple raters (manager, peers, direct reports) provide ratings of the leader’s effectiveness using a quantitative rating scale. Leaders also provide self-ratings. The ratings are collected anonymously and reported in the aggregate; therefore, the leader does not know who provided specific ratings. Because most leaders have only one direct manager, the anonymity of the manager’s ratings usually cannot be maintained.

  • A report is provided to leaders that summarizes the results of their feedback. In a feedback session, leaders identify their strengths and development needs (weaknesses) and examine differences between their own and others’ ratings of their effectiveness.

  • Based on this feedback, leaders work with feedback coaches (or their managers) to develop an action plan to improve their effectiveness.

Typically, in an MSF process, the leader selects a number of coworkers to participate in the feedback process. Working individually, the raters and the leader complete surveys designed to collect information about the leader’s specific skills, behaviors, and other attributes that are important for leader effectiveness. Leader effectiveness is defined as performance that makes leaders successful in their organizations (e.g., the leader’s team successfully meets its goals for the year; Fleenor et al., 2020).

After raters complete the surveys, their ratings are electronically sent to a centralized location for scoring. A report is produced and delivered to a feedback coach, who then meets with the leader to review the report. The coach can be an internal human resource (HR) professional or the leader’s manager who is trained to interpret the results of the assessment and assist the leader in understanding the report. The coach helps the leader use the feedback to create a plan to address developmental needs identified by the feedback.

Multisource feedback provides a structured means of collecting and processing data, and an opportunity to reflect on this valuable information. It may be the only opportunity some leaders have to consciously self-reflect on their effectiveness. MSF systems also guarantee the anonymity of the raters. There is evidence that anonymous feedback is more honest than open feedback (Kozlowski et al., 1998). This appears to be particularly true when direct reports are rating their leaders. A climate of trust must be created for the MSF process—when anonymity is ensured, the feedback will be more accurate. If raters believe that anonymity was violated, then less honesty can be expected in future MSF administrations, with a corresponding loss of reliability and validity (London & Wohlers, 1991). Anonymity differs from confidentiality. Confidentiality requires that access to MSF data be limited to individuals who are permitted to see the data in accordance with organizational policy. Confidentiality is important to ensure the participants that their data are protected and will not be seen by unauthorized individuals in the organization. A lack of confidentiality may result in lower participation rates in future MSF administrations.

2.2 Using Multisource Feedback for Leader Development

Because of its structure, thoroughness, and anonymity, MSF is likely to be accepted and acted on by the leaders receiving the feedback (Atwater et al., 2007). To ensure the effectiveness of MSF, it should be implemented within a broader leadership development context. For example, MSF should be integrated into the organization’s leader development and succession planning systems to help identify how leaders can become more effective in their organizations. The organization’s leadership development system is responsible for providing activities, such as MSF, that will increase the effectiveness of its leaders. The succession planning system is responsible for creating a pipeline of leadership talent for the future. The integration of the leader development and the succession planning systems should create conditions that allow leaders to receive ongoing feedback along with new job assignments, thus increasing their current competencies (McCauley & Brutus, 2019).

Many organizations use MSF as an integral part of development processes for individual leaders. Even when leaders have good insights about their own strengths and development needs, they may not be fully aware of how their behaviors affect their coworkers (Fleenor et al., 2010). After they receive the results of their MSF assessment, leaders have a clearer idea of how their behaviors consistently affect others.

In addition to its use in developing individual leaders, some organizations use aggregated MSF data to determine group strengths and weaknesses for needs analysis purposes. Furthermore, the process of responding to the assessment underscores desired behaviors and creates discussion of which behaviors are valued throughout the organization. This occurs because the items on the MSF assessment indicate what leadership behaviors are considered important by the organization (Bracken & Rotolo, 2019).

2.3 Characteristics of Multisource Feedback

The characteristics of MSF can be thought of as the interactive product of both the assessment and the raters (Bracken & Rose, 2011). According to Bracken and Rotolo (2019), the most important characteristics of MSF are: (a) awareness of the feedback (including reactions and receptivity); (b) acceptance of the feedback; and (c) accountability for acting on the feedback. These characteristics are important for ensuring MSF will result in desired behavior change in the focal leaders (Bracken et al., 2001). Each of these characteristics is discussed below:

2.3.1 Awareness of the Feedback

Awareness involves bringing the information to the attention of the leaders. Thus, they must be aware of the feedback before they can act on it. Awareness of their feedback is required before leaders will recognize their weaknesses and take action to correct them. Awareness of the feedback includes reactions and receptivity to the feedback by the recipients. Reactions can range from being pleased with the feedback to feeling hurt and resentment. A leader’s health and psychological well-being may be negatively affected by receiving unfavorable feedback (Nowack, 2019). Feedback coaches play an important role in helping leaders work through any emotional reactions (Fleenor et al., 2020).

Receptivity relates to a leader’s psychological readiness to receive the feedback. It is positively related to both, emotional intelligence and perceptions of the feedback environment (Dahling et al., 2012). Additionally, research indicates that feedback orientation, which is the degree to which a leader is ready to receive the feedback, can predict the leader’s emotional reactions to their feedback (Braddy et al., 2013).

2.3.2 Acceptance of the Feedback

Acceptance is the leaders’ belief that the feedback is an accurate description of their behavior (Ilgen et al., 1979). A key event occurs when the leader decides to accept the feedback as valid and useful information. For the feedback to be accepted, a leader must be aware of and receptive to it. When the feedback is not accepted, no behavior change will result (Bracken & Rose, 2011). First-time MSF participants may experience shock, anger, and rejection of the feedback before finally accepting it (Brett & Atwater, 2001). To ensure acceptance, resources for assisting leaders in dealing with their feedback should be provided by the organization (e.g., coaches, workshops, developmental activities, etc.; Fleenor et al., 2020).

2.3.3 Accountability for Acting on the Feedback

Accountability for acting on the feedback is necessary for a sustainable MSF process. This requires organizations to ensure leaders will conduct improvement-oriented actions on their feedback. Methods for ensuring accountability include the full support of the leader’s manager for the MSF process and providing access to developmental resources such as new job assignments and training (London, 2003). Accountability is the major component for moving from acceptance to improved leader effectiveness (Bracken & Rotolo, 2019).

A successful MSF process requires full accountability, not only from the leaders, but also from other groups involved, namely, raters, managers, and the organization (London et al., 1997). If raters believe leaders are not being held accountable for acting on their feedback, they will be less likely to provide effective feedback in future MSF administrations. On the other hand, when raters see their feedback is being used productively, they can be expected to continue to provide accurate, honest feedback (Bracken & Rotolo, 2019).

3 Reliability and Validity of Multisource Feedback

There are a number of factors that affect the validity of an MSF implementation (Bracken et al., 2001). These factors are directly related to the characteristics of a successful MSF process (Bracken & Rotolo, 2019). For MSF, the conceptualization of validity (e.g., content, construct, and criterion-related validity) is more complex than traditional notions of validity that arose from controlled, standardized settings such as intelligence testing. In those settings, validity was determined by a single measurement event in which an individual responds to the items on an assessment (i.e., single-source data). MSF depends on the collection of data from potentially unreliable sources (i.e., multiple raters). It is a complex process with the characteristics of both psychometric testing and large-scale data collection (Fleenor, 2019).

3.1 Validity Factors in Multisource Feedback

The primary factors that affect the validity of MSF are described below and summarized in Table 1 with design recommendations from Bracken et al. (2001):

Table 1 MSF Validity Factors with Design Recommendations
Full size table

3.1.1 Alignment

This is the traditional definition of content validity— the extent to which the feedback (e.g., competencies, behaviors) is important for success in the organization (Bracken & Rotolo, 2019). If the competencies being measured are not related to success, then the content validity of the process is deficient. Alignment occurs when the values and goals of the organization are translated into a set of competencies for the entire organization (Campion et al., 2019).

3.1.2 Accuracy

This includes the process of accurately collecting and processing data, and reporting the feedback. Errors in the feedback reports can negatively affect leaders’ confidence in the process. Considerations for increasing accuracy include scoring systems with the capacity to handle high volumes of data with secure reporting, quality control to eliminate errors, and pre-populating of demographic data.

3.1.3 Clarity

Raters must be given instructions on how to correctly complete the assessment and return it on time. Errors typically made by raters include miscoding the person they are rating, misusing the response scale, and providing inappropriate write-in comments. To increase clarity, orientation sessions should be held with the raters to increase their understanding of the process.

3.1.4 Cooperation

The quality of MSF depends on the willingness of the raters to fully participate and provide reliable responses. Design features that affect this factor are related to the magnitude of the task, such as instrument length and the number of surveys a rater must complete. Indicators of low cooperation include unreturned or incomplete surveys and the effects of rater fatigue on the feedback. A simple metric to evaluate cooperation is the overall organization-wide response rate. If less than 75% of the surveys are completed, this should be the reason for concern (Bracken et al., 2001).

3.1.5 Timeliness

Timeliness in providing feedback is an important factor to ensure acceptance of the feedback by the participant. Delays in providing results coupled with recency effects in the ratings may result in feedback that is no longer valid. This can have implications for how effective the feedback is in addressing the needs of the participant and the organization (Bracken et al., 2001).

3.1.6 Reliability

In this context, reliability refers to how dependably or consistently MSF measures the competencies on the assessment. This factor includes the importance of reliability in MSF, how it should be measured, and what level of reliability is acceptable (see Pulakos & Rose, 2019).

Some commonly-used reliability indices may not be appropriate for MSF ratings. For example, test–retest reliabilities may be affected by changes in the raters themselves (e.g., attitudes and opportunity to observe). The raters at Time 1 are often different than the raters at Time 2—with less than a 75% overlap in raters, the results can be misleading. Therefore, it is not recommended that test–retest reliability be used with MSF (Bracken et al., 2001). Internal consistency reliability (e.g., coefficient alpha) provides evidence that items on a scale (i.e., dimension or competency) are internally reliable. Often poorly written items negatively affect the internal consistency reliability of MSF ratings. The use of double and triple-barreled items in an attempt to shorten the length of surveys can also reduce the reliabilities. Overall, low reliabilities can obscure the meaningful interpretation of the feedback (Fleenor, 2019). Other factors that affect the internal consistency of MSF ratings include the misinterpretation of the rating scale by the raters. Typically, 5-to-7 point Likert scales with clearly defined anchors are recommended (Bracken & Rotolo, 2019).

Interrater reliability is used to determine the agreement within rater groups. Moderate levels of interrater reliability within these groups have been reported (e.g., Brett & Atwater, 2001). However, direct report ratings are often found to have the lowest reliabilities (Braddy et al., 2014). To increase the reliabilities within rater groups, all eligible raters should be used, particularly all direct reports. In general, more raters will result in more reliable ratings (Fleenor, 2019).

Typically, the correlation between the ratings from the various rater groups has been found to be low (Tornow, 1993). However, the reason for conducting MSF is to bring different perspectives of a leader’s performance to the process. While the rater groups may disagree, each group may have a valid perspective of a leader’s performance because leaders often interact differently with the various groups. For example, a leader may be interpersonally warm with peers, but cold and distant with direct reports.

3.1.7 Insight

Leaders should be provided with the necessary amount of information needed to take actions that are aligned with their feedback. The format of the assessment and feedback report should be designed to maximize participants’ understanding of their results.

Feedback should be provided at the item level—not just at the competency (i.e., scale) level. With item-level feedback, leaders have a basis for determining the specific behaviors (i.e., the items) that resulted in their ratings. Processes that collect written comments cannot be expected to replace item-level feedback and may even increase the burden on the raters by requiring them to provide detailed descriptions of a leader’s behavior. This may result in “rater fatigue,” especially when raters are required to complete MSF assessments for several employees (Rose et al., 2004).

3.2 Self-other Rating Agreement in Multisource Feedback

Often with MSF, self-ratings are found to differ significantly from the ratings of others (Fleenor et al., 2010). For example, individuals with high self-esteem may over-rate themselves relative to others’ ratings of them. For this reason, the use of self-ratings alone is not recommended. However, the level of agreement between self-and others’ ratings, can provide important and useful information (Furnham, 2019). There appears to be a relationship between self-other agreement and leader effectiveness. In general, leaders who rate themselves similarly to others (in-agreement raters) appear to be more effective than leaders who rate themselves differently (Fleenor et al., 2010). However, the relationship between self-other rating agreement and leader effectiveness is non-linear. For example, leaders who under-rate themselves appear to be more effective than those who over-rate themselves (Braddy et al., 2014). Similarly, teachers who under-rate themselves are likely to be more effective than teachers who over-rate themselves.

For MSF, the challenge is to develop a relatively simple index of self-other rating agreement that participants can easily understand in their feedback reports. For example, such an index would categorize a leader as an under-rater, in-agreement rater, or over-rater.

4 Recommendations for Facilitating Multisource Feedback

Best practices suggest that a confidential one-on-one feedback session be conducted between the leader and a coach. The coach provides an introduction to the MSF assessment, an analysis of the individual’s feedback, and assists with developmental planning. These sessions are particularly important for leaders receiving feedback for the first time. They usually appreciate discussing their feedback with an experienced coach. The coach helps the participant understand that conflicting ratings may be valid, and comparisons between the different rating sources are important (Fleenor et al., 2020).

Leaders must be given adequate time to process their feedback before the one-on-one session. Unfortunately, some organizations distribute the reports and allow the leaders only a few minutes to look over their results prior their feedback session. Without time to reflect on their report and process any immediate emotional reactions to the data, leaders may not be ready to fully accept the feedback.

The coach should prepare for the session in advance by thoroughly reviewing the feedback report. The session should be held in a private room and leaders should be given the opportunity to audio record their session, which will serve as a useful resource for participants to review progress on their development plans.

5 Why Multisource Feedback Processes Fail

While best practices are fully documented in the literature (Fleenor et al., 2020), practitioners continue to struggle with implementation issues with MSF. These issues affect the quality of the feedback (e.g., validity, reliability, accuracy) and therefore the future success of MSF processes. Many of these issues can be avoided by careful design, planning, and follow-up. A number of problems are common to failed MSF implementations (Fleenor et al., 2020):

  1. 1.

    Unclear Purpose When the business reasons for conducting MSF are unclear or key stakeholders disagree on its purpose, the process is likely to fail. Organizations need to consider how their business goals align with the goals of the MSF implementation (Campion et al., 2019). The purpose of the process should be clearly defined, and an appropriate MSF assessment selected for that purpose.

  2. 2.

    Lack of Organizational Readiness A supportive organization culture is critical to the success of an MSF process. There must be full senior management buy in and public support. All senior leaders should participate fully in the MSF process. Further, a high level of trust is needed among raters so that the feedback they provide will be used constructively by the leader and the organization (Smith & Fortunato, 2008).

  3. 3.

    Selecting the Wrong MSF Assessment The organization should have an underlying leadership competency model indicating what is important for success in their organizational context. If the purpose of MSF is to measure competencies specific to an organization (rather than general leader competencies), then a customized assessment will be needed that directly measures these competencies (Conger, 2019).

  4. 4.

    Poor Design and Logistics Reasons MSF processes fail often include inadequate planning and poorly implemented logistics. For example, if MSF is administered during an extremely busy time in the organization (e.g., during the budgeting cycle), it may result in lower participation levels. A thorough communication plan is critical, particularly for those directly involved in the process, including leaders, their managers, and all other raters. Some organizations try to compress the MSF process into an unrealistic timeframe, which results in poorly implemented processes.

  5. 5.

    Leader Preparation An appropriate amount of preparation for leaders is critical. They need to be informed why they are participating, how the process works (e.g., rater selection), and the level of confidentiality and anonymity they can expect. Raters need to be told that their input is important, and their ratings will be strictly anonymous.

  6. 6.

    Poor Rater Selection Employees who provide the most accurate ratings are those who interact with the leader on a frequent basis. This allows enough time to observe the behaviors they are rating. For most leaders, the best raters are the coworkers with whom they have frequent face-to-face interactions (Bracken & Rotolo, 2019). Selecting raters who are not fully aware of a leader’s behaviors will result in less valid feedback.

  7. 7.

    Post-Assessment Problems Some issues do not become problems until after the MSF assessment has been completed (Bracken et al., 2001). A common issue is the lack of clear expectations of what leaders are responsible for doing after they receive the feedback. They should meet with their managers to discuss their feedback, create a development plan and decide on the next steps. When this is not accomplished, leaders are less likely to be accountable for acting on their feedback.

  8. 8.

    Confidentiality and Anonymity Issues Confidentiality and anonymity are critical issues in the MSF process (Macey & Barbara, 2019). There can be serious issues if rater anonymity is compromised during the process. Raters are more likely to provide valid ratings when they know their individual ratings will remain anonymous and confidential.

  9. 9.

    Failure to Evaluate the MSF Process As with any leader development process, it is important to assess the impact of MSF. An evaluation should include interviews, surveys, or focus groups with the participants to determine how the organization can improve its MSF process to increase its impact.

6 The Transferability of Multisource Feedback Research to Student-to-Teacher Feedback in Schools

There seems to be considerable overlap between leading in organizations and teaching in schools. Schools themselves are, of course, organizations with particular hierarchies and cultures relevant to the educational context. Teachers could be considered as the “leaders” of the students, and the students as the “direct reports” of the teachers. As discussed previously, the most relevant organizational counterpart to student-to-teacher feedback is upward feedback. In upward feedback, ratings are solicited from the direct reports of the leader being assessed because they are thought to be in the best position to judge the leader’s effectiveness. In the same vein, students are probably in the best position to judge a teachers’ effectiveness. The upward feedback model could be expanded to include self-ratings and the ratings of others (peers and leaders such as lead teachers or principals). This would result in full “360-degree” feedback, which may provide more valid and reliable feedback for teachers.

For teachers, MSF would provide access to structured feedback from students on their teaching quality, a source of feedback they rarely receive. Given that students have a unique perspective of teacher effectiveness, such feedback could be helpful for teachers who want to improve their teaching effectiveness. Administering MSF in the classroom has an advantage over the typical organization because teachers are likely to have mostly the same students during the school year. In organizations, a leader’s direct reports may change frequently because of reorganizations, reassignments, and turnover. Because of the relatively stable rater population, teachers can be more easily evaluated over time to determine how much they have improved.

If administered as a full 360-degree feedback model, MSF would provide teachers with multiple perspectives of their performance. They would be able to compare their ratings from various sources (students, peers, principals) to determine if they are perceived differently by these groups. Teachers would be able to compare their self-ratings to the ratings of others to see if they have an unrealistic view of their own performance.

Following the feedback facilitation model typically used in organizations, lead teachers or principals could act as coaches for the teachers and assist them with digesting their feedback and developing plans for acting on the feedback. Teachers would create development plans for improving in areas of weaknesses and leveraging their strengths. Additionally, the use of a validated teacher competency model would inform them of what capabilities are needed to be an effective teacher. In schools, MSF should be administered on a regular basis so the teachers’ performance can be evaluated over time.

There is some evidence, however, that providing MSF alone may not result in sustained behavior change for teachers. For example, Bijlsma et al. (2019) found that teachers did not improve their teaching quality in response to student feedback received via smartphones. As a result of the student feedback, however, the teachers did gain more insight into how they could improve and reported improvement-oriented efforts in response to the feedback. For student feedback to be more effective in creating sustained behavior change in teachers, Bijlsma et al. recommend that:

  • The teachers have a strong improvement motivation and are willing to step out of their comfort zones and search for their weaknesses.

  • Definitions of desired behaviors, improvement goals, and developmental activities are clearly defined.

  • A coach is provided who understands effective teaching behaviors (e.g. quality classroom management), how these behaviors can be developed, and practices that are effective if problems arise during the development process.

The above recommendations align closely with the characteristics of effective MSF in organizations. The basic tenets of MSF in organizations, therefore, should also be applied to student-to-teacher feedback:

  • It should not be implemented as a stand-alone event. In addition to the assessment, there must be developmental planning and follow-up. A plan should be created that details recommendations to help the teacher improve based on the feedback.

  • The feedback assessment must reflect competencies that are important for teacher effectiveness. A fully validated teacher competency model should be used.

  • The support of the top leadership of the school is critical for persuading teachers to set specific development goals. Teachers must be held accountable for acting on their feedback.

  • A flawed feedback process can be fatal to future administrations. The anonymity of the students’ ratings and the confidentiality of the teachers’ feedback reports must be strictly maintained. Students must be convinced that their teachers will not see their individual ratings. This means that the feedback should be aggregated for each class before presenting it to the teachers.

  • The timing of the feedback process should consider organizational realities that could reduce its impact. For example, the process should not be implemented during end-of-semester grading periods or other times when teachers are not able to fully focus on their feedback.

  • Students should be trained to be aware that only constructive feedback will help their teachers improve. MSF is not a way of venting frustration, but meant to catalyze a learning process that will benefit both students and teachers.

In summary, if the recommendations discussed in this chapter are closely followed, then student–teacher feedback is more likely to be successful in schools.