Introduction

This paper describes the development and quality of the questionnaire on supervisor–doctoral student interaction (QSDI). The QSDI is aimed at gathering information about doctoral students’ perceptions of the interpersonal style of their supervisor. The results of this questionnaire can be useful for giving detailed feedback to doctoral student supervisors on their interpersonal style towards a particular student and for research on this relationship.

In most research universities in the Anglo-Saxon countries and counties like the Netherlands, PhD candidates do a research study under the supervision of one or two faculty members. These faculty members not only guide and support the PhD candidate, but also play an important role in the assessment of the quality of the final manuscript submitted. Heath (2002) argues that the success of the PhD system heavily depends on the supervisors, who must provide the time, expertise and support to foster the candidate’s research skills and attitudes, and to ensure the production of a thesis of acceptable standard. Importantly he concludes from analyses of PhD students’ views on supervision that, although the frequency of meetings between supervisor and candidate is essential, the quality of these meetings is even more (cf. Li and Seale 2007). Unfortunately, however, there seems to be more research on the frequency of contact than on its quality (e.g., Pearson 1996).

Research indicates that the supervisor–doctoral student interpersonal relationship is important for the success of a PhD-project (Golde 2000; Kam 1997; Marsh et al. 2002; McAlpine and Norton 2006). Ives and Rowley (2005) for example reported that good interpersonal working relationships between supervisors and their PhD students were associated with good progress and student satisfaction. Studies of mentoring showed that in particular the psychosocial aspect of mentoring was connected to the protégé’s sense of competence, confidence and role effectiveness (Luna and Cullen 1998; Paglis et al. 2006). Denicolo (2004) reports that in the eyes of PhD students positive attributes of supervisors are amongst others reliable, confidence in the student, encouraging, knowledgeable, informative, and sharing. Supervisors should have listening skills, encourage argument and debate, provide continuous feedback and support, be enthusiastic, and show warmth and understanding. Seagram et al. (1998) showed that important positive characteristics of supervisors according to their doctoral students were professional, pleasant, and supportive behavior.

Problems in the supervisory relationship

Several problems in the supervisor–doctoral candidate relationship may emerge; here we list a few.

First, a certain tension might exist between the supportive helping role of the supervisor and the requirements of the role to warrant dissertation quality. Murphy et al. (2007) refer to this double role of assessor and guide. Hockey (1996, p. 363) cites Rapoport (1989): “… the significance of the relationship stems from its duality; the co-existence of intimacy, care and personal commitment on the one hand, and commitment to specific academic goals on the other”. Holligan (2005) analyses this tension in a case study on the conflicting demands put on the supervisor by the research production requirements of an institution versus the support of the PhD student’s autonomy and independence.

Second, a supervisory style that is apt for a particular student could be at odds with the preferred style of the supervisor, or the style he or she is competent to provide. An example of a broadly found distinction in style is what Sinclair (2004) calls a “hands on” approach that is relatively interventionist and a “hands off” approach that leaves candidates to their own devices. The candidates’ needs for one of these approaches may depend on the phase of their project. The ideal mentor scale of Rose (2003) has among others been developed to help resolve this problem. The problem of misalignment of supervisor’s and candidate’s style could also be found in the orientation towards supervision, for example in the field of task or person orientation such as distinguished by Murphy et al. (2007).

Finally, in many institutions it is not common to evaluate supervisory experience or discuss among staff how supervision is (or should be) provided. Nonetheless, such discussions might be profitable for the quality of the PhD students’ work. Leonard et al. (2006) conclude in a review of the literature on the impact of the working context and support of postgraduate research students that several studies show a need for supervisors to be more aware of the way in which their relationship with a student is developing. Being unaware of the development of the supervisory relationship, both at the part of the supervisor and the candidate, may be a major threat for the development of supervisory trajectories into a productive direction.

Perceptions and evaluations

With the QSDI one can collect data on doctoral candidates’ perceptions of the relationship with their supervisor. Although instruments to collect such data do not exist, data on students’ perceptions of learning environments have been used extensively in educational research and in professional development activities both in secondary and tertiary education. Several reviews of the validity of students’ evaluation of the effectiveness of instruction in universities have been carried out. Composite judgements of students display high validity and reliability (d’Appolonia and Abrami 1997; Braskamp and Ory 1994; Cashin and Downey 1992; Marsh and Roche 1997). Marsh et al. (2002) conclude that, with careful attention to measurement and theoretical issues, students’ evaluations of teaching are reliable and stable. Another reason to investigate students’ perceptions of supervisors’ behaviors is the use as a feedback instrument: student perceptions mediate between the supervisor behaviour and effects on the students (Shuell 1996).

Marsh et al. (2002) indicate that, although there is a vast body of research on undergraduate students’ evaluations of teacher classroom effectiveness, only few studies on the supervision of research and PhD students have been carried out. Not only little research systematically employed student questionnaires to evaluate the quality of the PhD research supervision, but even an instrument is missing that is specific for the doctoral student experience of the relationship with their supervisor. Several more general supervisory instruments have been used such as the supervisory style inventory (Nelson and Friedlander 2001) but this questionnaire is not thoroughly adjusted for the situation of PhD candidates. The ideal mentor scale (Rose 2003, 2005) clearly has relevance for the supervisor PhD candidate relationship. By aiming at the assessment of the communication it wants to improve satisfaction with doctoral education by giving a means to align mentor’s and candidate’s profiles. It includes three subscales: integrity, guidance, and relationship that, however, in a study by Bell-Ellison and Dedrick (2008) were not consistently replicated. This questionnaire is grounded in the literature on mentor’s roles in life time development and therefore emphasizes features slightly different from characteristics of the PhD supervisory relationship.

On the other hand, more specific instruments have been used for the PhD situation. These usually include many different aspects of this situation (e.g., Anderson and Seashore-Louis 1994; Marsh et al. 2002) and thus do not give detailed information on the interpersonal relationships.

Theoretical framework

The research presented in this paper studies supervision of doctoral students from an interpersonal perspective. The interpersonal perspective describes and analyzes supervision in terms of the relationship between the supervisor and the doctoral student. Two elements are central to this perspective: the communicative systems approach and a model to describe the relationship aspect of supervisor behavior.

A major axiom of the systems approach to communication (Watzlawick et al. 1967) is that all behavior has a content and relationship aspect. This implies that supervisor behavior not only carries the content of the words being used, but also has an underlying relationship message. Interaction can be regarded as the exchange of content and relationship messages. When people interact for a longer period of time, mutual expectations will develop and, based on these expectations, patterns can be identified in the exchange of relationship messages. The patterns in the relationship messages that are communicated by the behavior of the people involved in a social system can be regarded as their interpersonal style in a relationship. What the style of a person in a relationship looks like is dependent on both parties in the communication. That means that how a relationship develops into a pattern depends on the behaviors of both parties. Therefore someone’s style depends also on the other in the communication and the style that someone displays may vary over different relationships of a person.

To describe this relationship-aspect of the supervisor behavior, we use a model developed by Wubbels et al. (2006) to analyze teacher behavior: the model of interpersonal teacher behavior. This model is based on Leary’s interpersonal circle (Leary 1957) in which the relationship aspect of behavior is described using an Influence and a Proximity dimension.

Although these two dimensions have occasionally been given other names, they have generally been accepted as universal descriptors of human interaction (e.g., Fiske et al. 2007; Judd et al. 2005). For the PhD supervisor–student relationship Gatfield (2005) in his model describes management styles with the help of two similar dimensions: structure and support. Lindén (1999) in a narrative study mentions two aspects of relationships: freedom and control, which seems to cover a smaller part of the relationship than the model by Wubbels et al. (2006). Murphy et al. (2007) studied supervisors’ and PhD candidates’ beliefs about higher degree supervision and reported controlling and guiding beliefs. These aspects clearly are present in the model used in our study.

The two dimensions of the model for interpersonal supervisor behavior, represented as two axes, underlie eight types of behavior: leadership, helpful/friendly, understanding, giving students freedom and responsibility, uncertain, dissatisfied, admonishing and strict (see Fig. 1).

Fig. 1
figure 1

The model for interpersonal supervisor behavior

Full size image

An important aspect of our model is that the dimensions map a degree of behavior. A behavior that a supervisor displays has a degree of Influence and Proximity. The higher the degree of Influence the higher the behavior is displayed on the vertical axis and similarly for the degree of Proximity on the horizontal axis. For the eight sectors this means that the closer a behavior is to the center of the model the lower the intensity of the behavior is.

Another characteristic of our model is that the dimensions are independent. One might feel that showing behavior with a high degree of Influence needs to imply to be close to the other person, or the other way around that Influence always implies to be also a bit to the left on the Proximity dimension, showing oppositional behavior. However, such associations are not necessarily: high Influence behaviors as well as low Influence behaviors can go together with high or low Proximity behaviors. For example, a supervisor may provide guidance either by setting strict rules solely based on his/her own experience (high Influence, somewhat opposition) or by anticipating on or adapting to the student’s wishes (high Influence, somewhat cooperation). In this sense our model provides a richer description of the relationships than is provided by Gatfield (2005) who refers to poles instead of degrees of intensity of behavior.

Gatfield (2005) identified four supervisory styles by combining the two poles of the dimensions structure and support. The laissez faire type (low structure and low support) in terms of the sectors of our model for interpersonal teacher behavior primarily offers students responsibility and freedom. The pastoral type (high on support and low on structure) combines understanding and helpful and friendly behavior, whereas for the contractual style (high on support and high on structure) the emphasis is on leadership and helpful friendly behavioral aspects. Finally, Gatfield’s directorial type (high structure, low support) is employing a lot of strict and leadership behavior.

Based on the model of interpersonal teacher behavior, Wubbels et al. (2006) have developed the questionnaire on teacher interaction (QTI). This instrument can be used to gather information about teachers’ interpersonal styles in teacher–students communication in secondary classrooms (for an overview of the development of this questionnaire and research findings based on data gathered with this instrument, see Wubbels et al. 2006). The original Dutch version consists of 77 items that are answered on a five-point Likert scale. Several studies have been conducted on the reliability and validity of the QTI. In all these studies both reliability and validity were satisfying (Wubbels et al. 2006). The instrument exists in several languages, amongst others Dutch, English, French, Hebrew, Slovenian, and Turkish.

Later, the QTI was adapted to other educational settings, such as the interaction between student teacher and supervising teacher in teacher education (QSI; Kremer-Hayon and Wubbels 1993a), and the interaction between school principals and their teachers in primary and secondary education (Kremer-Hayon and Wubbels 1993b). The student teacher-supervising teacher situation resembles the PhD student supervisor situation. Similar to the interaction between student teacher and supervising teacher, the interaction between PhD student and supervisor is a one-to-one interaction, rather than a one-to-many interaction.

The present paper describes the development of the questionnaire on supervisor–doctoral student interaction (QSDI) as an adaptation of the QTI and the QSI by Kremer-Hayon and Wubbels (1993a). The doctoral student supervisor interaction is a bi-directional relationship with both parties influencing the developing communication pattern. Because we want to use the QSDI for feedback purposes toward the supervisor we focus on measuring the style of the supervisor from the perspective of the doctoral student. Note that although we write about the interpersonal style of the supervisor, we always mean the style in relation to a particular student. What this style looks like depends on the behavior of the doctoral student as well.

Methods

Procedure and instruments

In order to represent the model for supervisor behavior (Fig. 1), the QSDI has to have eight scales corresponding to the eight sectors of the model. To represent the theoretical model the scales have to be ordered in a circumplex structure; this implies that two independent factors should underlie the eight scales. Every scale therefore should correlate highest with its neighbors in the model and the lower the farther away a scale is in the model; a scale should correlate highly negative with the scale opposite in the model. For the first version of the QSDI, six items per scale were formulated depicting different supervisor behaviors. These items were developed from the existing items used in secondary schools or teacher education. For example the item “my supervisor says that I am unskilled” emerged from the secondary item “this teacher says we do not perform well.” Items are scored on a five-point Likert scale ranging from ‘never/not at all’ to ‘always/very’. This early version was administered to 25 PhD students in the field of social sciences. Results were tested for circumplex structure (factor analyses and scale correlations) and scale reliabilities (Cronbach’s α). After additional adaptations a 48-item version of the QSDI was crafted. Subsequently, this version was administered to a larger sample of doctoral students.

To verify the concurrent validity of the QSDI, a modified version of the postgraduate research experience questionnaire (PREQ; Marsh et al. 2002) was administered to the target group alongside the QSDI. The PREQ was developed in a project of the Australian Council for Educational Research (1999) in a thorough process of literature review, analyses of good practice, institutional evaluation, and involving existing instruments. In several steps the questionnaire evolved to a form further investigated by Marsh et al. (2002). The PREQ appeared not to be useful to compare institutions but is a valuable measurement instrument for perceptions of individual students with good content and face validity and good psychometric characteristics such as the factor structure and scale reliability. The PREQ can be used to evaluate individual student’s experience of their PhD period in retrospect. It consists of six scales called ‘Supervision’, ‘Skill development’, ‘Climate’, ‘Infrastructure’, ‘Thesis examination’, and ‘Clarity’. Items were originally answered on an agree–disagree scale. We chose to use a five-point Likert scale ‘never/not at all’ and ‘always/very’ for the sake of uniformity of the combined QSDI–PREQ questionnaire. Because the current sample consisted of PhD students who were still working on their doctorate, our version was formulated in present rather than past tense. In addition, the ‘Thesis examination’ scale was excluded.

Finally, various doctoral student background characteristics (age, gender, and time spent on the project), gender of the supervisor, and the setting (meeting hours per week) were included. The questionnaire was administered online in English.

Sample

In total 155 members of the PhD division of the Netherlands Educational Research Association were invited by mail for participation; 98 questionnaires were completed. Of the remaining 57, 24 persons reported that they had received their doctorate already and thus had been invited erroneously. An additional 33 emails remained unreplied. Thus, the effective response ratio may be estimated to be 75%. Of the 98 participants 33% were male and 67% female; 54% were between 25 and 30 years old. Although a few students might have answered the questionnaire about the same supervisor most of the students will have had a different supervisor.

Results

Reliability and internal validity of the QSDI

As mentioned above, the eight QSDI scales should be ordered on a circumplex. An important assumption of circumplex models is that correlations between scales are getting smaller as a function of the distance between scales and that a scale correlates highest negatively with its opposite scale. The 48 items version of the QSDI did not completely satisfactory show this correlation structure. Therefore, from four scales a total of seven items were removed. Thus the final version of the QSDI consisting of 41 items emerged (see Table 5).

The reliabilities (Cronbach’s α’s) of the eight resulting scales ranged between .70 and .87 (see Table 1) which is considered to be satisfactory to good.

Table 1 Number of items, Cronbach’s α, and sample item per scale of the QSDI (n = 96 PhD students)
Full size table

Table 2 shows the correlations between the different scales of the QSDI. The circumplex assumption is only slightly violated with reference to the CD scale: the SO and OD scales showed greater negative correlations with the CD scale (−.69 and −.73, respectively) instead of the theoretically to be expected OS scale (−.66). Similarly, this disrupted correlation pattern led to a disturbance of the correlation pattern of the SC scale. In our sample SC showed the greatest negative correlation with the OD scale instead of the theoretically to be expected DO scale (−.52 and −.34, respectively).

Table 2 Correlations between scales of the QSDI (n = 96 PhD students)
Full size table

Factor analyses showed that indeed two dimensions underlie the eight scales, and that the scales do follow each other in the correct order (Fig. 2). However, the scales are not evenly distributed over the circle. Because the two dimensions Influence and Proximity are supposed to be orthogonal the correlation between the two factors should be low. The actual correlation is 0.30, which is a bit higher than expected.

Fig. 2
figure 2

Results of factor analysis on the eight scales: rotated solution for two factors explaining 75% of the total variance

Full size image

Concurrent validity of the QSDI

The α coefficients in the current sample for the PREQ ranged from .84 to .91 (see Table 3).

Table 3 Cronbach’s α and item example per scale of the PREQ
Full size table

Although the QSDI and PREQ do not have the same aspiration with regard to the underlying constructs the questionnaires aim to measure, the QSDI might be understood as a further elaboration of the supervisor-scale of the PREQ. In order to investigate the concurrent validity correlations were calculated between the Influence and Proximity dimensions of the QSDI and the scales of the PREQ. In case of a satisfactory concurrent validity high correlations should be found between Influence and Proximity of the QSDI and the supervisor-scale, and the correlation pattern of the supervisor-scale with the other scales of the PREQ should resemble the pattern of the Influence and Proximity dimensions. The correlation matrix is provided in Table 4.

Table 4 Correlations between QSDI dimensions and PREQ scales
Full size table

As expected the highest correlations were found between the supervisor-scale of the PREQ and Influence and Proximity (.58 and .66, respectively). Influence and Proximity followed the general correlation pattern of the PREQ except of small differences for two correlations. The correlations between Influence and the climate scale (.30) was slightly lower than with the infrastructure scale (.35), and slightly higher that with the clarity scale (.29). Marsh et al. (2002) mention the supervisor and skill development scales as the ones most prominent connected to the supervisors’ role. They argue that the climate, infrastructure and clarity scales in addition include perceptions related to the academic unit and the entire university. This might explain lower correlations of the supervisor–doctoral student relationships with the last three scales.

Average profile

The scores for the perceptions of a specific doctoral student about his or her supervisor can be displayed in the model shown in Fig. 1: the profile of a supervisor according to the particular student. As an example, the average profile in the data collected in this study is presented in Fig. 3.

Fig. 3
figure 3

Average profile collected from doctoral students about theirs supervisors in this study (n = 96)

Full size image

This graph shows that the supervisors in our sample are on average seen as displaying rather much leadership, helping/friendly, understanding behavior and providing a lot of student freedom an responsibility. They do not show a lot of uncertain, dissatisfied or admonishing behavior and the amount of strict behavior is moderate.

Correlational analyses showed that most background characteristics measured were not associated with the scores on dimensions or scales. Only for the Influence dimension two significant, but small correlations were found: supervisor gender correlated .24, and the number of meeting hours with the supervisor .31 with the Influence dimension. Female supervisors were seen more influential than male supervisors, and the more hours doctoral student and supervisor were meeting a week the more influential the student saw the supervisor.

Conclusion and discussion

Considering the Cronbach’s α’s of the eight scales in this study, the QSDI appeared to be a reliable instrument to gather data about doctoral students’ perceptions of their supervisor’s interpersonal style in the relationship with a particular student. The validity in terms of representing a circumplex model was reasonable but the scales were not evenly distributed on the circumplex. Comparison of the QSDI with the supervisor scale of the postgraduate research experience questionnaire showed the concurrent validity of the QSDI to be good.

The QSDI is an instrument that can be used to study the relationship between supervisors and their doctoral students. Research questions about this relationship are open to investigation with the QSDI in combination with instruments to measure other variables. One can study for example what supervisory styles are most effective in terms of length of doctoral studies, doctoral student’s satisfaction with the supervision or quality of dissertations. When combined with measuring doctoral students’ characteristics positive alignments of supervisory style with doctoral students can be sought. In learning environments research an important line of study includes two versions of student questionnaires: one asking for the preferred and the other for the actual experienced environment (cf. Fraser 1991). For doctoral supervision employing the QSDI in these two versions might reveal doctoral students’ preferences for supervision styles and combinations with preferences of supervisors and their actual styles can be studied. Finally, the QSDI can help evaluate interventions to improve supervisory relationships.

The QSDI can be used to provide supervisors with feedback about their interpersonal style with the aim to improve the quality of their supervision. Although communication between supervisor and doctoral student often will be so open that no data from a questionnaire are needed, using this questionnaire offers a framework to discuss the relationship and the data will add insights that not always will come to the fore in an unstructured discussion between supervisor and doctoral student. For a quick indication of the quality of the supervisor–doctoral student relationship sector scores can be used, and on a more detailed level scores on individual items may be utilized. By using actual and preferred forms of the QSDI discrepancies between situations strived for and what is accomplished can be brought to the surface, thus providing an avenue for improvement, and a basis for discussing supervision. Similarly, for the candidates such an assessment at the beginning of the project might help articulating what he or she wants from a supervisor. Thus, the QSDI can be used in the matching process when actual supervisors’ styles and preferred styles of students are known.

In addition to doctoral student’s perceptions, also supervisors’ perceptions of their own style and their preferred style can be collected by asking the supervisor to answer the questionnaire for this purpose. The sector scores conveniently can be displayed in the model both for doctoral student and supervisor perceptions (see Fig. 4 for an example).

Fig. 4
figure 4

Example of supervisor ideal, supervisor self perception and doctoral student perception

Full size image

Several studies have shown that students’ feedback on instructors’ performance may have positive effects on an instructors’ teaching (see for a review Marsh and Dunkin 1997; Marsh and Roche 1993). Appropriate consultation strengthens this effect. Experiences with the QTI with teachers at the primary and secondary level are promising (Scott et al. 2003; Derksen 1995).With the QSDI we have some positive experience as a feedback instrument but no formal research study has been conducted until now. As has been shown for teacher’s feedback for students, important conditions must be met to make feedback supportive (Hattie and Timperley 2007). Such conditions are a good climate, thorough knowledge of the teacher of the content area, and an emphasis rather on the task than on the person. Important ingredients for an effective strategy to improve supervision based on students’ feedback probably are specific feedback, concrete suggestions for improvement and involvement of a trained consultant (Marsh et al. 2002). Through the items included, the QSDI gives ample opportunity to provide such concrete, specific information about aspects of the behavior that might need improvement. It gives for example a more differentiated view than the four styles distinguished by Gatfield (2005).

Several tenets of the model for interpersonal supervisor behavior seem important when the model is used for the purpose of coaching supervisors of PhD candidates. First, it is important to remember that the two dimensions of the model are independent. In our experience supervisors often assume that showing behavior that is high on the Influence dimension implies to be also a bit to the left on the Proximity dimension. Similarly they tend to combine high Proximity with low Influence behavior. This, however, is not necessary: high Influence as well as low Influence behaviors may go together with behaviors that are high or low on Proximity. As mentioned above, a supervisor for example may provide guidance either by setting strict rules solely based on his/her own experience (high Influence, somewhat opposition) or by anticipating on or adapting to the student’s wishes (high Influence, somewhat cooperation). Keeping this in mind might be helpful for supervisors.

Second, behaviors that are situated opposite to each other in the model are the most difficult to combine. For example, the tension that was mentioned in the section on supervisory problems in this paper between guidance and assessment is in the model reflected by the opposite position of the dissatisfied and helpful/friendly sectors. A supervisor must both be able to support a PhD student and display dissatisfaction when a product of the student does not meet the required standards. It might help to make supervisors aware of this opposite position and the need to learn to combine these behaviors productively, or to become flexible in their use in different situations.

Third, the concept of self reinforcing processes is important (Wubbels et al. 1988). In relationships between teachers and students different principles apply for the Influence and Proximity dimensions. Behaviors of participants associated with the Influence dimension tend to evoke opposite behaviors (i.e., reciprocity): for example, the more a supervisor might provide structure, the more a student might become dependent on the supervisor. For the Proximity dimension another process is involved (i.e., complementarity): behaviors on this dimension tend to evoke similar behavior of the other participant; the more friendliness the supervisor expresses, the more friendliness the student will show. Similarly, hostile behavior of the supervisor is likely to invoke hostility from the PhD student.

In conclusion, the QSDI can be used both as a research and as a feedback instrument. It can contribute to solving the three supervisory problems mentioned in the introduction of this paper. First, the opposite placement of helping/friendly and dissatisfied behavior in the model helps clarify the tension between guidance and assessment in the supervisory process. Second, analyzing doctoral student’s preferred style and the supervisor’s ideal may help in the matching of PhD candidates and supervisors. Finally, the use of the QSDI for feedback will help to strengthen or to create a climate in research institutes where evaluation of doctoral student’s experiences is common practice.

The QSDI maps the relationship between a doctoral student and his or her supervisor from the perspective of the student. For future research it could be interesting to complement this view with the supervisor’s perception of the student’s style. Kam (1997) for example showed the importance of a student’s tendency to rely on the supervisor is important in shaping the relationship.