4 Could It Be That It Does Make Sense? A Program Review Process for Integrating Activities

This chapter presents a model for a comprehensive program review process that can be used on any campus. Faculty developers maintain a critical role in a campus-wide program review initiative. This model is based upon the development of institutional priorities that guide the development of goals and objectives far academic units across the campus. The program review process is based on a core of regularly produced institutional data that can be used by all units to inform decision-making. The review process is conducted on an annual or biannual basis with periodic major review coinciding with accreditation visits. The ultimate success of the model is tied to making budgetary and resource allocation decisions based on the assessment that grows out of the program review process.

INTRODUCTION

There are times when many faculty and administrators in higher education find themselves in one of those periodic places where they are ask, “Is this why I went into higher education?” Thinking back to when I entered graduate school, I was attracted to an academic career by a combination of opportunities to pursue research interests and the excitement of teaching students about my field of study. Of course, the flexible time frame for engaging in professional activities and the degree of autonomy afforded by the professorate were also appealing.

Many years later, the attractions haven’t really changed, but the demands on faculty and administrators have. The costs of providing higher education have risen, but the resources have not kept pace. The opening of higher education to a much broader range of students has succeeded, but the corresponding preparation of faculty to teach these new students has not. The demand from tuition payers for accountability has increased, but the support and enthusiasm for generating evidence of performance has lagged behind. It would appear that the days of reading books, writing articles, and teaching students are gone, lost in an increasingly complex academic world which does not resemble the vision set forth in our doctoral programs.

However, all is not lost. It is true that faculty are being asked to do many more “administrative” activities than ever before: writing more reports, engaging in more committee work, spending more time on institutional planning, etc. On the whole, faculty members have been fairly good about assuming these new responsibilities. The serious complaints arise primarily when faculty perceive that all of their time, energy, and good thinking results in nothing being done, no changes or improvements, and recommendations that are ignored. When an institution solicits the involvement of its faculty and asks them to engage their most important resource, their minds, it is incumbent upon the institution to demonstrate how it has paid attention to the investment of its faculty.

The program review process can provide a way to organize many of the activities that faculty love to hate but need to do, and help provide meaning to a whole set of activities in which we ask faculty to be engaged. Program review is a process for knitting together an institution’s mission with the various data-gathering and reporting activities required by external and internal decision-making and accreditation of programs.

PROGRAM REVIEW

Program review is designed to enhance planning within and among academic units to effectively use campus resources and advance university priorities. Program review brings together the assessment process, enrollment management, and external and professional accreditation reviews with strategic budget planning and decision-making. The overall goal of program review is to allow academic units within normal activities to articulate their goals and objectives in relation to the university’s priorities through a regular process of internal and external review of qualitative and quantitative information about program activities, demonstration of progress toward achievement of goals, and the use of outcomes for program improvement.

Figure 4.1 presents a conceptual model for review of university programs. The university operates within a larger context of external entities that are connected to the university in various ways (e.g., governmental agencies that regulate processes and behaviors, employ graduates, etc.). Community groups and agencies, alumni, graduate schools, employers, and professional and regional accrediting bodies all affect the operation of the university in one form or another. Each in some way influences the program mix offered by the university, the student body that attends the institution, and the mission and priorities of the campus.

Within the institutional environment, the university engages in many activities. The inputs are composed of the people, the physical and social capital, the available resources, and the publicly enunciated mission and priorities. A whole set of processes result from the activities conducted by the input actors, resulting in an array of outputs. In the normal process of conducting activities, individuals and programs determine priorities for their attention. These priorities become the foci of any review process.

The review process itself begins with the enunciation of unit goals and objectives related to the activities of the unit. Data are collected that illuminate the purposes of the review. Funds and institutional support are required to conduct the program review. A whole set of processes is involved once the goals, objectives, and data are collected. Perhaps the most important processes involve analysis of data and reflection by members of the unit on the progress toward achieving goals and objectives, and on the appropriateness of continuing to maintain the goals and objectives or to develop new goals.

The program review process is accomplished through a recurring schedule of goal setting, data-gathering, and reporting. Each unit

• establishes its goals and objectives related to teaching, scholarship, and service for its respective programs

• provides analysis of data received and/or collected to demonstrate progress toward the stated goals and objectives

• reports on its progress toward meeting its goals and objectives within the unit’s and the university’s mission and priorities

The key to the entire program review process is the outcome of the review process in terms of its influence on institutional decision-making.

Figure 4.1 Conceptual Model for Reviews of University Programs

If the outcome part of the process is ignored (e.g., if decisions are not based on review outcomes and resources are not reallocated) the process will soon become an empty exercise. Feedback occurs at all stages, influencing every other aspect of the process. This recursive process very quickly will reveal whether the program review activities result in tangible benefits, decisions, or outcomes grounded in the review findings (Holdaway, 1991).

Program Review Criteria

Decisions to allocate or reallocate resources must be based on criteria that are accepted by the actors within the institution. Typically, the criteria used to assess achievement of goals and objectives are the same criteria that are used for making many other decisions in the institution, or that are required to be gathered and reported to reflect various types of performance by individuals, units, or the campus as a whole. Figure 4.2 presents common criteria for program review derived from an examination of types of information that higher education institutions across the country regularly gather and report to a variety of agencies and for a variety of purposes.

Figure 4.2 Common Criteria for Program Review

The criteria and possible measures are grouped into primary categories and are linked to documents that exist in one form or another at every higher education institution.

Many of the criteria will look very familiar to any faculty member or administrator (e.g., publications, citations, and professional presentations at conferences, and student retention and graduation rates). In addition, each higher education institution will have its own criteria that reflect the mission of its institution (e.g., at Portland State University, the proportion of graduates who took community-based learning courses and number of community-based learning courses).

Six primary categories of criteria emerge from the examination of performance documents common to higher education. All other categories of criteria flow from the first category: university mission. Every accrediting agency, whether it be a professional association or a regional accrediting body, starts with the mission of the institution. The activities of the institution must flow from and be consistent with the mission. By basing program review on the mission–by asking the question, “How well does the program/unit meet the university’s expectations and priorities related to the mission?”–a critical component of accreditation review is embedded across the institution’s day-to-day activities.

The next three categories of criteria reflect central portions of any university’s mission: the quality of faculty, the effectiveness of the curriculum in serving students, and the effectiveness of the program as a whole in serving the university. Many of the criteria listed under these categories are found in routine data-gathering and reporting by academic units for purposes of tenure, promotion, new program development, rankings, etc.

The final two categories of criteria reflect financial factors related to cost effectiveness of programs and the level of institutional support for the programs in achieving their goals and objectives in relation to other units. Since higher education increasingly has to compete for resources with many other priorities, the ability to demonstrate cost effectiveness becomes important in reviewing an institution’s programs. This is not to suggest that programs that are less cost effective than others will result in decisions to eliminate or scale back programs, but it must be at least part of the review process in the overall decision-making for the university. Centrality to mission, or to the core curriculum, might easily outweigh lower performance on cost effectiveness measures. In addition, the level of institutional support for the program is an important ingredient in assessing program performance. Programs that achieve wonderful outcomes with little institutional support may be able to make a case for more institutional support to allow the program to achieve even higher performance levels, or to try development of new or innovative extensions of the program.

No single criterion or category is designed to overwhelm the others. It is the overall performance of the program on all measures that provides the broad basis for examining outcomes and decisions. It is important to remember that the program review process provides as much value to the unit itself through its analysis and reflection on program improvement as it does to the institution generally for decisions related to resource allocation.

The Annual Review

Units report in a relatively brief format on progress toward goals on an annual basis to their respective faculties and deans, then to the office of academic affairs (OAA) and the budget process (Figure 4.3). The academic unit annual report would incorporate the criteria as presented in Figure 4.2. The unit would use the report to plan for the next year(s) and to make adjustments based on the year’s data collection. Anomalies and unanticipated outcomes can also be addressed and discussed by the unit. The reflective component each year enables the unit to continuously check on effectiveness, provides an opportunity for mid-course adjustments in goals and objectives or allocation of activities and resources within the unit, and informs administrative units beyond them about progress toward agreed upon goals.

Colleges and schools, through the dean’s office, then prepare a similar annual document reporting on the college/school progress toward achieving its goals and objectives. The college/school goals and objectives are not simply a compilation of the unit goals and objectives; rather, they are the priorities of the college/school that position it within the institution’s overall mission. The department and program units within the college/school have built their goals and objectives from the college/school goals and objectives, so it is a fairly easy process to demonstrate college/school progress through compiling achievements of its units. Since all levels of reporting use the same set of criteria, a common language is maintained throughout the process which facilitates ease of communication both within and across units, colleges, and schools. Colleges and schools forward their annual report to the provost along with the individual reports from the units within the college/school.

Figure 4.3 Annual Review Process

The office of academic affairs reviews the college/school reports and provides feedback through the deans on the directions and accomplishments of units across the campus. The college/school reports allow OAA to then prepare an overall set of priorities for the budget process. The annual cycle of the program review process readily reveals progress on accomplishing institutional goals and objectives, allowing the resources of the institution to be directed on an ongoing basis to where they can best be utilized to support the efforts of faculty within the units in all parts of the campus.

A large proportion of academic units already engage in regular program review in conjunction with their professional accrediting bodies, or in the case of some departments that don’t have professional or external accrediting agencies, the periodic review developed by their dean. The latter is typically the case in many arts and sciences colleges and some fine arts departments. To streamline the program review process, maximize its value, and reduce its redundancy, the program review process needs to build on the existing accreditation requirements as much as possible. The accreditation schedule would serve as the major point for a periodic, summative self-study and review by the dean and the provost’s office (Figure 4.4).

In the program review process presented in Figure 4.4, annual reviews that incorporate the criteria discussed earlier would allow academic units to track progress on achieving goals and objectives as well as reduce the amount of work required at the time of an accreditation visit. Through annual reviews, most of the necessary information would already be collected in a form required by the accrediting agency. In addition, the annual documentation of progress on student learning outcomes and other goals would add a necessary dimension expected by most of the professional accreditation agencies for assessing outcomes and using the assessment information to improve decision-making about the program.

Figure 4.4 Program Review Cycle

In the annual review, the unit’s stated goals and objectives would each be addressed through use of the information gathered in conjunction with the criteria identified in Figure 4.2. The combination of quantitative data (typically supplied by a campus’s office of institutional research and planning), data on student learning outcomes, and any other data collected by the unit in responses to specific goals and objectives of the unit, is the foundation for documenting progress toward accomplishment of the unit’s goals and objectives. The assumption is that progress will not be uniform on all goals and objectives, or that every goal and objective is designed to be achieved every year. Rather, every goal and objective is included in the annual report because it was considered important enough to be a stated goal or objective, and therefore, should remain a continuous part of the unit’s collective consciousness. Indeed, many student learning outcomes may only have data collected on a periodic schedule rather than every year.

The unit analysis of the data, the use of the data to inform decision making, and the unit’s reflection on the meaning of the data for action by the unit form the basis of the annual report. The annual reports over a series of years provide a continuing indication of performance for the unit in relation to goals and objectives that are linked to the priorities of the institution.

CONCLUSION

The organization of unit decision-making and action-taking around agreed-upon goals and objectives results in greater institutional coherence. Program review allows for the focusing of institutional resources on priorities. It also allows for every unit on a campus to determine how they want to link with institutional priorities, and to demonstrate how they contribute to the goals of the institution. The key to the process is the collection of relevant data and the use of the data to inform decisions about each unit’s goals and objectives. The success of program review rests on the allocation and reallocation of resources based on progress toward achieving priority institutional goals. The engagement of the institution’s many constituencies in the development of institutional priorities at the outset of the process ensures the necessary broad-based support for the over arching goals from which unit goals and objectives will flow.

Program review can be the vehicle for creating broadly understood meaning within an institution and among the many groups who comprise the extended university community. Program review can create coherence in activities and greater effectiveness in the allocation of resources. Ultimately, the program review process can help focus an institution and provide a mechanism for conducting necessary conversations about institutional mission, direction, and priorities.

REFERENCE

• Holdaway, E., et al. (1991, Fall). Program reviews: Practices and lessons. Canadian Society for the Study of Higher Education, 9, 2-11.

Contact:

• Terrel Rhodes

• Vice Provost for Curriculum and Undergraduate Studies

• Portland State University

• Box 751

• Portland, OR 97207

• (503) 725-3000

• Email: trhodes@pdx.edu

Terrel Rhodes is Vice Provost for Curriculum and Undergraduate Studies at Portland State University. He is the institutional liaison with the Northwest Association of Colleges and Schools. His teaching and research interests have involved policy analysis, program evaluation, and assessment. He holds an appointment as Professor of Public Administration.