An evaluation of transit procurement training
Introduction
The National Transit Institute (NTI) was established at Rutgers, The State University of New Jersey, in 1991 and has been delivering workforce-training courses to the transit industry for over 20 years. At the end of each course, NTI conducts evaluations using an in-class paper survey; this measures the immediate reaction of participants in the course and is essentially a measure of ‘customer satisfaction’ (Kirkpatrick, 1998). However, these course-contemporaneous surveys cannot measure longer-term changes and outcomes that arise as a result of the participants taking a course. To that end, the approach presented here delves deeper to program-evaluate actual changes to practices that are made subsequent to course participation, and, in so doing, adds to the broader transit-training and program evaluation literature.
Procurement procedures required by the Federal Transit Administration (FTA) are one of the critical elements of federal oversight of the many transit agencies throughout the country. The aim of these regulations is to maintain adequate competition among bidders and to provide a framework for the best selection of goods and services acquired by transit agencies. FTA performs triennial reviews of each transit agency’s performance, which includes an analysis of procurement procedures and compliance with regulatory requirements (US Department of Transportation (USDOT), 2014). These reviews frequently find that some transit agencies have difficulties complying with procurement rules; the procurement course offered by NTI, then, is seen as vital to the mission of FTA to improve regulatory compliance.
Our evaluation largely follows the framework outlined by Kirkpatrick (1998), who specified four levels of training: reaction, learning, behavior, and results.1 The first step, reaction, measures customer satisfaction with the training; to capture this initial component, we included questions in our survey aimed at assessing satisfaction with the course and the instructors. Learning involves the ability of course participants to improve their knowledge, change attitudes, and increase their skill set. These are seen as precursors to the third step, behavior, which assesses whether the training course led to actual changes in work practices. A successful outcome of this sequence is partly contingent on a supportive work environment that allows changes to occur (Holton, Chen, & Naquin, 2003). The final step encompasses results; for a transit agency this would be improved or continuing compliance with FTA regulations. This fourth step was not directly assessed in our evaluation.
Alliger and Janak (1989) critiqued some of the assumptions of Kirkpatrick’s framework by asking, in particular, whether the four “steps” (or “levels” as they prefer) are causally linked, i.e., does a “good reaction” necessarily imply learning? Is learning necessary for behavior change? And are results always dependent on behavior change? While Alliger and Janak (1989) do not propose an alternative model, their caution is warranted and well-taken. For example, it is well known that “satisfied” students, by which we mean those giving good reactions, often learn the least (Rodin & Rodin, 1972); in contrast, Kirkpatrick (2006) notes that good reactions (i.e., satisfied customers) are required for organizations to continue to engage in training. We discuss some of the disparity between results on our knowledge-based questions and actual behavioral change. Another criticism is that research designs are often not complete (Aguinis & Kraiger, 2009); one argument is that control samples are needed. In program evaluation practice, however, this is often not feasible and was not in our case. Rather, we used a mixed mode research design, complementing the quantitative survey with qualitative in-depth interviews to add context and sufficient detail to better understand the benefits course participants gained from their training.
Holton et al. (2003) have examined factors associated with what he defines as “transfer”; i.e., how training knowledge is transferred into practice within organizations. They conclude that each organization is unique and it is probably not possible to make generalizations based on common organizational attributes. Other work suggests that support from supervisors and a good work environment are key conditions that make transfer possible and lead to successful training (Aguinis & Kraiger, 2009). Our evaluation includes questions that probe the ability to apply skills learned in the training course.
Our evaluation of the procurement course focused on how transit agency staff achieve change at their agencies. While most transit agencies comply with procurement regulations they need to make sure new staff are properly trained and that existing staff are retrained as regulations change. With that in mind, and following Kirkpatrick (1998), our evaluation focused on four elements:
- •
overall satisfaction with the course (reaction);
- •
course knowledge retention (learning);
- •
changes in agency practices as a result of the course (behavior); and
- •
participant suggestions for improving the course including identification of any deficiencies.
We did not evaluate long-term results, largely due to time constraints; Kirkpatrick (2006) makes the point that longitudinal evaluation requires frequent follow up and monitoring and, as such, is generally resource-intensive. Moreover, as noted above, FTA monitors agencies via their triennial review of performance.2
Section snippets
Methodology
An online survey and follow-up interviews with participants in the NTI “Orientation to Transit Procurement” course were administered to transit agency employees who took the course between December 2011 and June 2014; NTI provided contact details for all participants during this time period which covered 25 offerings of the course. In consultation with NTI staff a questionnaire was developed and implemented as an on-line survey instrument. Some questions were included from another survey
Satisfaction with the course
Overall, survey respondents were very satisfied, found the course very useful, and provided universal praise for NTI and the instructors. Respondents indicated their satisfaction in both their responses to multiple-choice and open-ended questions about their course experience in general, as well as with the instructors, in particular.
While 86 percent of respondents reported “strong agreement” or “agreement” with the statement “I learned a great deal in the course,” only 5 of the 257 respondents
Conclusions
Our evaluation sought to examine four key aspects of NTI’s “Orientation to Transit Procurement” course. These were overall satisfaction with the course, knowledge retention, changes in agency practices, and participant suggestions for improvements.
Transit professionals who take the course overwhelmingly report satisfaction with the course and with the instructors. The course provides a useful introduction to the work they do and they use the skills they learned when they return to their offices.
Acknowledgement
This research was funded by the Federal Transit Administration.
References (13)
- et al.
Benefits of training and development for individuals and teams, organizations, and society
The Annual Review of Psychology
(2009) - et al.
Kirkpatrick’s levels of training criteria: Thirty years later
Personnel Psychology
(1989) - Canada School of Public Service, Evaluation of Training Provided to the Procurement, Materiel Management and Real...
- et al.
An examination of learning transfer system characteristics across organizational settings
Human Resource Development Quarterly
(2003) Techniques for evaluating training programs
Journal of the American Society for Training and Development
(1959)Techniques for evaluating training programs: Part 2—learning
Journal of the American Society for Training and Development
(1959)
Cited by (2)
A qualitative study on the urgency of attitude in designing effective procurement certification training
2023, Journal of High Technology Management ResearchProcurement function modelled as a taxonomy of perishable goods along the cold chain
2020, International Journal of Procurement Management