An evaluation of transit procurement training

Highlights

• In the USA, public transport providers must follow detailed procurement regulations.

• Agency staff receive training in how to comply with procurement regulations.

• An evaluation of the effectiveness of this training was done, changes in practices were evaluated.

• Overall, the training is effective in improving regulatory compliance, although detailed knowledge of regulations appears weak.

Abstract

We evaluated a training course called “Orientation to Transit Procurement”, designed and conducted by the National Transit Institute. This course is designed to provide Federal Transit Administration (FTA) grantees an overview of regulations and best practices related to the procurement process. Our objective in conducting the evaluation was to understand how transit agency staff made changes in procurement practices in response to the course training. The evaluation was mixed mode: an Internet survey followed by in-depth interviews with a small group of respondents. Survey respondents were also provided with an open-ended question providing us with additional context for our evaluation. Results show that the training is substantially successful at meeting the goal of improving procurement practices at transit agencies; indeed, most respondents report making changes at their agencies as the proximate result of the training. This was at odds with our exploration of knowledge of procurement topics, as most respondents gave inaccurate answers on multiple-choice “knowledge questions”. This may have been due to question structure or, more likely, the nature of online surveys. Suitable training on the procurement of information technology was also a main concern. The lack of training in this area is indicative of the broader challenge facing public transit agencies in how to incorporate new forms of technology into their existing practices and bureaucratic structures.

Introduction

The National Transit Institute (NTI) was established at Rutgers, The State University of New Jersey, in 1991 and has been delivering workforce-training courses to the transit industry for over 20 years. At the end of each course, NTI conducts evaluations using an in-class paper survey; this measures the immediate reaction of participants in the course and is essentially a measure of ‘customer satisfaction’ (Kirkpatrick, 1998). However, these course-contemporaneous surveys cannot measure longer-term changes and outcomes that arise as a result of the participants taking a course. To that end, the approach presented here delves deeper to program-evaluate actual changes to practices that are made subsequent to course participation, and, in so doing, adds to the broader transit-training and program evaluation literature.

Procurement procedures required by the Federal Transit Administration (FTA) are one of the critical elements of federal oversight of the many transit agencies throughout the country. The aim of these regulations is to maintain adequate competition among bidders and to provide a framework for the best selection of goods and services acquired by transit agencies. FTA performs triennial reviews of each transit agency’s performance, which includes an analysis of procurement procedures and compliance with regulatory requirements (US Department of Transportation (USDOT), 2014). These reviews frequently find that some transit agencies have difficulties complying with procurement rules; the procurement course offered by NTI, then, is seen as vital to the mission of FTA to improve regulatory compliance.

Our evaluation largely follows the framework outlined by Kirkpatrick (1998), who specified four levels of training: reaction, learning, behavior, and results.¹ The first step, reaction, measures customer satisfaction with the training; to capture this initial component, we included questions in our survey aimed at assessing satisfaction with the course and the instructors. Learning involves the ability of course participants to improve their knowledge, change attitudes, and increase their skill set. These are seen as precursors to the third step, behavior, which assesses whether the training course led to actual changes in work practices. A successful outcome of this sequence is partly contingent on a supportive work environment that allows changes to occur (Holton, Chen, & Naquin, 2003). The final step encompasses results; for a transit agency this would be improved or continuing compliance with FTA regulations. This fourth step was not directly assessed in our evaluation.

Alliger and Janak (1989) critiqued some of the assumptions of Kirkpatrick’s framework by asking, in particular, whether the four “steps” (or “levels” as they prefer) are causally linked, i.e., does a “good reaction” necessarily imply learning? Is learning necessary for behavior change? And are results always dependent on behavior change? While Alliger and Janak (1989) do not propose an alternative model, their caution is warranted and well-taken. For example, it is well known that “satisfied” students, by which we mean those giving good reactions, often learn the least (Rodin & Rodin, 1972); in contrast, Kirkpatrick (2006) notes that good reactions (i.e., satisfied customers) are required for organizations to continue to engage in training. We discuss some of the disparity between results on our knowledge-based questions and actual behavioral change. Another criticism is that research designs are often not complete (Aguinis & Kraiger, 2009); one argument is that control samples are needed. In program evaluation practice, however, this is often not feasible and was not in our case. Rather, we used a mixed mode research design, complementing the quantitative survey with qualitative in-depth interviews to add context and sufficient detail to better understand the benefits course participants gained from their training.

Holton et al. (2003) have examined factors associated with what he defines as “transfer”; i.e., how training knowledge is transferred into practice within organizations. They conclude that each organization is unique and it is probably not possible to make generalizations based on common organizational attributes. Other work suggests that support from supervisors and a good work environment are key conditions that make transfer possible and lead to successful training (Aguinis & Kraiger, 2009). Our evaluation includes questions that probe the ability to apply skills learned in the training course.

Our evaluation of the procurement course focused on how transit agency staff achieve change at their agencies. While most transit agencies comply with procurement regulations they need to make sure new staff are properly trained and that existing staff are retrained as regulations change. With that in mind, and following Kirkpatrick (1998), our evaluation focused on four elements:

• overall satisfaction with the course (reaction);

• course knowledge retention (learning);

• changes in agency practices as a result of the course (behavior); and

• participant suggestions for improving the course including identification of any deficiencies.

We did not evaluate long-term results, largely due to time constraints; Kirkpatrick (2006) makes the point that longitudinal evaluation requires frequent follow up and monitoring and, as such, is generally resource-intensive. Moreover, as noted above, FTA monitors agencies via their triennial review of performance.²