Issue
Authors retain the copyright without restrictions for their published content in this journal. HSSR is a SHERPA ROMEO Green Journal.
Publishing License
This is an open-access article distributed under the terms of
A COMPARATIVE ANALYSIS OF THE EFFICACY OF THREE PROGRAM-EVALUATION MODELS –A REVIEW ON THEIR IMPLICATION IN EDUCATIONAL PROGRAMS
Corresponding Author(s) : Zafar Iqbal
Humanities & Social Sciences Reviews,
Vol. 9 No. 3 (2021): May
Abstract
Purpose of the study: This article reviews the comparative efficacy, theoretical and practical background of three program evaluation models (Stufflebeam’s CIPP model, Kirkpatrick’s model, and outcome-based evaluation models) and their implications in educational programs. The article discusses the strengths and limitations of the three evaluation models.
Methodology: Peer-reviewed and scholarly journals were searched for articles related to program evaluation models and their importance. Keywords included program evaluation’, ‘assessment’, ‘CIPP model’, ‘evaluation of educational programs, ‘outcome-based model, and ‘planning’. Articles on Stufflebeam’s CIPP model, Kirkpatrick’s model, and outcome-based evaluation models were particularlyfocused because the review aimed at analysing these three models. The strengths and inadequacies of the three models were weighed and presented.
Main Findings: The three models –outcome-based evaluation model, the Kirkpatric model, and the CIPP evaluation model –discussed in this review, have some strengths and weaknesses. Among the compared models, the CIPP model seems more appropriate for its implantation in evaluating educational programs because it is broader, comprehensive, flexible, cost-effective, and feasible.
Applications of this study: Like other programs and projects, evaluation of educational programs is necessary to achieve high standards, better outcomes, and meet the objectives. Evaluation is employed before designing a particular educational program or during the already designed program. This review concludes that among different evaluation models, the CIPP evaluation model is more appropriate in evaluating educational programs because it is more comprehensive, efficient, and feasible.Employment of the CIPP model for evaluating educational programs can achieve plausible results about the overall progress of the educational programs.
Novelty/Originality of this study: This review highlights the importance of different program evaluation models. It concludes that the CIPP evaluation model offers an excellent mechanism to evaluate educational programs at different stages.
Keywords
Download Citation
Endnote/Zotero/Mendeley (RIS)BibTeX
- AbdiShahshahani, M., Ehsanpour, S., Yamani, N., Kohan, S., & Hamidfar, B. (2015). The evaluation of reproductive health PhD program in Iran: a CIPP model approach. Procedia-Social and Behavioral Sciences, 197, 88-97. https://doi.org/10.1016/j.sbspro.2015.07.059 DOI: https://doi.org/10.1016/j.sbspro.2015.07.059
- Afzal, A., & Yousaf, M. (2017). Development of an Evaluation Model for BS 4 Years Program at University Level. Pakistan Journal of Social Sciences, 37(1), 114-124.
- Akpur, U., Alci, B., & Karatas, H. (2016). Evaluation of the Curriculum of English Preparatory Classes at Yildiz Technical University Using CIPP Model. Educational Research and Reviews, 11(7), 466-473. DOI: https://doi.org/10.5897/ERR2016.2638
- Alexandrov, N., & Sancho, M. R. (2017). Learning outcomes based evaluation of HPC professional training. Procedia Computer Science, 108, 2141-2150. https://doi.org/10.1016/j.procs.2017.05.135 DOI: https://doi.org/10.1016/j.procs.2017.05.135
- Alkin, M. C., & King, J. A. (2017). Definitions of evaluation use and misuse, evaluation influence, and factors affecting use. American Journal of Evaluation, 38(3), 434-450. DOI: https://doi.org/10.1177/1098214017717015
- Anh, V. T. K. (2018). Evaluation models in educational program: Strengths and weaknesses. VNU Journal of Foreign Studies, 34(2), 140-150. https://doi.org/10.25073/2525-2445/vnufs.4252 DOI: https://doi.org/10.25073/2525-2445/vnufs.4252
- Asch, D. A., Nicholson, S., Srinivas, S. K., Herrin, J., & Epstein, A. J. (2014). How do you deliver a good obstetrician? Outcome-based evaluation of medical education. Academic Medicine, 89(1), 24-26. https://doi.org/10.1097/ACM.0000000000000067 DOI: https://doi.org/10.1097/ACM.0000000000000067
- Aziz, S., Mahmood, M., & Rehman, Z. (2018). Implementation of CIPP model for quality evaluation at school level: A case study. Journal of Education and Educational Development, 5(1), 189-206. http://dx.doi.org/10.22555/joeed.v5i1.1553 DOI: https://doi.org/10.22555/joeed.v5i1.1553
- Bates, R. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341-347. https://doi.org/10.1016/j.evalprog plan.2004.04.011 DOI: https://doi.org/10.1016/j.evalprogplan.2004.04.011
- Boulay, D. A., & Han, H. (2008). Examining Values, Use, and Role in Evaluation: Prospects for a Broadened View. Online Submission.
- Brewer, E. W. (2011). Evaluation models for evaluating educational programs. In Assessing and evaluating adult learning in career and technical education (pp. 106-126). IGI Global. https://doi.org/10.4018/978-1-61520-745-9.ch007 DOI: https://doi.org/10.4018/978-1-61520-745-9.ch007
- Butler, M. O. (2020). Evaluation. In Oxford Research Encyclopaedia of Anthropology. DOI: https://doi.org/10.1093/acrefore/9780190854584.013.3
- Cahapay, M. B. (2021). Kirkpatrick Model: Its Limitations as Used in Higher Education Evaluation. International Journal of Assessment Tools in Education, 8(1), 135-144. https://doi.org/10.21449/ijate.856143 DOI: https://doi.org/10.21449/ijate.856143
- Carpenter, J. (2011). Evaluating social work education: A review of outcomes, measures, research designs and practicalities. Social Work Education, 30(02), 122-140. https://doi.org/10.1080/02615479.2011.540375 DOI: https://doi.org/10.1080/02615479.2011.540375
- Celinska, K., Furrer, S., & Cheng, C. C. (2013). An outcome-based evaluation of functional family therapy for youth with behavioral problems. Journal of Juvenile Justice, 2(2), 23-36.
- Chang, Y. H. E. (2010). An empirical study of Kirkpatrick’s Evaluation Model in the hospitality industry. PhD dissertation, Florida International University. https://doi.org/10.25148/etd.FI10120807 DOI: https://doi.org/10.25148/etd.FI10120807
- Chen, W., Kelley, B., & Haggar, F. (2013). Assessing faculty development programs: Outcome-based evaluation. Journal on Centers for Teaching and Learning, 5, 107-19.
- Chen, H. T. (2016). Interfacing theories of program with theories of evaluation for advancing evaluation practice: Reductionism, systems thinking, and pragmatic synthesis. Evaluation and Program Planning, 59, 109-118. https://doi.org/10.1016/j.evalprogplan.2016.05.012 DOI: https://doi.org/10.1016/j.evalprogplan.2016.05.012
- Cunningham, R. (2003). Complexity theory and school improvement: some possible connections.Paper presented at the British Education Research Association conference, Edinburgh University, September, 2003. http://www.leeds.ac.uk/educol/documents/00003214.doc
- Darma, I. K. (2019). The effectiveness of teaching program of CIPP evaluation model. International ResearchJournal of Engineering, IT and Scientific Research, 5(3), 1-13. https://doi.org/10.21744 /irjeis.v5n3.619 DOI: https://doi.org/10.21744/irjeis.v5n3.619
- Dewi, L. R., & Kartowagiran, B. (2018). An evaluation of internship program by using Kirkpatrick evaluation model. Research and Evaluation in Education, 4(2), 155-163. https://doi.org/10.21831/reid.v4i2.22495 DOI: https://doi.org/10.21831/reid.v4i2.22495
- Dorri, S., Akbari, M., & Sedeh, M. D. (2016). Kirkpatrick evaluation model for in-service training on cardiopulmonary resuscitation. Iranian Journal of Nursing and Midwifery Research, 21(5), 493-497. https://doi.org/10.4103/1735-9066.193396 DOI: https://doi.org/10.4103/1735-9066.193396
- Epp, K. M. (2008). Outcome-based evaluation of a social skills program using art therapy and group therapy for children on the autism spectrum. Children and Schools, 30(1), 27-36. https://doi.org/10.1093/cs/30.1.27 DOI: https://doi.org/10.1093/cs/30.1.27
- Ewell, P. (2008). Building academic cultures of evidence: a perspective on learning outcomes in higher education. In Symposium of the Hong Kong University Grants Committee on Quality Education, Quality Outcomes–the way forward for Hong Kong, Hong Kong (Vol. 15, p. 2011).
- Finney, T. L. (2020). Confirmative Evaluation: New CIPP Evaluation Model. Journal of Modern Applied Statistical Methods, 18(2), 1-24. https://doi.org/10.22237/jmasm/1598889893 DOI: https://doi.org/10.22237/jmasm/1598889893
- Frenken, K. (2006). Technological innovation and complexity theory. Economics of Innovation and New Technology, 15(2), 137-155. https://doi.org/10.1080/10438590500141453 DOI: https://doi.org/10.1080/10438590500141453
- Frye, A. W., & Hemmer, P. A. (2012). Program evaluation models and related theories: AMEE guide no. 67. Medical Teacher, 34(5), 288-299. https://doi.org/10.3109/0142159X.2012.668637 DOI: https://doi.org/10.3109/0142159X.2012.668637
- Galloway, D. L. (2005). Evaluating distance delivery and eâ€learning is Kirkpatrick's model relevant?. Performance Improvement, 44(4), 21-27. DOI: https://doi.org/10.1002/pfi.4140440407
- Gill, M., & Sharma, G. (2013). Evaluation of vocational training program from the trainees' perspective: An empirical study. Pacific Business Review International, 6(5).
- Gunung, I. N., & Darma, I. K. (2019). Implementing the context, input, process, product (CIPP) evaluation model to measure the effectiveness of the implementation of teaching at Politeknik Negeri Bali (PNB). International Journal of Environmental and Science Education, 14(1).
- Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating training programs: The four levels. Berrett-Koehler Publishers, Inc., USA
- Hakan, K., & Seval, F. (2011). CIPP evaluation model scale: development, reliability and validity. Procedia-Social and Behavioral Sciences, 15, 592-599. https://doi.org/10.1016/j.sbspro.2011.03.146 DOI: https://doi.org/10.1016/j.sbspro.2011.03.146
- Hsieh, S. I., & Hsu, L. L. (2013). An outcome-based evaluation of nursing competency of baccalaureate senior nursing students in Taiwan. Nurse Education Today, 33(12), 1536-1545. https://doi.org/10.1016/j.nedt. 2013.01.012 DOI: https://doi.org/10.1016/j.nedt.2013.01.012
- Kaufman, R., & Keller, J. M. (1994). Levels of evaluation: Beyond Kirkpatrick. Human Resource Development Quarterly, 5(4), 371-80. https://doi.org/10.1002/hrdq.3920050408 DOI: https://doi.org/10.1002/hrdq.3920050408
- Liao, S. C., & Hsu, S. Y. (2019). Evaluating a continuing medical education program: New world Kirkpatrick model approach. International Journal of Management, Economics and Social Sciences, 8(4), 266-279. https://doi.org/10.32327/IJMESS/8.4.2019.17 DOI: https://doi.org/10.32327/IJMESS/8.4.2019.17
- Lippe, M., & Carter, P. (2018). Using the CIPP model to assess nursing education program quality and merit. Teaching and Learning in Nursing, 13(1), 9-13. https://doi.org/10.1016/j.teln.2017.09.008 DOI: https://doi.org/10.1016/j.teln.2017.09.008
- Madaus, D. L. S. G. F., & Kellaghan, T. (2000). Evaluation models: Viewpoints on educational and human services evaluation (Vol. 49). Springer Science & Business Media.
- MartÃnez, G. R., Pinta, D. A., & Santacruz, L. Ã. B. (2018). CIPP model to evaluate the principles applied to develop the Input Skills in the Bachelor Degree Program of EFL. International Journal of Education and Learning Systems, 3, 117-128.
- Miller, B. J. (2018). Utilizing the Kirkpatrick model to evaluate a collegiate high-impact leadership development program (Doctoral dissertation). Texas A&M University, ‎College Station, Texas‎. https://oakt rust.library.tamu.edu
- Misut, M., Pribilova, K., Orolinova, M., & Kotulakova, K. (2013). Quality of Education Improvement through Adapted Kirkpatrick Evaluation Model. Advances in Education Research, 44, 109-114.
- Mizikaci, F. (2006). A systems approach to program evaluation model for quality in higher education. Quality Assurance in Education.14 (1), 37-53. DOI: https://doi.org/10.1108/09684880610643601
- Mohebbi, N., Akhlaghi, F., Yarmohammadian, M. H., & Khoshgam, M. (2011). Application of CIPP model for evaluating the medical records education course at Master of Science level at Iranian medical sciences universities. Procedia-Social and Behavioral Sciences, 15, 3286-3290. https://doi.org/10.1016/j.sbspro.20 11.04.287 DOI: https://doi.org/10.1016/j.sbspro.2011.04.287
- Mokhtarzadegan, M., Amini, M., Takmil, F., Adamiat, M., & Sarveravan, P. (2015). Inservice trainings for Shiraz University of Medical Sciences employees: effectiveness assessment by using the CIPP model. Journal of Advances in Medical Education and Professionalism, 3(2), 77-83.
- Moreau, K. A. (2017). Has the new Kirkpatrick generation built a better hammer for our evaluation toolbox?. Medical Teacher, 39(9), 999-1001. https://doi.org/10.1080/0142159X.2017.1337874 DOI: https://doi.org/10.1080/0142159X.2017.1337874
- Morris, M. M. (2008). Evaluating university teaching and learning in an outcome-based model: Replanting bloom. PhD thesis, School of Mathematics and Applied Statistics, University of Wollongong, 2008. http://ro.uow.edu.au/theses/784
- Morrison, K. (2008). Educational philosophy and the challenge of complexity theory. Educational Philosophy and Theory, 40(1), 19-34. https://doi.org/10.1111/j.1469-5812.2007.00394.x DOI: https://doi.org/10.1111/j.1469-5812.2007.00394.x
- Naugle, K. A., Naugle, L. B., & Naugle, R. J. (2000). Kirkpatrick's evaluation model as a means of evaluating teacher performance. Education, 121(1), 135-145.
- Nisbet, G., Hendry, G. D., Rolls, G., & Field, M. J. (2008). Interprofessional learning for pre-qualification health care students: An outcomes-based evaluation. Journal of inter-professional Care, 22(1), 57-68. https://doi.org/10.1080/13561820701722386 DOI: https://doi.org/10.1080/13561820701722386
- Norberg, J., & Cumming, G. (2008). Complexity theory for a sustainable future. Columbia University Press.
- Nusche, D. (2008). Assessment of learning outcomes in higher education: A comparative review of selected practices.OECD Education Working Paper No. 15. www.oecd.org/dataoecd/14/8/40257354.pdf
- Posavac, E. J. (2015). Program evaluation: Methods and case studies. Routledge. DOI: https://doi.org/10.4324/9781315664972
- Praslova, L. (2010). Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning outcomes and program evaluation in higher education. Educational Assessment, Evaluation and Accountability, 22(3), 215-225. https://doi.org/10.1007/s11092-010-9098-7 DOI: https://doi.org/10.1007/s11092-010-9098-7
- Rajeev, P., Madan, M. S., & Jayarajan, K. (2009). Revisiting Kirkpatrick's model–an evaluation of an academic training course. Current Science, 272-276.
- Ravicchio, F., & Trentin, G. (2015). Evaluating Vocational Educators' Training Programs: A Kirkpatrick-lnspired Evaluation Model. Educational Technology, 22-28.
- Reio, T. G., Rocco, T. S., Smith, D. H., & Chang, E. (2017). A critique of Kirkpatrick's evaluation model. New Horizons in Adult Education and Human Resource Development, 29(2), 35-53. https://doi.org/10.1002/n ha3.20178 DOI: https://doi.org/10.1002/nha3.20178
- Rogers, P. J. (2000). Program theory: Not whether programs work but how they work. In Evaluation models (pp. 209-232). Springer, Dordrecht. DOI: https://doi.org/10.1007/0-306-47559-6_13
- Rooholamini, A., Amini, M., Bazrafkan, L., Dehghani, M. R., Esmaeilzadeh, Z., Nabeiei, P., & Kojuri, J. (2017). Program evaluation of an integrated basic science medical curriculum in Shiraz Medical School, using CIPP evaluation model. Journal of Advances in Medical Education & Professionalism, 5(3), 148-154.
- Rouse, D. N. (2011). Employing Kirkpatrick's evaluation framework to determine the effectiveness of health information management courses and programs. Perspectives in Health Information Management, 8, 1-5.
- Schalock, R. L. (2001a). Outcome-based evaluation. Springer Science & Business Media.
- Schalock, R. L. (2001b). An Overview of Outcome-Based Evaluation and Its Application. Outcome-Based Evaluation, 5-16.
- Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Newbury Park: Sage.
- Smidt, A., Balandin, S., Sigafoos, J., & Reed, V. A. (2009). The Kirkpatrick model: A useful tool for evaluating training outcomes. Journal of Intellectual and Developmental Disability, 34(3), 266-274. https://doi.org/10.1080/13668250903093125 DOI: https://doi.org/10.1080/13668250903093125
- Stufflebeam, D. (2001). Evaluation models. New Directions for Evaluation, 89, 7-98. https://doi.org/10.10 02/ev.3 DOI: https://doi.org/10.1002/ev.3
- Stufflebeam, D. L. (2000). The CIPP model for evaluation. In Evaluation models (pp. 279-317). Springer, Dordrecht. DOI: https://doi.org/10.1007/0-306-47559-6_16
- Stufflebeam, D. L., & Zhang, G. (2017). The CIPP evaluation model: How to evaluate for improvement and accountability. Guilford Publications.
- Tam, M. (2014). Outcomes-based approach to quality assessment and curriculum improvement in higher education. Quality Assurance in Education.22(2), 158-168. https://doi.org/10.1108/QAE-09-2011-0059 DOI: https://doi.org/10.1108/QAE-09-2011-0059
- Usun, S. (2016). A review on the program evaluation strategies in distance education. International Journal on New Trends in Education & Their Implications, 7(3), 33-45.
- Walton, M. (2014). Applying complexity theory: A review to inform evaluation design. Evaluation and Program Planning, 45, 119-126. https://doi.org/10.1016/j.evalprogplan.2014.04.002 DOI: https://doi.org/10.1016/j.evalprogplan.2014.04.002
- Wu, B., Hu, Y., Gu, X., & Lim, C. P. (2016). Professional development of new higher education teachers with information and communication technology in Shanghai: A Kirkpatrick’s evaluation approach. Journal of Educational Computing Research, 54(4), 531-562. DOI: https://doi.org/10.1177/0735633115621922
References
AbdiShahshahani, M., Ehsanpour, S., Yamani, N., Kohan, S., & Hamidfar, B. (2015). The evaluation of reproductive health PhD program in Iran: a CIPP model approach. Procedia-Social and Behavioral Sciences, 197, 88-97. https://doi.org/10.1016/j.sbspro.2015.07.059 DOI: https://doi.org/10.1016/j.sbspro.2015.07.059
Afzal, A., & Yousaf, M. (2017). Development of an Evaluation Model for BS 4 Years Program at University Level. Pakistan Journal of Social Sciences, 37(1), 114-124.
Akpur, U., Alci, B., & Karatas, H. (2016). Evaluation of the Curriculum of English Preparatory Classes at Yildiz Technical University Using CIPP Model. Educational Research and Reviews, 11(7), 466-473. DOI: https://doi.org/10.5897/ERR2016.2638
Alexandrov, N., & Sancho, M. R. (2017). Learning outcomes based evaluation of HPC professional training. Procedia Computer Science, 108, 2141-2150. https://doi.org/10.1016/j.procs.2017.05.135 DOI: https://doi.org/10.1016/j.procs.2017.05.135
Alkin, M. C., & King, J. A. (2017). Definitions of evaluation use and misuse, evaluation influence, and factors affecting use. American Journal of Evaluation, 38(3), 434-450. DOI: https://doi.org/10.1177/1098214017717015
Anh, V. T. K. (2018). Evaluation models in educational program: Strengths and weaknesses. VNU Journal of Foreign Studies, 34(2), 140-150. https://doi.org/10.25073/2525-2445/vnufs.4252 DOI: https://doi.org/10.25073/2525-2445/vnufs.4252
Asch, D. A., Nicholson, S., Srinivas, S. K., Herrin, J., & Epstein, A. J. (2014). How do you deliver a good obstetrician? Outcome-based evaluation of medical education. Academic Medicine, 89(1), 24-26. https://doi.org/10.1097/ACM.0000000000000067 DOI: https://doi.org/10.1097/ACM.0000000000000067
Aziz, S., Mahmood, M., & Rehman, Z. (2018). Implementation of CIPP model for quality evaluation at school level: A case study. Journal of Education and Educational Development, 5(1), 189-206. http://dx.doi.org/10.22555/joeed.v5i1.1553 DOI: https://doi.org/10.22555/joeed.v5i1.1553
Bates, R. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341-347. https://doi.org/10.1016/j.evalprog plan.2004.04.011 DOI: https://doi.org/10.1016/j.evalprogplan.2004.04.011
Boulay, D. A., & Han, H. (2008). Examining Values, Use, and Role in Evaluation: Prospects for a Broadened View. Online Submission.
Brewer, E. W. (2011). Evaluation models for evaluating educational programs. In Assessing and evaluating adult learning in career and technical education (pp. 106-126). IGI Global. https://doi.org/10.4018/978-1-61520-745-9.ch007 DOI: https://doi.org/10.4018/978-1-61520-745-9.ch007
Butler, M. O. (2020). Evaluation. In Oxford Research Encyclopaedia of Anthropology. DOI: https://doi.org/10.1093/acrefore/9780190854584.013.3
Cahapay, M. B. (2021). Kirkpatrick Model: Its Limitations as Used in Higher Education Evaluation. International Journal of Assessment Tools in Education, 8(1), 135-144. https://doi.org/10.21449/ijate.856143 DOI: https://doi.org/10.21449/ijate.856143
Carpenter, J. (2011). Evaluating social work education: A review of outcomes, measures, research designs and practicalities. Social Work Education, 30(02), 122-140. https://doi.org/10.1080/02615479.2011.540375 DOI: https://doi.org/10.1080/02615479.2011.540375
Celinska, K., Furrer, S., & Cheng, C. C. (2013). An outcome-based evaluation of functional family therapy for youth with behavioral problems. Journal of Juvenile Justice, 2(2), 23-36.
Chang, Y. H. E. (2010). An empirical study of Kirkpatrick’s Evaluation Model in the hospitality industry. PhD dissertation, Florida International University. https://doi.org/10.25148/etd.FI10120807 DOI: https://doi.org/10.25148/etd.FI10120807
Chen, W., Kelley, B., & Haggar, F. (2013). Assessing faculty development programs: Outcome-based evaluation. Journal on Centers for Teaching and Learning, 5, 107-19.
Chen, H. T. (2016). Interfacing theories of program with theories of evaluation for advancing evaluation practice: Reductionism, systems thinking, and pragmatic synthesis. Evaluation and Program Planning, 59, 109-118. https://doi.org/10.1016/j.evalprogplan.2016.05.012 DOI: https://doi.org/10.1016/j.evalprogplan.2016.05.012
Cunningham, R. (2003). Complexity theory and school improvement: some possible connections.Paper presented at the British Education Research Association conference, Edinburgh University, September, 2003. http://www.leeds.ac.uk/educol/documents/00003214.doc
Darma, I. K. (2019). The effectiveness of teaching program of CIPP evaluation model. International ResearchJournal of Engineering, IT and Scientific Research, 5(3), 1-13. https://doi.org/10.21744 /irjeis.v5n3.619 DOI: https://doi.org/10.21744/irjeis.v5n3.619
Dewi, L. R., & Kartowagiran, B. (2018). An evaluation of internship program by using Kirkpatrick evaluation model. Research and Evaluation in Education, 4(2), 155-163. https://doi.org/10.21831/reid.v4i2.22495 DOI: https://doi.org/10.21831/reid.v4i2.22495
Dorri, S., Akbari, M., & Sedeh, M. D. (2016). Kirkpatrick evaluation model for in-service training on cardiopulmonary resuscitation. Iranian Journal of Nursing and Midwifery Research, 21(5), 493-497. https://doi.org/10.4103/1735-9066.193396 DOI: https://doi.org/10.4103/1735-9066.193396
Epp, K. M. (2008). Outcome-based evaluation of a social skills program using art therapy and group therapy for children on the autism spectrum. Children and Schools, 30(1), 27-36. https://doi.org/10.1093/cs/30.1.27 DOI: https://doi.org/10.1093/cs/30.1.27
Ewell, P. (2008). Building academic cultures of evidence: a perspective on learning outcomes in higher education. In Symposium of the Hong Kong University Grants Committee on Quality Education, Quality Outcomes–the way forward for Hong Kong, Hong Kong (Vol. 15, p. 2011).
Finney, T. L. (2020). Confirmative Evaluation: New CIPP Evaluation Model. Journal of Modern Applied Statistical Methods, 18(2), 1-24. https://doi.org/10.22237/jmasm/1598889893 DOI: https://doi.org/10.22237/jmasm/1598889893
Frenken, K. (2006). Technological innovation and complexity theory. Economics of Innovation and New Technology, 15(2), 137-155. https://doi.org/10.1080/10438590500141453 DOI: https://doi.org/10.1080/10438590500141453
Frye, A. W., & Hemmer, P. A. (2012). Program evaluation models and related theories: AMEE guide no. 67. Medical Teacher, 34(5), 288-299. https://doi.org/10.3109/0142159X.2012.668637 DOI: https://doi.org/10.3109/0142159X.2012.668637
Galloway, D. L. (2005). Evaluating distance delivery and eâ€learning is Kirkpatrick's model relevant?. Performance Improvement, 44(4), 21-27. DOI: https://doi.org/10.1002/pfi.4140440407
Gill, M., & Sharma, G. (2013). Evaluation of vocational training program from the trainees' perspective: An empirical study. Pacific Business Review International, 6(5).
Gunung, I. N., & Darma, I. K. (2019). Implementing the context, input, process, product (CIPP) evaluation model to measure the effectiveness of the implementation of teaching at Politeknik Negeri Bali (PNB). International Journal of Environmental and Science Education, 14(1).
Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating training programs: The four levels. Berrett-Koehler Publishers, Inc., USA
Hakan, K., & Seval, F. (2011). CIPP evaluation model scale: development, reliability and validity. Procedia-Social and Behavioral Sciences, 15, 592-599. https://doi.org/10.1016/j.sbspro.2011.03.146 DOI: https://doi.org/10.1016/j.sbspro.2011.03.146
Hsieh, S. I., & Hsu, L. L. (2013). An outcome-based evaluation of nursing competency of baccalaureate senior nursing students in Taiwan. Nurse Education Today, 33(12), 1536-1545. https://doi.org/10.1016/j.nedt. 2013.01.012 DOI: https://doi.org/10.1016/j.nedt.2013.01.012
Kaufman, R., & Keller, J. M. (1994). Levels of evaluation: Beyond Kirkpatrick. Human Resource Development Quarterly, 5(4), 371-80. https://doi.org/10.1002/hrdq.3920050408 DOI: https://doi.org/10.1002/hrdq.3920050408
Liao, S. C., & Hsu, S. Y. (2019). Evaluating a continuing medical education program: New world Kirkpatrick model approach. International Journal of Management, Economics and Social Sciences, 8(4), 266-279. https://doi.org/10.32327/IJMESS/8.4.2019.17 DOI: https://doi.org/10.32327/IJMESS/8.4.2019.17
Lippe, M., & Carter, P. (2018). Using the CIPP model to assess nursing education program quality and merit. Teaching and Learning in Nursing, 13(1), 9-13. https://doi.org/10.1016/j.teln.2017.09.008 DOI: https://doi.org/10.1016/j.teln.2017.09.008
Madaus, D. L. S. G. F., & Kellaghan, T. (2000). Evaluation models: Viewpoints on educational and human services evaluation (Vol. 49). Springer Science & Business Media.
MartÃnez, G. R., Pinta, D. A., & Santacruz, L. Ã. B. (2018). CIPP model to evaluate the principles applied to develop the Input Skills in the Bachelor Degree Program of EFL. International Journal of Education and Learning Systems, 3, 117-128.
Miller, B. J. (2018). Utilizing the Kirkpatrick model to evaluate a collegiate high-impact leadership development program (Doctoral dissertation). Texas A&M University, ‎College Station, Texas‎. https://oakt rust.library.tamu.edu
Misut, M., Pribilova, K., Orolinova, M., & Kotulakova, K. (2013). Quality of Education Improvement through Adapted Kirkpatrick Evaluation Model. Advances in Education Research, 44, 109-114.
Mizikaci, F. (2006). A systems approach to program evaluation model for quality in higher education. Quality Assurance in Education.14 (1), 37-53. DOI: https://doi.org/10.1108/09684880610643601
Mohebbi, N., Akhlaghi, F., Yarmohammadian, M. H., & Khoshgam, M. (2011). Application of CIPP model for evaluating the medical records education course at Master of Science level at Iranian medical sciences universities. Procedia-Social and Behavioral Sciences, 15, 3286-3290. https://doi.org/10.1016/j.sbspro.20 11.04.287 DOI: https://doi.org/10.1016/j.sbspro.2011.04.287
Mokhtarzadegan, M., Amini, M., Takmil, F., Adamiat, M., & Sarveravan, P. (2015). Inservice trainings for Shiraz University of Medical Sciences employees: effectiveness assessment by using the CIPP model. Journal of Advances in Medical Education and Professionalism, 3(2), 77-83.
Moreau, K. A. (2017). Has the new Kirkpatrick generation built a better hammer for our evaluation toolbox?. Medical Teacher, 39(9), 999-1001. https://doi.org/10.1080/0142159X.2017.1337874 DOI: https://doi.org/10.1080/0142159X.2017.1337874
Morris, M. M. (2008). Evaluating university teaching and learning in an outcome-based model: Replanting bloom. PhD thesis, School of Mathematics and Applied Statistics, University of Wollongong, 2008. http://ro.uow.edu.au/theses/784
Morrison, K. (2008). Educational philosophy and the challenge of complexity theory. Educational Philosophy and Theory, 40(1), 19-34. https://doi.org/10.1111/j.1469-5812.2007.00394.x DOI: https://doi.org/10.1111/j.1469-5812.2007.00394.x
Naugle, K. A., Naugle, L. B., & Naugle, R. J. (2000). Kirkpatrick's evaluation model as a means of evaluating teacher performance. Education, 121(1), 135-145.
Nisbet, G., Hendry, G. D., Rolls, G., & Field, M. J. (2008). Interprofessional learning for pre-qualification health care students: An outcomes-based evaluation. Journal of inter-professional Care, 22(1), 57-68. https://doi.org/10.1080/13561820701722386 DOI: https://doi.org/10.1080/13561820701722386
Norberg, J., & Cumming, G. (2008). Complexity theory for a sustainable future. Columbia University Press.
Nusche, D. (2008). Assessment of learning outcomes in higher education: A comparative review of selected practices.OECD Education Working Paper No. 15. www.oecd.org/dataoecd/14/8/40257354.pdf
Posavac, E. J. (2015). Program evaluation: Methods and case studies. Routledge. DOI: https://doi.org/10.4324/9781315664972
Praslova, L. (2010). Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning outcomes and program evaluation in higher education. Educational Assessment, Evaluation and Accountability, 22(3), 215-225. https://doi.org/10.1007/s11092-010-9098-7 DOI: https://doi.org/10.1007/s11092-010-9098-7
Rajeev, P., Madan, M. S., & Jayarajan, K. (2009). Revisiting Kirkpatrick's model–an evaluation of an academic training course. Current Science, 272-276.
Ravicchio, F., & Trentin, G. (2015). Evaluating Vocational Educators' Training Programs: A Kirkpatrick-lnspired Evaluation Model. Educational Technology, 22-28.
Reio, T. G., Rocco, T. S., Smith, D. H., & Chang, E. (2017). A critique of Kirkpatrick's evaluation model. New Horizons in Adult Education and Human Resource Development, 29(2), 35-53. https://doi.org/10.1002/n ha3.20178 DOI: https://doi.org/10.1002/nha3.20178
Rogers, P. J. (2000). Program theory: Not whether programs work but how they work. In Evaluation models (pp. 209-232). Springer, Dordrecht. DOI: https://doi.org/10.1007/0-306-47559-6_13
Rooholamini, A., Amini, M., Bazrafkan, L., Dehghani, M. R., Esmaeilzadeh, Z., Nabeiei, P., & Kojuri, J. (2017). Program evaluation of an integrated basic science medical curriculum in Shiraz Medical School, using CIPP evaluation model. Journal of Advances in Medical Education & Professionalism, 5(3), 148-154.
Rouse, D. N. (2011). Employing Kirkpatrick's evaluation framework to determine the effectiveness of health information management courses and programs. Perspectives in Health Information Management, 8, 1-5.
Schalock, R. L. (2001a). Outcome-based evaluation. Springer Science & Business Media.
Schalock, R. L. (2001b). An Overview of Outcome-Based Evaluation and Its Application. Outcome-Based Evaluation, 5-16.
Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Newbury Park: Sage.
Smidt, A., Balandin, S., Sigafoos, J., & Reed, V. A. (2009). The Kirkpatrick model: A useful tool for evaluating training outcomes. Journal of Intellectual and Developmental Disability, 34(3), 266-274. https://doi.org/10.1080/13668250903093125 DOI: https://doi.org/10.1080/13668250903093125
Stufflebeam, D. (2001). Evaluation models. New Directions for Evaluation, 89, 7-98. https://doi.org/10.10 02/ev.3 DOI: https://doi.org/10.1002/ev.3
Stufflebeam, D. L. (2000). The CIPP model for evaluation. In Evaluation models (pp. 279-317). Springer, Dordrecht. DOI: https://doi.org/10.1007/0-306-47559-6_16
Stufflebeam, D. L., & Zhang, G. (2017). The CIPP evaluation model: How to evaluate for improvement and accountability. Guilford Publications.
Tam, M. (2014). Outcomes-based approach to quality assessment and curriculum improvement in higher education. Quality Assurance in Education.22(2), 158-168. https://doi.org/10.1108/QAE-09-2011-0059 DOI: https://doi.org/10.1108/QAE-09-2011-0059
Usun, S. (2016). A review on the program evaluation strategies in distance education. International Journal on New Trends in Education & Their Implications, 7(3), 33-45.
Walton, M. (2014). Applying complexity theory: A review to inform evaluation design. Evaluation and Program Planning, 45, 119-126. https://doi.org/10.1016/j.evalprogplan.2014.04.002 DOI: https://doi.org/10.1016/j.evalprogplan.2014.04.002
Wu, B., Hu, Y., Gu, X., & Lim, C. P. (2016). Professional development of new higher education teachers with information and communication technology in Shanghai: A Kirkpatrick’s evaluation approach. Journal of Educational Computing Research, 54(4), 531-562. DOI: https://doi.org/10.1177/0735633115621922