David A. Joyner
ABSTRACT
Traditionally, education relies on a linear relationship between enrollment and staff; rising enrollment dictates increases to staff with some expertise (such as teaching assistants, TAs) for evaluation. This relationship is expensive, so learning at scale has largely deemphasized expert evaluation and feedback. Two organizations, though, have used different models to scale up class size online while retaining this expert evaluation and feedback. In this paper, we analyze the methods these two organizations have used to increase enrollment while preserving scalability and feedback. We observe an academic program has scaled feedback with traditional TAs by relying on unique characteristics of its student body, while a commercial program has done so with a novel, network-based model. These successes show the potential of learning from experts at scale.
- Aleven, V., Mclaren, B. M., Sewall, J., & Koedinger, K. R. (2009). A new paradigm for intelligent tutoring systems: Example-tracing tutors. International Journal of Artificial Intelligence in Education, 19(2), 105--154.Google Scholar
Digital Library
- Boud, D., Cohen, R., & Sampson, J. (2001). Peer Learning in Higher Education: Learning from & with Each Other. Psychology Press.Google Scholar
- Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of educational research, 65(3), 245--281. Google ScholarCross Ref
- Carey, K. (2016, October 5). Georgia Tech's $7,000 Online Master's Degree Could Start a Revolution. New York Times, pp. A15.Google Scholar
- Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287--322. Google ScholarCross Ref
- Geigle, C., Zhai, C., & Ferguson, D. C. (2016, April). An Exploration of Automated Grading of Complex Assignments. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (pp. 351--360). ACM.Google Scholar
- Goodman, J., Melkers, J., & Pallais, A. (2016). Can Online Delivery Increase Access to Education? HKS Faculty Research Working Paper Series RWP16-035.Google Scholar
- Goel, A. & Joyner, D. A. (2016a). An Experiment in Teaching Cognitive Systems Online. In Haynes, D. (Ed.) International Journal for Scholarship of Technology-Enhanced Learning 1(1).Google Scholar
- Goel, A. & Joyner, D. (2016b). Formative Assessment and Implicit Feedback in Online Learning. Presentation, Learning with MOOCs III.Google Scholar
- Hext, J. B., & Winings, J. W. (1969). An automatic grading scheme for simple programming exercises. Communications of the ACM, 12(5), 272--275. Google ScholarDigital Library
- Jensen, J. C., Lee, E. A., & Seshia, S. A. (2013, April). Virtualizing cyber-physical systems: Bringing CPS to online education. In Proc. First Workshop on CPS Education (CPS-Ed).Google Scholar
- Joyner, D. A., Ashby, W., Irish, L., Lam, Y., Langston, J., Lupiani, I., ... & Bruckman, A. (2016, April). Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for Large Online Classrooms. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (pp. 399--408). ACM.Google Scholar
- Joyner, D. A., Goel, A., & Isbell, C. (2016). The Unexpected Pedagogical Benefits of Making Higher Education Accessible. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale. ACM. Google ScholarDigital Library
- Kendall, K., & Schussler, E. (2012). Does instructor type matter? Undergraduate student perception of graduate teaching assistants and professors. CBE-Life Sciences Education, 11(2), 187--199. Google ScholarCross Ref
- Kirschner, P., Sweller, J., & Clark, R. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75--86. Google ScholarCross Ref
- Kizilcec, R. F., & Schneider, E. (2015). Motivation as a lens to understand online learners: Toward datadriven design with the OLEI scale. ACM Transactions on Computer-Human Interaction (TOCHI), 22(2), 6. Google ScholarDigital Library
- Kolhe, P., Littman, M. L., & Isbell, C. L. (2016, April). Peer Reviewing Short Answers using Comparative Judgement. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (pp. 241--244). ACM.Google Scholar
- Kulkarni, C., Bernstein, M. S., & Klemmer, S. (2015). PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings from The Second ACM Conference on Learning @ Scale. ACM. Google ScholarDigital Library
- Lu, Y., Warren, J., Jermaine, C., Chaudhuri, S., & Rixner, S. (2015, May). Grading the Graders: Motivating Peer Graders in a MOOC. In Proceedings of the 24th International Conference on World Wide Web (pp. 680--690). ACM. Google ScholarDigital Library
- Lundstrom, K., & Baker, W. (2009). To give is better than to receive: The benefits of peer review to the reviewer's own writing. Journal of Second Language Writing, 18(1), 30--43. Google ScholarCross Ref
- Murray, T., Blessing, S., & Ainsworth, S. (2003). Authoring tools for advanced technology learning environments: Toward cost-effective adaptive, interactive and intelligent educational software. Springer Science & Business Media. Google ScholarCross Ref
- Polson, M. C., & Richardson, J. J. (2013). Foundations of intelligent tutoring systems. Psychology Press.Google Scholar
- Raman, K., & Joachims, T. (2015). Bayesian Ordinal Peer Grading. In Proceedings from The Second ACM Conference on Learning @ Scale. ACM. 149--156. Google ScholarDigital Library
- Renkl, A. (2002). Worked-out examples: Instructional explanations support learning by self-explanations. Learning and Instruction, 12(5), 529--556. Google ScholarCross Ref
- Salzmann, C., Gillet, D., & Piguet, Y. (2016, February). MOOLs for MOOCs: A first edX scalable implementation. In 2016 13th International Conference on Remote Engineering and Virtual Instrumentation (REV) (pp. 246--251). IEEE. Google ScholarCross Ref
- Saraf, K. & Smith, C. (2016, October 5). "Coursera's superheroes: Meet the Mentor team." Coursera Blog. Retrieved from coursera.tumblr.com/post/151389966612/Google Scholar
- Srikant, S., & Aggarwal, V. (2014, August). A system to grade computer programming skills using machine learning. In Proceedings of the 20th ACM International Conference on Knowledge Discovery and Data Mining (pp. 1887--1896). ACM. Google ScholarDigital Library
- VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227--265.Google ScholarDigital Library
- VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197--221. Google ScholarCross Ref
- Van Gog, T., & Rummel, N. (2010). Example-based learning: Integrating cognitive and social-cognitive research perspectives. Educational Psychology Review, 22(2), 155--174. Google ScholarCross Ref
- Van Zundert, M., Sluijsmans, D., & Van Merriënboer, J. (2010). Effective peer assessment processes: Research findings and future directions. Learning and Instruction, 20(4), 270--279. Google Scholar
- Vogelsang, T., & Ruppertz, L. (2015, March). On the validity of peer grading and a cloud teaching assistant system. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (pp. 41--50). ACM. Google ScholarDigital Library
- Yaron, D., Karabinos, M., Lange, D., Greeno, J. G., & Leinhardt, G. (2010). The ChemCollective-virtual labs for introductory chemistry courses. Science, 328(5978), 584--585. Google ScholarCross Ref
Index Terms
- Scaling Expert Feedback: Two Case Studies
Recommendations
Adapting the TPACK Framework for Online Teaching Within Higher Education
There are ongoing obstacles in accurately assessing and effectively applying TPACK within online higher education contexts. In this study, we adapt the original TPACK constructs for application within online higher education, and share the results of a ...
Scaling up for CS10K: Teaching and Supporting New Computer Science High School Teachers (Abstract Only)
SIGCSE '16: Proceedings of the 47th ACM Technical Symposium on Computing Science EducationIncreasing need for computing expertise in our everyday lives and in the workforce, paired with declining enrollments in computing by women and underrepresented minorities have made it critical to provide students with experiences in computing before ...
Peer versus expert feedback: An investigation of the quality of peer feedback among secondary school students
Few studies have focused on peer assessment at the secondary school level. Consequently, we know very little about the quality of the feedback secondary school students can produce and its potential usefulness. This study was implemented in the context ...
Comments