Abstract
Although pursuing accuracy is important in language learning or teaching, knowing what types of errors interfere with communication and what types do not would be more beneficial for efficiently enhancing communicative competence. Language learners could be greatly helped by a system that detected errors in learner language and automatically measured their effect on intelligibility. In this paper, we reported our attempt, based on machine learning, to measure the intelligibility of learner language. In the learning process, the system referred to the BLEU and NIST scores between the learners’ original sentences and their back translation (or corrected sentences), the log-probability of the parse, sentence length, and error types (manually or automatically assigned) as a key feature. We found that the system can distinguish between intelligible sentences and others (unnatural and unintelligible) rather successfully, but still has a lot of difficulties in distinguishing the three levels of intelligibility.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Briscoe, E., Carroll, J.: Robust Accurate Statistical Annotation of General Text. In: Proceedings of the Third International Conference on Language Resources and Evaluation, Las Palmas, Gran Canaria, pp. 1499–1504 (2002)
Burnstein, J.: The E-rater scoring engine: Automated Essay Scoring with Natural Language Processing. In: Shermis, M.D., Burnstein, J. (eds.) Automated Essay Scoring: A Cross-Disciplinary Perspective, pp. 113–122. Lawrence Erlbaum Associates, Inc., New Jersey (2003)
Canale, M., Swain, M.: Theoretical Bases of Communicative Approach to Second Language Teaching and Testing. Applied Linguistics 1, 1–47 (1980)
Ellis, R.: Task-based Language Learning and Teaching. Oxford University Press, Oxford (2003)
Izumi, E., Saiga, T., Uchimoto, K., Supnithi, T., Isahara, H.: Automatic Error Detection in the Japanese Learners English Spoken Data. In: Companion Volume of the Proceedings of the 41st Annual Meeting of the Association of Computational Linguistics (ACL 2003), Japan, pp. 145–148 (2003)
Izumi, E., Uchimoto, K., Isahara, H.: Standard Speaking Test (SST) Speech Corpus of Japanese Learners’ English and Automatic Detection of Learners’ Errors. International Computer Achieves of Modern and Medieval English (ICAME) Journal 28, 31–48 (2004)
Joachims, T.: Text Categorization with Support Vector Machines. In: Nédellec, C., Rouveirol, C. (eds.) ECML 1998. LNCS, vol. 1398, pp. 137–142. Springer, Heidelberg (1998)
Kudo, T., Matsumoto, Y.: Chunking with Support Vector Machines. In: Proceedings of the 2nd Meeting of the North American Chapter of the Association of Computational Linguistics, Pittsburgh, pp. 192–199 (2001)
Larkey, L.S.: Automatic Essay Grading Using Text Categorization Techniques. In: Proceedings of the 21st Annual International Conference on Research and Development in Information Retrieval (SIGIR 1998), Australia, pp. 90–95 (1998)
NIST: Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurrence Statistics, Philadelphia (2002)
Page, E.B.: Computer Grading of Student Prose, Using Modern Concepts and Software. Journal of Experimental Education 62, 127–142 (1994)
Page, E.B., Petersen, N.S.: The Computer Moves into Essay Grading: Updating the Ancient Test. Phi Delta Kappan 76(7), 561–565 (1995)
Papineni, K., Roukos, S., Ward, T., Zhu, W.: BLEU: A Method for Automatic Evaluation of Machine Translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), USA, pp. 311–318 (2002)
Shermis, M.D., Burnstein, J. (eds.): Automated Essay Scoring: A Cross-Disciplinary Perspective, pp. xiii–xvi. Lawrence Erlbaum Associates, Inc., New Jersey (2003)
Skehan, P.: A Cognitive Approach to Language Learning. Oxford University Press, Oxford (1998)
Uchimoto, K., Hayashida, N., Ishida, T., Isahara, H.: Automatic Rating of Machine Translatability. In: Proceedings of the MT Summit X, Thailand, pp. 235–242 (2005)
Williams, R.: Automated Essay Grading: An Evaluation of Four Conceptual Models. In: Proceedings of the 10th Annual Teaching Learning Forum, Curtin University of Technology, Perth, pp. 7–9 (2001), http://lsn.curtin.edu.au/tlf/tlf2001/contents.html
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Izumi, E., Uchimoto, K., Isahara, H. (2006). Measuring Intelligibility of Japanese Learner English. In: Salakoski, T., Ginter, F., Pyysalo, S., Pahikkala, T. (eds) Advances in Natural Language Processing. FinTAL 2006. Lecture Notes in Computer Science(), vol 4139. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11816508_48
Download citation
DOI: https://doi.org/10.1007/11816508_48
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-37334-6
Online ISBN: 978-3-540-37336-0
eBook Packages: Computer ScienceComputer Science (R0)