Skip to main content
SpringerLink
Log in

Novel Benchmark Data Set for Automatic Error Detection and Correction

  • Conference paper
  • First Online:
Natural Language Processing and Information Systems (NLDB 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13913))

  • 981 Accesses

Abstract

The present paper introduces a novel benchmark data set for automatic error detection as well as error correction in text documents based on language models or other techniques. The data set contains a large number of sentences from various domains annotated with various types of errors (orthographic, grammatical, punctuation, and typography errors). The paper presents the method used to collect and annotate the documents, provides statistical analyses of the data set’s properties and evaluates two preliminary baseline models for automatic error detection on a specific benchmark task. The results show, on the one hand, the effectiveness of the proposed data set for the evaluation of automatic error detection systems. On the other hand, these initial analyses also reveal that the data set contains challenging cases that are difficult to detect. Finally, the paper discusses potential applications of the proposed data set in the development and research of error detection and error correction systems.

Supported by Innosuisse project 101.128 IP-ICT: A PROSE - Advanced PROofreading SErvices and Rotstift AG.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The fourth official language of the country is Rhaeto-Romanic and is used relatively rarely.

  2. 2.

    Due to several technical problems currently arising on the PDF documents.

  3. 3.

    All examples are in the German language.

  4. 4.

    https://github.com/Pattern-Recognition-Group-UniBe.

References

  1. Rei, M., Felice, M., Yuan, Z., Briscoe, T.: Artificial error generation with machine translation and syntactic patterns. CoRR, abs/1707.05236 (2017)

    Google Scholar 

  2. Bryant, C., Yuan, Z., Qorib, M.R., Cao, H., Ng, H.T., Briscoe, T.: Grammatical error correction: a survey of the state of the art (2022)

    Google Scholar 

  3. Hládek, D., Staš, J., Pleva, M.: Survey of automatic spelling correction. Electronics 9(10), 1670 (2020)

    Article  Google Scholar 

  4. Pirinen, T.A., Lindén, K.: State-of-the-art in weighted finite-state spell-checking. In: Gelbukh, A. (ed.) CICLing 2014, Part II. LNCS, vol. 8404, pp. 519–532. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-54903-8_43

    Chapter  Google Scholar 

  5. Ali, M., Khalid, S., Rana, M.I., Azhar, F.: A probabilistic framework for short text classification. In: IEEE 8th Annual Computing and Communication Workshop and Conference, CCWC 2018, Las Vegas, NV, USA, 8–10 January 2018, pp. 742–747. IEEE (2018)

    Google Scholar 

  6. Wilcox-O’Hearn, A., Hirst, G., Budanitsky, A.: Real-word spelling correction with trigrams: a reconsideration of the mays, Damerau, and Mercer model. In: Gelbukh, A. (ed.) CICLing 2008. LNCS, vol. 4919, pp. 605–616. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78135-6_52

    Chapter  Google Scholar 

  7. Fossati, D., Di Eugenio, B.: A mixed trigrams approach for context sensitive spell checking. In: Gelbukh, A. (ed.) CICLing 2007. LNCS, vol. 4394, pp. 623–633. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-70939-8_55

    Chapter  Google Scholar 

  8. Islam, A., Inkpen, D.: Real-word spelling correction using google web 1t 3-grams. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6–7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1241–1249. ACL (2009)

    Google Scholar 

  9. Rozovskaya, A., Roth, D.: Grammatical error correction: machine translation and classifiers. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, 7–12 August 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics (2016)

    Google Scholar 

  10. Yannakoudakis, H., Rei, M., Andersen, Ø.E., Yuan, Z.: Neural sequence-labelling models for grammatical error correction. In: Palmer, M., Hwa, R., Riedel, S. (eds.) Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, 9–11 September 2017, pp. 2795–2806. Association for Computational Linguistics (2017)

    Google Scholar 

  11. Yuan, Z., Briscoe, T.: Grammatical error correction using neural machine translation. In: Knight, K., Nenkova, A., Rambow, O. (eds.) NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, 12–17 June 2016, pp. 380–386. The Association for Computational Linguistics (2016)

    Google Scholar 

  12. Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Moschitti, A., Pang, B., Daelemans, W. (eds.) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, 25–29 October 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1724–1734. ACL (2014)

    Google Scholar 

  13. Chollampatt, S., Ng, H.T.: A multilayer convolutional encoder-decoder neural network for grammatical error correction. In: McIlraith, S.A., Weinberger, K.Q. (eds.) Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, 2–7 February 2018, pp. 5755–5762. AAAI Press (2018)

    Google Scholar 

  14. Zhang, S., Huang, H., Liu, J., Li, H.: Spelling error correction with soft-masked BERT. CoRR, abs/2005.07421 (2020)

    Google Scholar 

  15. Moslem, Y., Haque, R., Way, A.: Adaptive machine translation with large language models. CoRR, abs/2301.13294 (2023)

    Google Scholar 

  16. Bryant, C., Briscoe, T.: Language model based grammatical error correction without annotated training data. In: Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 247–253, New Orleans, Louisiana. Association for Computational Linguistics (2018)

    Google Scholar 

  17. Alikaniotis, D., Raheja, V.: The unreasonable effectiveness of transformer language models in grammatical error correction. CoRR, abs/1906.01733 (2019)

    Google Scholar 

  18. Tan, M., Chen, D., Li, Z., Wang, P.: Spelling error correction with BERT based on character-phonetic. In: 2020 IEEE 6th International Conference on Computer and Communications (ICCC), pp. 1146–1150. IEEE (2020)

    Google Scholar 

  19. Bryant, C., Yuan, Z., Qorib, M.R., Cao, H., Ng, H.T., Briscoe, T.: Grammatical error correction: a survey of the state of the art. CoRR, abs/2211.05166 (2022)

    Google Scholar 

  20. Dahlmeier, D., Ng, H.T., Wu, S.M.: Building a large annotated corpus of learner English: the NUS corpus of learner English. In: Tetreault, J.R., Burstein, J., Leacock, C. (eds.) Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, BEA@NAACL-HLT 2013, 13 June 2013, Atlanta, Georgia, USA, pp. 22–31. The Association for Computer Linguistics (2013)

    Google Scholar 

  21. Bryant, C., Felice, M., Andersen, Ø.E., Briscoe, T.: The BEA-2019 shared task on grammatical error correction. In: Yannakoudakis, H., Kochmar, E., Leacock, C., Madnani, N., Pilán, I., Zesch, T. (eds.) Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, BEA@ACL 2019, Florence, Italy, 2 August 2019, pp. 52–75. Association for Computational Linguistics (2019)

    Google Scholar 

  22. Etoori, P., Chinnakotla, M., Mamidi, R.: Automatic spelling correction for resource-scarce languages using deep learning. In: Shwartz, V., Tabassum, J., Voigt, R., Che, W., de Marneffe, M.-C., Nissim, M. (eds.) Proceedings of ACL 2018, Melbourne, Australia, 15–20 July 2018, Student Research Workshop, pp. 146–152. Association for Computational Linguistics (2018)

    Google Scholar 

  23. Näther, M.: An in-depth comparison of 14 spelling correction tools on a common benchmark. In: Calzolari, N., et al. (eds.) Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, 11–16 May 2020, pp. 1849–1857. European Language Resources Association (2020)

    Google Scholar 

  24. Xue, L., et al.: mT5: a massively multilingual pre-trained text-to-text transformer. In: Toutanova, K., et al. (eds.) Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, 6–11 June 2021, pp. 483–498. Association for Computational Linguistics (2021)

    Google Scholar 

  25. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 140:1–140:67 (2020)

    Google Scholar 

  26. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Computer Science, University of Bern, 3012, Bern, Switzerland

    Corina Masanti & Kaspar Riesen

  2. Institute for Informations Systems, University of Applied Sciences and Arts Northwestern Switzerland, 4600, Olten, Switzerland

    Hans-Friedrich Witschel & Kaspar Riesen

Authors
  1. Corina Masanti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hans-Friedrich Witschel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kaspar Riesen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Corina Masanti .

Editor information

Editors and Affiliations

  1. Conservatoire National des Arts et Métiers, Paris, France

    Elisabeth Métais

  2. University of Derby, Derby, UK

    Farid Meziane

  3. Oakland University, Rochester, NY, USA

    Vijayan Sugumaran

  4. University of Derby, Derby, UK

    Warren Manning

  5. University of Derby, Derby, UK

    Stephan Reiff-Marganiec

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Masanti, C., Witschel, HF., Riesen, K. (2023). Novel Benchmark Data Set for Automatic Error Detection and Correction. In: Métais, E., Meziane, F., Sugumaran, V., Manning, W., Reiff-Marganiec, S. (eds) Natural Language Processing and Information Systems. NLDB 2023. Lecture Notes in Computer Science, vol 13913. Springer, Cham. https://doi.org/10.1007/978-3-031-35320-8_38

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-35320-8_38

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-35319-2

  • Online ISBN: 978-3-031-35320-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation