Skip to main content
SpringerLink
Log in

Augmented incremental recognition of online handwritten mathematical expressions

  • Original Paper
  • Published:
International Journal on Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

This paper presents an augmented incremental recognition method for online handwritten mathematical expressions (MEs). If an ME is recognized after all strokes are written (batch recognition), the waiting time increases significantly when the ME becomes longer. On the other hand, the pure incremental recognition method recognizes an ME whenever a new single stroke is input. It shortens the waiting time but degrades the recognition rate due to the limited context. Thus, we propose an augmented incremental recognition method that not only maintains the advantage of the two methods but also reduces their weaknesses. The proposed method has two main features: one is to process the latest stroke, and the other is to find the erroneous segmentations and recognitions in the recent strokes and correct them. In the first process, the segmentation and the recognition by Cocke–Younger–Kasami (CYK) algorithm are only executed for the latest stroke. In the second process, all the previous segmentations are updated if they are significantly changed after the latest stroke is input, and then, all the symbols related to the updated segmentations are updated with their recognition scores. These changes are reflected in the CYK table. In addition, the waiting time is further reduced by employing multi-thread processes. Experiments on our dataset and the CROHME datasets show the effectiveness of this augmented incremental recognition method, which not only maintains recognition rate even compared with the batch recognition method but also reduces the waiting time to a very small level.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Anderson, R.H.: Syntax-directed recognition of hand-printed two-dimensional mathematics. In: Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc. Symposium, pp. 436-459, Washington, USA (1967)

  2. Chang, S.K.: A method for the structural analysis of two-dimensional mathematical expressions. Int. J. Inf. Sci. 2(3), 253–272 (1970)

    MATH  Google Scholar 

  3. lvaro, F., Snchez, Bened, J.: Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models. Pattern Recognit. Lett. 31, 58–67 (2014)

    Google Scholar 

  4. Awal, A.M., Mouchre, H., Viard-Gaudin, C.: A global learning approach for an online handwritten mathematical expression recognition system. Pattern Recognit. Lett. 35, 68–77 (2012)

    Article  Google Scholar 

  5. Yamamoto, R., Sako, S., Nishimoto, T., Sagayama, S.: Online recognition of handwritten mathematical expressions based on stroke-based stochastic context-free grammar. In: Proceedings of the 7th International Conference on Document Analysis and Recognition, vol. 1, pp. 249-254, Edinburgh, UK (2006)

  6. Simistira, F., Katsouros, V., Carayannis, G.: Recognition of online handwritten mathematical formulas using probabilistic SVMs and stochastic context free grammars. Pattern Recognit. Lett. 53, 85–92 (2015)

    Article  Google Scholar 

  7. Le, A.D., Nakagawa, M.: A system for recognizing online handwritten mathematical expressions by using improved structural analysis. Int. J. Doc. Anal. Recognit. 19(4), 305–319 (2016)

    Article  Google Scholar 

  8. Okamoto, M., Miao, B.: Recognition of mathematical expressions by using the layout structure of symbols. In: Proceeding of the 1st International Conference on Document Analysis and Recognition, pp. 242–250, Saint-Malo, France (1991)

  9. Zanibbi, R., Blostein, D., Cordy, J.R.: Recognizing mathematical expressions using tree transformation. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1455–1467 (2002)

    Article  Google Scholar 

  10. Hu, L., Zanibbi, R.: MST-based Visual Parsing of Online Handwritten Mathematical Expressions. In: Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition, pp. 337–342, Shenzhen, China (2016)

  11. Julca-Aguilar, F., Mouchre, H., Viard-Gaudin, C., Hirata, N.S.T.: Top-Down Online Handwritten Mathematical Expression Parsing with Graph Grammar. In: Proceedings of the 20th Iberoamerican Congress on Pattern Recognition, pp. 444–451, Montevideo, Uruguay (2015)

  12. Garain, U., Chaudhuri, B.B.: Recognition of online handwritten mathematical expressions. IEEE Trans. Syst. Man Cybern. B Cybern. 34(6), 2366–2376 (2004)

    Article  Google Scholar 

  13. Le, A.D., Nakagawa, M.: A system for recognizing online handwritten mathematical expressions by using improved structural analysis. Int. J. Doc. Anal. Recognit. 19(4), 305–319 (2016)

    Article  Google Scholar 

  14. Zhang, J., Du, J., Zhang, S., Liu, D., Hu, Y., Hu, J., Wei, S., Dai, L.: Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recognit. Lett. 71, 196–206 (2017)

    Article  Google Scholar 

  15. MacLean, S., Labahn, G.: A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. Int. J. Doc. Anal. Recognit. 16(2), 139–163 (2013)

    Article  Google Scholar 

  16. Unger, S.H.: A global parser for context-free phrase structure grammars. Commun. ACM 11(4), 240–247 (1968)

    Article  Google Scholar 

  17. Predovic, G., Abdulkader, A., Dresevic, B., Viola, P. A, Vukosavljevic, M.: Recognition of mathematical expressions. U.S. Patent US8009915 B2 (2011)

  18. Vuong, B.Q., Hui, S.C., He, Y.: Progressive structural analysis for dynamic recognition of on-line handwritten mathematical expressions. Pattern Recognit. Lett. 29(5), 647–655 (2008)

    Article  Google Scholar 

  19. Phan, K.M., Nguyen, C.T., Le, A.D., Nakagawa, M.: An incremental recognition method for online handwritten mathematical expressions. In: Proceedings of the 3rd IAPR Asian Conference on Pattern Recognition, pp. 171–175, Kuala Lumpur, Malaysia (2015)

  20. Phan, K.M., Le, A.D., Nakagawa, M.: Semi-incremental recognition of online handwritten mathematical expressions. In: Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition, pp. 258–264, Shenzhen, China (2016)

  21. Nakagawa, M., Machii, K., Kato, N., Souya, T.: Lazy recognition as a principle of pen interfaces. In: Proceedings of the ACM INTERCHI, pp. 89–90, Amsterdam, Netherlands (1993)

  22. Le, A.D., Nakagawa, M.: Comparison of parsing algorithms for recognizing online handwritten mathematical expressions. In: Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition, pp. 390–394, Shenzhen, China (2016)

  23. Cocke, J., Schwartz, J.T.: Programming Languages and Their Compilers: Preliminary Notes, 2nd edn. Courant Institute of Mathematical Sciences, New York (1970)

    MATH  Google Scholar 

  24. Younger, D.H.: Recognition and parsing of context-free languages in time \(n^3\). Inf. Comput. 10(2), 189–208 (1967)

    MATH  Google Scholar 

  25. Kasami, T.: An Efficient Recognition and Syntax-Analysis Algorithm for Context-Free Languages. University of Illinois Coordinated Science Laboratory, Amsterdam (1966)

    Google Scholar 

  26. Zhu, B., Gao, J., Nakagawa, M.: Objective function design for MCE-based combination of on-line and off-line character recognizers for on-line handwritten Japanese text recognition. In: Proceedings of the 11th International Conference on Document Analysis and Recognition, pp. 594–599, Beijing, China (2011)

  27. Mouchre, H., Viard-Gaudin, C., Zanibbi, R., Garain, U., Kim, D.H., Kim, J.H.: ICDAR 2013 CROHME: third international competition on recognition of online handwritten mathematical expressions. In: Proceedings of the 12th International Conference on Document Analysis and Recognition, pp. 1428–1432, Washington, USA (2013)

  28. Mouchre, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions (CROHME 2014). In: Proceedings of the 14th International Conference on Frontiers in Handwriting Recognition, pp. 791–796, Heraklion, Greece (2014)

  29. Mouchre, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: ICFHR2016 CROHME: competition on recognition of online handwritten mathematical expressions. In: Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition, pp. 607–612, Shenzhen, China (2016)

  30. Nielsen, J.: Usability Engineering. Academic Press Inc, Boston (1993)

    MATH  Google Scholar 

  31. Shneiderman, B.: Response time and display rate in human performance with computer. ACM Comput. Surv. 16(3), 265–285 (1984)

    Article  Google Scholar 

  32. Miller, R.B.: Response time in man-computer conversational transactions. In: Proceeding of the AFIPS Fall Joint Computer Conference, vol. 33, pp. 267–277, San Francisco, USA (1968)

  33. Liu, C.L., Jaeger, S., Nakagawa, M.: Online recognition of Chinese characters: the state-of-the-art. IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 198–213 (2004)

    Article  Google Scholar 

  34. Plamondon, R., Srihari, S.N.: On-line and off-line handwriting recognition: a comprehensive survey. IEEE Trans. Pattern Anal. Mach. Intell. 22(1), 63–84 (2000)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer and Information Sciences, Tokyo University of Agriculture and Technology, 2-24-16, Naka-cho, Koganei-shi, Tokyo, 184-8588, Japan

    Khanh Minh Phan & Masaki Nakagawa

  2. NTT Hi-Tech Institute, Nguyen Tat Thanh University, 300A Nguyen Tat Thanh, District 4, Ho Chi Minh city, Vietnam

    Anh Duc Le

  3. Department of Computer Science, AGH University of Science and Technology, Al. Mickiewicza 30, 30-059, Kraków, Poland

    Bipin Indurkhya

Authors
  1. Khanh Minh Phan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anh Duc Le
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bipin Indurkhya
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Masaki Nakagawa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Masaki Nakagawa.

Appendix

Appendix

See Tables 7, 8 and Fig. 13.

Table 7 Geometric features for symbol segmentation
Full size table
Fig. 13
figure 13

Extracted features listed in Table 8

Full size image
Table 8 Terms for features
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Phan, K.M., Le, A.D., Indurkhya, B. et al. Augmented incremental recognition of online handwritten mathematical expressions. IJDAR 21, 253–268 (2018). https://doi.org/10.1007/s10032-018-0306-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10032-018-0306-1

Keywords

Search

Navigation