Abstract
We present a novel sub-stroke level transformer approach to convert offline images of handwriting to online. We start by extracting sub-strokes from the offline images by inferring a skeleton with a CNN and applying a basic cutting algorithm. We introduce sub-stroke embeddings by encoding the sub-stroke point sequence with a Sub-stroke Encoding Transformer (SET). The embeddings are then fed to the Sub-strokes ORdering Transformer (SORT) which predicts the discrete sub-strokes ordering and the pen state. By constraining the Transformer input and output to the inferred sub-strokes, the recovered online is highly precise. We evaluate our method on Latin words from the IRONOFF dataset and on maths expressions from CROHME dataset. We measure the performance with two criteria: fidelity with Dynamic Time Warping (DTW) and semantic coherence using recognition rate. Our method outperforms the state-of-the-art in both datasets, achieving a word recognition rate of \(81.06\%\) and a 2.41 DTW on IRONOFF and an expression recognition rate of \(62.00\%\) and a DTW of 13.93 on CROHME 2019. This work constitutes an important milestone toward full offline document conversion to online.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
MyScript iink SDK is available at https://developer.myscript.com/docs/interactive-ink/2.0/overview/about/.
References
Aksan, E., Deselaers, T., Tagliasacchi, A., Hilliges, O.: CoSE: compositional stroke embeddings. In: Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 2020, pp. 10041–10052. Curran Associates Inc., Red Hook, December 2020. https://proceedings.neurips.cc/paper/2020/file/723e8f97fde15f7a8d5ff8d558ea3f16-Paper.pdf
Archibald, Taylor, Poggemann, Mason, Chan, Aaron, Martinez, Tony: TRACE: a differentiable approach to line-level stroke recovery for offline handwritten text. In: Lladós, Josep, Lopresti, Daniel, Uchida, Seiichi (eds.) ICDAR 2021. LNCS, vol. 12823, pp. 414–429. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86334-0_27
Bhunia, A.K., et al.: Handwriting trajectory recovery using end-to-end deep encoder-decoder network. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3639–3644, August 2018. https://doi.org/10.1109/ICPR.2018.8546093
Chan, C.: Stroke extraction for offline handwritten mathematical expression recognition. IEEE Access 8, 61565–61575 (2020). https://doi.org/10.1109/ACCESS.2020.2984627
Doermann, D., Intrator, N., Rivin, E., Steinherz, T.: Hidden loop recovery for handwriting recognition. In: Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition, pp. 375–380, August 2002. https://doi.org/10.1109/IWFHR.2002.1030939
Egiazarian, Vage, et al.: Deep vectorization of technical drawings. In: Vedaldi, Andrea, Bischof, Horst, Brox, Thomas, Frahm, Jan-Michael. (eds.) ECCV 2020. LNCS, vol. 12358, pp. 582–598. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58601-0_35
Guo, Y., Zhang, Z., Han, C., Hu, W., Li, C., Wong, T.T.: Deep line drawing vectorization via line subdivision and topology reconstruction. Comput. Graph. Forum 38(7), 81–90 (2019). https://doi.org/10.1111/cgf.13818
Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR 2018 (2018). https://openreview.net/pdf?id=Hy6GHpkCW
Holten, D., van Wijk, J.J.: A user study on visualizing directed edges in graphs. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2009, pp. 2299–2308. Association for Computing Machinery, New York, April 2009. https://doi.org/10.1145/1518701.1519054
Jager, S.: Recovering writing traces in off-line handwriting recognition: using a global optimization technique. In: Proceedings of 13th International Conference on Pattern Recognition, vol. 3, pp. 150–154, August 1996. https://doi.org/10.1109/ICPR.1996.546812
Ji, B., Chen, T.: Generative Adversarial Network for Handwritten Text, February 2020
Mahdavi, M., Zanibbi, R., Mouchere, H., Viard-Gaudin, C., Garain, U.: ICDAR 2019 CROHME + TFD: competition on recognition of handwritten mathematical expressions and typeset formula detection. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1533–1538, September 2019. https://doi.org/10.1109/ICDAR.2019.00247
Marti, U.V., Bunke, H.: The IAM-database: an English sentence database for offline handwriting recognition. Int. J. Doc. Anal. Recogn. 5(1), 39–46 (2002). https://doi.org/10.1007/s100320200071
Mohamed Moussa, Elmokhtar, Lelore, Thibault, Mouchère, Harold: Applying end-to-end trainable approach on stroke extraction in handwritten math expressions images. In: Lladós, Josep, Lopresti, Daniel, Uchida, Seiichi (eds.) ICDAR 2021. LNCS, vol. 12823, pp. 445–458. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86334-0_29
Mohamed Moussa, E., Lelore, T., Mouchère, H.: Point to Segment Distance DTW for online handwriting signals matching. In: Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods, pp. 850–855. SCITEPRESS - Science and Technology Publications, Lisbon, Portugal (2023). https://doi.org/10.5220/0011672600003411
Nguyen, V., Blumenstein, M.: Techniques for static handwriting trajectory recovery: a survey. In: Proceedings of the 9th IAPR International Workshop on Document Analysis Systems, DAS 2010, pp. 463–470. Association for Computing Machinery, New York, June 2010. https://doi.org/10.1145/1815330.1815390
Qiao, Y., Nishiara, M., Yasuhara, M.: A framework toward restoration of writing order from single-stroked handwriting image. IEEE Trans. Pattern Anal. Mach. Intell. 28(11), 1724–1737 (2006). https://doi.org/10.1109/TPAMI.2006.216
Seki, Y.: Online and offline data collection of Japanese handwriting. In: 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), vol. 8, pp. 13–18, September 2019. https://doi.org/10.1109/ICDARW.2019.70135
Simo-Serra, E., Iizuka, S., Ishikawa, H.: Mastering Sketching: adversarial augmentation for structured prediction. ACM Trans. Graph. 37(1), 11:1–11:13 (2018). https://doi.org/10.1145/3132703
Viard-Gaudin, C., Lallican, P.M., Knerr, S., Binter, P.: The IRESTE On/Off (IRONOFF) dual handwriting database. In: Proceedings of the Fifth International Conference on Document Analysis and Recognition, ICDAR ’99 (Cat. No.PR00318), pp. 455–458, September 1999. https://doi.org/10.1109/ICDAR.1999.791823
Vinciarelli, A., Perone, M.: Combining online and offline handwriting recognition. In: Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings, pp. 844–848, August 2003. https://doi.org/10.1109/ICDAR.2003.1227781
Zhang, J., Du, J., Dai, L.: Track, Attend, and Parse (TAP): an end-to-end framework for online handwritten mathematical expression recognition. IEEE Trans. Multimedia 21(1), 221–233 (2019). https://doi.org/10.1109/TMM.2018.2844689
Zhang, T.Y., Suen, C.Y.: A fast parallel algorithm for thinning digital patterns. Commun. ACM 27(3), 236–239 (1984). https://doi.org/10.1145/357994.358023
Zhao, B., Yang, M., Tao, J.: Pen tip motion prediction for handwriting drawing order recovery using deep neural network. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 704–709, August 2018. https://doi.org/10.1109/ICPR.2018.8546086
Zhao, W., Gao, L.: CoMER: modeling coverage for transformer-based handwritten mathematical expression recognition. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, pp. 392–408. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19815-1_23
Acknowledgements
We would like to express our gratitude to Robin Mélinand for his invaluable feedback and suggestions for this article.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Mohamed Moussa, E., Lelore, T., Mouchère, H. (2023). SET, SORT! A Novel Sub-stroke Level Transformers for Offline Handwriting to Online Conversion. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds) Document Analysis and Recognition - ICDAR 2023. ICDAR 2023. Lecture Notes in Computer Science, vol 14187. Springer, Cham. https://doi.org/10.1007/978-3-031-41676-7_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-41676-7_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-41675-0
Online ISBN: 978-3-031-41676-7
eBook Packages: Computer ScienceComputer Science (R0)
