Skip to main content
SpringerLink
Log in

CMeIE: Construction and Evaluation of Chinese Medical Information Extraction Dataset

  • Conference paper
  • First Online:
Natural Language Processing and Chinese Computing (NLPCC 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12430))

Abstract

In this paper, we present the Chinese Medical Information Extraction (CMeIE) dataset, consisting of 28, 008 sentences, 85, 282 triplets, 11 entities, and 44 relations derived from medical textbooks and clinical practices, constructed by several rounds of manual annotation. Additionally, we evaluate performances of the most recent state-of-the-art frameworks and pre-trained language models for the joint extraction of entities and relations task on the CMeIE dataset. Experiment results show that even these most advanced models still have a large space to improve on our dataset; currently, the best F1 score on the dataset is 58.44%. Our analysis points out several challenges and multiple potential future research directions for the task specialized in the medical domain.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/google-research/bert.

  2. 2.

    https://github.com/ymcui/Chinese-BERT-wwm.

  3. 3.

    https://github.com/PaddlePaddle/ERNIE.

References

  1. WATC: Anatomical therapeutic chemical classification system. WHO Collaborating Center for Drug Statistics (2009)

    Google Scholar 

  2. Byambasuren, O., et al.: Preliminary study on the construction of Chinese medical knowledge graph. J. Chin. Inf. Process. 33(10), 1–7 (2019)

    Google Scholar 

  3. Cui, Y., et al.: Pre-training with whole word masking for Chinese Bert. arXiv preprint arXiv:1906.08101 (2019)

  4. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186 (2019)

    Google Scholar 

  5. Fu, T.J., Li, P.H., Ma, W.Y.: GraphRel: modeling text as relational graphs for joint entity and relation extraction. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1409–1418 (2019)

    Google Scholar 

  6. Gardent, C., Shimorina, A., Narayan, S., Perez-Beltrachini, L.: Creating training corpora for NLG micro-planners. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 179–188 (2017)

    Google Scholar 

  7. GuoDong, Z., Jian, S., Jie, Z., Min, Z.: Exploring various knowledge in relation extraction. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pp. 427–434. Association for Computational Linguistics (2005)

    Google Scholar 

  8. Hanisch, D., Fundel, K., Mevissen, H.T., Zimmer, R., Fluck, J.: ProMiner: rule-based protein and gene entity recognition. BMC Bioinform. 6(1), S14 (2005)

    Google Scholar 

  9. Hripcsak, G., Rothschild, A.S.: Agreement, the f-measure, and reliability in information retrieval. J. Am. Med. Inf. Assoc. 12(3), 296–298 (2005)

    Article  Google Scholar 

  10. Katiyar, A., Cardie, C.: Going out on a limb: joint extraction of entity mentions and relations without dependency trees. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 917–928 (2017)

    Google Scholar 

  11. Li, Q., Ji, H.: Incremental joint extraction of entity mentions and relations. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 402–412 (2014)

    Google Scholar 

  12. Lipscomb, C.E.: Medical subject headings (mesh). Bull. Med. Lib. Assoc. 88(3), 265 (2000)

    Google Scholar 

  13. Liu, Y., et al.: Roberta: a robustly optimized Bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  14. Miwa, M., Sasaki, Y.: Modeling joint entity and relation extraction with table representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1858–1869 (2014)

    Google Scholar 

  15. Nikfarjam, A., Emadzadeh, E., Gonzalez, G.: Towards generating a patient’s timeline: extracting temporal relationships from clinical notes. J. Biomed. Inf. 46, S40–S47 (2013)

    Article  Google Scholar 

  16. Ogren, P.V., Savova, G.K., Chute, C.G., et al.: Constructing evaluation corpora for automated clinical named entity recognition. LREC 8, 3143–3150 (2008)

    Google Scholar 

  17. Riedel, S., Yao, L., McCallum, A.: Modeling relations and their mentions without labeled text. In: Balcázar, J.L., Bonchi, F., Gionis, A., Sebag, M. (eds.) ECML PKDD 2010. LNCS (LNAI), vol. 6323, pp. 148–163. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15939-8_10

    Chapter  Google Scholar 

  18. Savova, G.K., et al.: Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications. J. Am. Med. Inf. Assoc. 17(5), 507–513 (2010)

    Article  Google Scholar 

  19. Seol, J.W., Yi, W., Choi, J., Lee, K.S.: Causality patterns and machine learning for the extraction of problem-action relations in discharge summaries. Int. J. Med. Inf. 98, 1–12 (2017)

    Article  Google Scholar 

  20. Shen, X., Gui, Y.: Clinical Pediatrics, 2nd edn. People’s Medical Publishing House (2013)

    Google Scholar 

  21. Sun, Y., et al.: ERNIE: enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223 (2019)

  22. Sundararajan, V., Henderson, T., Perry, C., Muggivan, A., Quan, H., Ghali, W.A.: New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality. J. Clinical Epidemiol. 57(12), 1288–1294 (2004)

    Article  Google Scholar 

  23. Uzuner, Ö., South, B.R., Shen, S., DuVall, S.L.: 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J. Am. Med. Inf. Assoc. 18(5), 552–556 (2011)

    Article  Google Scholar 

  24. Wang, W., Sun, K., Chang, L.: Pediatrics. 9th edn. People’s Medical Publishing House (2018)

    Google Scholar 

  25. Wei, Z., Su, J., Wang, Y., Tian, Y., Chang, Y.: A novel cascade binary tagging framework for relational triple extraction. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1476–1488 (2019)

    Google Scholar 

  26. Yang, Y.L., Lai, P.T., Tsai, R.T.H.: A hybrid system for temporal relation extraction from discharge summaries. In: Cheng, S.M., Day, M.Y. (eds.) International Conference on Technologies and Applications of Artificial Intelligence, vol. 8916, pp. 379–386. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-319-13987-6_35

    Chapter  Google Scholar 

  27. Zan, H., et al.: Construction of Chinese medical knowledge graph based on multi-source corpus. J. Zhengzhou Univ. (Nat. Sci. Edn.) 52(02), 45–51 (2020)

    Google Scholar 

  28. Zan, H., Liu, T., Chen, J., Li, J., Niu, C., Zhao, Y.: Corpus construction for named-entity and entity relations for paediatric diseases. J. Chin. Inf. Process. 34(5), 19–26 (2020)

    Google Scholar 

  29. Zelenko, D., Aone, C., Richardella, A.: Kernel methods for relation extraction. J. Mach. Learn. Res. 3(Feb), 1083–1106 (2003)

    Google Scholar 

  30. Zeng, D., Liu, K., Lai, S., Zhou, G., Zhao, J.: Relation classification via convolutional deep neural network. In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 2335–2344 (2014)

    Google Scholar 

  31. Zeng, X., Zeng, D., He, S., Liu, K., Zhao, J.: Extracting relational facts by an end-to-end neural model with copy mechanism. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 506–514 (2018)

    Google Scholar 

  32. Zhang, K., Zhao, X., Guan, T., Shang, B., Zan, H.: Construction and application of entity and relationship labeling platform for medical texts. J. Chin. Inf. Process. 34(6), 117–125 (2020)

    Google Scholar 

  33. Zhang, Y., Yang, J.: Chinese NER using lattice LSTM. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1554–1564 (2018)

    Google Scholar 

  34. Zheng, S., Wang, F., Bao, H., Hao, Y., Zhou, P., Xu, B.: Joint extraction of entities and relations based on a novel tagging scheme. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1227–1236 (2017)

    Google Scholar 

Download references

Acknowledgements

We greatly appreciate anonymous reviewers for their hard work and insightful suggestions. This work is supported by National Key Research and Development Project (Grant No. 2017YFB1002101), Science and Technique Program of Henan Province (Grant No. 192102210260), Medical Science and Technique Program Co-sponsored by Henan Province and Ministry (Grant No. SB201901021), Hongfei Xu acknowledges the support of China Scholarship Council ([2018]3101, 201807040056).

Author information

Authors and Affiliations

  1. School of Information Engineering, Zhengzhou University, Zhengzhou, China

    Tongfeng Guan, Hongying Zan & Kunli Zhang

  2. Peng Cheng Laboratory, Shenzhen, China

    Tongfeng Guan, Hongying Zan & Kunli Zhang

  3. School of Computer Science and Technology, Soochow University, Suzhou, China

    Xiabing Zhou

  4. Saarland University, Saarland, Germany

    Hongfei Xu

  5. German Research Center for Artificial Intelligence, Saarland, Germany

    Hongfei Xu

Authors
  1. Tongfeng Guan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongying Zan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiabing Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hongfei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kunli Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Hongying Zan or Xiabing Zhou .

Editor information

Editors and Affiliations

  1. ECE & Ingenuity Labs Research Institute, Queen’s University, Kingston, ON, Canada

    Xiaodan Zhu

  2. Department of Computer Science and Technology, Tsinghua University, Beijing, China

    Min Zhang

  3. School of Computer Science and Technology, Soochow University, Suzhou, China

    Yu Hong

  4. College of Intelligence and Computing, Tianjin University, Tianjin, China

    Ruifang He

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Guan, T., Zan, H., Zhou, X., Xu, H., Zhang, K. (2020). CMeIE: Construction and Evaluation of Chinese Medical Information Extraction Dataset. In: Zhu, X., Zhang, M., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2020. Lecture Notes in Computer Science(), vol 12430. Springer, Cham. https://doi.org/10.1007/978-3-030-60450-9_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-60450-9_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-60449-3

  • Online ISBN: 978-3-030-60450-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation