Abstract
Medication mistaking is one of the risks that can result in unpredictable consequences for patients. To mitigate this risk, we develop an automatic system that correctly identifies pill-prescription from mobile images. Specifically, we define a so-called pill-prescription matching task, which attempts to match the images of the pills taken with the pills’ names in the prescription. We then propose PIMA, a novel approach using Graph Neural Network (GNN) and contrastive learning to address the targeted problem. In particular, GNN is used to learn the spatial correlation between the text boxes in the prescription and thereby highlight the text boxes carrying the pill names. In addition, contrastive learning is employed to facilitate the modeling of cross-modal similarity between textual representations of pill names and visual representations of pill images. We conducted extensive experiments and demonstrated that PIMA outperforms baseline models on a real-world dataset of pill and prescription images that we constructed. Specifically, PIMA improves the accuracy from \(19.09\%\) to \(46.95\%\) compared to other baselines. We believe our work can open up new opportunities to build new clinical applications and improve medication safety and patient care.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The dataset can be downloaded from our project Web-page at https://vaipe.org/#resource.
References
Chang, W.J., Chen, L.B., Hsu, C.H., Lin, C.P., Yang, T.C.: A deep learning-based intelligent medicine recognition system for chronic patients. IEEE Access 7, 44441–44458 (2019)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. Computation Research Repository arXiv Preprint, arXiv:1810.04805 (2018)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. Computation Research Repository arXiv Preprint, arXiv:2010.11929 (2020
Frome, A., et al.: DeViSE: a deep visual-semantic embedding model. Neural Inf. Process. Syst. 26, 2121–2129 (2013)
Gao, D., et al.: FashionBERT: text and image matching with adaptive loss for cross-modal retrieval. Computation Research Repository arXiv Preprint, arXiv:2005.09801 (2020)
Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. Neural Inf. Process. Syst. 30, 1025–1035 (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hendrycks, D., Gimpel, K.: Gaussian error linear units (GELUs). Computation Research Repository arXiv Preprint, arXiv:1606.08415 (2016)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)
Howard, A., et al.: Searching for MobileNetV3. Computation Research Repository arXiv Preprint, arXiv:1905.02244 (2019)
Kiros, R., Salakhutdinov, R., Zemel, R.: Unifying visual-semantic embeddings with multimodal neural language models. Computation Research Repository arXiv Preprint, arXiv:1411.2539 (2014)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. Computation Research Repository arXiv Preprint, arXiv:1411.2539 (2019)
Radford, A., et al.: Learning transferable visual models from natural language supervision 139, 8748–8763 (2021)
Reimers, N., Gurevych, I.: Sentence-BERT: sentence embeddings using Siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pp. 3973–3983 (2019)
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)
Sechidis, K., Tsoumakas, G., Vlahavas, I.: On the stratification of multi-label data. In: Proceedings of the 2011 European Conference on Machine Learning and Knowledge Discovery in Databases, vol. 3, pp. 145–158 (2011)
Vaswani, A., et al.: Attention is all you need. In: 30th Proceedings of the Conference on Advances in Neural Information Processing Systems (2017)
Yaniv, Z., et al.: The national library of medicine pill image recognition challenge: an initial report. In: Proceedings of the 2016 IEEE Applied Imagery Pattern Recognition Workshop (2016)
Acknowledgement
This work was funded by Vingroup Joint Stock Company (Vingroup JSC), Vingroup, and supported by Vingroup Innovation Foundation (VINIF) under project code VINIF.2021.DA00128.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Nguyen, T.T., Nguyen, H.D., Nguyen, T.H., Pham, H.H., Ide, I., Nguyen, P.L. (2022). A Novel Approach for Pill-Prescription Matching with GNN Assistance and Contrastive Learning. In: Khanna, S., Cao, J., Bai, Q., Xu, G. (eds) PRICAI 2022: Trends in Artificial Intelligence. PRICAI 2022. Lecture Notes in Computer Science, vol 13629. Springer, Cham. https://doi.org/10.1007/978-3-031-20862-1_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-20862-1_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20861-4
Online ISBN: 978-3-031-20862-1
eBook Packages: Computer ScienceComputer Science (R0)