Abstract
With the widespread creation of artificial intelligence (AI) models in biosciences, bio-medical researchers are reusing trained AI models from other applications. This work is motivated by the need to characterize trained AI models for reuse based on metrics derived from optimization curves captured during model training. Such AI model characterizations can aid future model accuracy refinement, inform users about model hyper-parameter sensitivity, and assist in model reuse according to multi-purpose objectives. The challenges lie in understanding relationships between trained AI models and optimization curves, defining and validating quantitative AI model metrics, and disseminating metrics with trained AI models. We approach these challenges by analyzing optimization curves generated for image segmentation and classification tasks to assist in a multi-objective reuse of AI models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). https://doi.org/10.5281/zenodo.5898685. https://www.tensorflow.org/, software available from tensorflow.org
Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: a next-generation hyperparameter optimization framework (2019)
Bajcsy, P., Feldman, S., Majurski, M., Snyder, K., Brady, M.: Approaches to training ai-based multi-class semantic image segmentation. J. Microscopy 279(2), 98–113 (2020). https://doi.org/10.1111/jmi.12906. https://pubmed.ncbi.nlm.nih.gov/32406521/
Bajcsy, P., Schaub, N.J., Majurski, M.: Designing trojan detectors in neural networks using interactive simulations. Appl. Sci. 11(4) (2021). https://doi.org/10.3390/app11041865. https://www.mdpi.com/2076-3417/11/4/1865
Community: Open neural network exchange (ONNX) (2022). https://onnx.ai/
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., Wallach, H., III, H.D., Crawford, K.: Datasheets for datasets. arXiv 1803.09010 (2021)
Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J.E., Sculley, D. (eds.): Google Vizier: A Service for Black-Box Optimization (2017). http://www.kdd.org/kdd2017/papers/view/google-vizier-a-service-for-black-box-optimization
Haibe-Kains, B., et al.: Transparency and reproducibility in artificial intelligence. Nature 586(7829), E14–E16 (2020). https://doi.org/10.1038/s41586-020-2766-y. https://aclanthology.org/Q18-1041
Holland, S., Hosny, A., Newman, S., Joseph, J., Chmielinski, K.: The dataset nutrition label: A framework to drive higher data quality standards. arXiv 1805.03677 (2018)
IARPA: Trojans in Artificial Intelligence (TrojAI) (2020). https://pages.nist.gov/trojai/, https://www.iarpa.gov/index.php/research-programs/trojai
Koch, P., Golovidov, O., Gardner, S., Wujek, B., Griffin, J., Xu, Y.: Autotune. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 2018. https://doi.org/10.1145/3219819.3219837
Krizhevsky, A., Nair, V., Hinton, G.: Cifar-10 (canadian institute for advanced research). http://www.cs.toronto.edu/kriz/cifar.html
Lin, T.Y., et al.: Microsoft COCO: common objects in context. arXiv 1405.0312 (2014). http://arxiv.org/abs/1405.0312
Long, J., Shelhamer, E., Darrell, T.: Experiment manager (2022). https://www.mathworks.com/help/deeplearning/ref/experimentmanager-app.html
Mitchell, M., et al.: Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, January 2019. https://doi.org/10.1145/3287560.3287596
NIST: Data-driven simulations of measured neutron interferometric microscopy images (2022). https://www.nist.gov/programs-projects/interferometry-infer-neutron-interferometric-microscopy-small-forces-and
NIST: Fluorescent microscopy images of A-10 rat smooth muscle cells and NIH-3T3 mouse fibro-blasts (2022). https://isg.nist.gov/deepzoomweb/data/dissemination
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
Ranjit, M., Ganapathy, G., Sridhar, K., Arumugham, V.: Efficient deep learning hyperparameter tuning using cloud infrastructure: Intelligent distributed hyperparameter tuning with bayesian optimization in the cloud. In: 2019 IEEE 12th International Conference on Cloud Computing (CLOUD), pp. 520–522. IEEE Computer Society, Los Alamitos, July 2019. https://doi.org/10.1109/CLOUD.2019.00097, https://doi.ieeecomputersociety.org/10.1109/CLOUD.2019.00097
Schaub, N., et al.: Deep learning predicts function of live retinal pigment epithelium from quantitative microscopy. J. Clin. Invest. 130(2), 1010–1023 (2020). https://doi.org/10.1172/JCI131187, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6994191/
Yang, K., Stoyanovich, J., Asudeh, A., Howe, B., Jagadish, H., Miklau, G.: A nutritional label for rankings. Proceedings of the 2018 International Conference on Management of Data, May 2018. https://doi.org/10.1145/3183713.3193568
Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., Hutter, F.: NAS-bench-101: Towards reproducible neural architecture search. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 7105–7114. PMLR, Long Beach, California, USA (09–15 Jun 2019), http://proceedings.mlr.press/v97/ying19a.html
You, S., et al.: A review on artificial intelligence for grid stability assessment. In: 2020 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), pp. 1–6 (2020). https://doi.org/10.1109/SmartGridComm47815.2020.9302990
Acknowledgments
The funding for Peter Bajcsy, Michael Majurski, and Walid Keyrouz was partially provided from the IARPA project: IARPA-20001-D2020-2007180011. We would also like to thank Craig Greenberg, Ivy Liang, and Daniel Gao for providing very insightful feedback and contributing to the code.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bajcsy, P., Majurski, M., Cleveland IV, T.E., Carrasco, M., Keyrouz, W. (2023). Characterization of AI Model Configurations for Model Reuse. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13804. Springer, Cham. https://doi.org/10.1007/978-3-031-25069-9_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-25069-9_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25068-2
Online ISBN: 978-3-031-25069-9
eBook Packages: Computer ScienceComputer Science (R0)