Abstract
As input distributions evolve over a mission lifetime, maintaining performance of learning-based models becomes challenging. This paper presents a framework to incrementally retrain a model by selecting a subset of test inputs to label, which allows the model to adapt to changing input distributions. Algorithms within this framework are evaluated based on (1) model performance throughout mission lifetime and (2) cumulative costs associated with labeling and model retraining. We provide an open-source benchmark of a satellite pose estimation model trained on images of a satellite in space and deployed in novel scenarios (e.g., different backgrounds or misbehaving pixels), where algorithms are evaluated on their ability to maintain high performance by retraining on a subset of inputs. We also propose a novel algorithm to select a diverse subset of inputs for labeling, by characterizing the information gain from an input using Bayesian uncertainty quantification and choosing a subset that maximizes collective information gain using concepts from batch active learning. We show that our algorithm outperforms others on the benchmark, e.g., achieves comparable performance to an algorithm that labels 100% of inputs, while only labeling 50% of inputs, resulting in low costs and high performance over the mission lifetime.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Code for the benchmark and our experiments is available at https://github.com/StanfordASL/data-lifecycle-management.
References
Abdar, M., et al.: A review of uncertainty quantification in deep learning: techniques, applications and challenges. Inf. Fusion 76, 243–297 (2021)
Chen, B., Cao, J., Parra, A., Chin, T.J.: Satellite pose estimation with deep landmark regression and nonlinear pose refinement. In: IEEE International Conference on Computer Vision (2019)
Doran, G., Daigavane, A., Wagstaff, K.: Resource consumption and radiation tolerance assessment for data analysis algorithms onboard spacecraft. IEEE Trans. Aerosp. Electron. Syst. 58, 5180–5189 (2022)
Freytag, A., Rodner, E., Denzler, J.: Selecting influential examples: active learning with expected model output changes. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 562–577. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_37
Grzenda, M., Gomes, H.M., Bifet, A.: Delayed labelling evaluation for data streams. Data Min. Knowl. Disc. 34(5), 1237–1266 (2020)
Guo, Y., Schuurmans, D.: Discriminative batch mode active learning. In: Conference on Neural Information Processing Systems (2007)
Hoi, S.C., Jin, R., Zhu, J., Lyu, M.R.: Batch mode active learning and its application to medical image classification. In: International Conference on Machine Learning (2006)
Hsu, Y.C., Liu, Y.C., Kira, Z.: Re-evaluating continual learning scenarios: a categorization and case for strong baselines. In: Conference on Neural Information Processing Systems (2018)
Kisantal, M., Sharma, S., Park, T.H., Izzo, D., Märtens, M., D’Amico, S.: Satellite pose estimation challenge: dataset, competition design, and results. IEEE Trans. Aerosp. Electron. Syst. 56(5), 4083–4098 (2020)
Koh, P.W., et al.: WILDS: a benchmark of in-the-wild distribution shifts. In: International Conference on Machine Learning (2021)
Lin, Z., Shi, J., Pathak, D., Ramanan, D.: The CLEAR benchmark: continual learning on real-world imagery. In: Conference on Neural Information Processing Systems - Datasets and Benchmarks Track (2021)
Murray, G., Bourlai, T., Spolaor, M.: Mask R-CNN: detection performance on SPEED spacecraft with image degradation. In: IEEE International Conference on Big Data (2021)
Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: A review. Neural Netw. 113, 54–71 (2019)
Pinsler, R., Gordon, J., Nalisnick, E., Hernández-Lobato, J.M.: Bayesian batch active learning as sparse subset approximation. In: Conference on Neural Information Processing Systems (2019)
Poghosyan, A., Golkar, A.: CubeSat evolution: analyzing CubeSat capabilities for conducting science missions. Prog. Aerosp. Sci. 88, 59–83 (2017)
Roy, N., McCallum, A.: Toward optimal active learning through sampling estimation of error reduction. In: International Conference on Machine Learning (2001)
Sener, O., Savarese, S.: Active learning for convolutional neural networks: a core-set approach. In: International Conference on Learning Representations (2018)
Settles, B.: Active learning. Synth. Lect. Artif. Intell. Mach. Learn. 6(1), 1–114 (2012)
Sharma, A., Azizan, N., Pavone, M.: Sketching curvature for efficient out-of-distribution detection for deep neural networks. In: Proceedings of Conference on Uncertainty in Artificial Intelligence (2021)
Slingerland, P., et al.: Adapting a trusted AI framework to space mission autonomy. In: IEEE Aerospace Conference (2022)
Weiher, H., Mabry, D.J., Utter, A.C.: Slingshot: in-space modularity test platform. In: AIAA Aerospace Sciences Meeting (2022)
Weiss, K., Khoshgoftaar, T.M., Wang, D.D.: A survey of transfer learning. J. Big Data 3(1), 1–40 (2016). https://doi.org/10.1186/s40537-016-0043-6
Wellhausen, L., Ranftl, R., Hutter, M.: Safe robot navigation via multi-modal anomaly detection. IEEE Robot. Autom. Lett. 5(2), 1326–1333 (2020)
Yang, J., Zhou, K., Li, Y., Liu, Z.: Generalized out-of-distribution detection: a survey (2021). arxiv.org:2110.11334
Yang, Y., Loog, M.: Active learning using uncertainty information. In: IEEE International Conference on Pattern Recognition (2016)
Acknowledgments
This work is supported by The Aerospace Corporation’s University Partnership Program, and by the Stanford Graduate Fellowship (SGF). The NASA University Leadership initiative (grant #80NSSC20M0163) provided funds to assist the authors with their research, but this article solely reflects the opinions and conclusions of its authors and not any NASA entity. The authors would like to thank Rohan Sinha and the reviewers for their helpful comments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Banerjee, S., Sharma, A., Schmerling, E., Spolaor, M., Nemerouf, M., Pavone, M. (2023). Data Lifecycle Management in Evolving Input Distributions for Learning-based Aerospace Applications. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13801. Springer, Cham. https://doi.org/10.1007/978-3-031-25056-9_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-25056-9_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25055-2
Online ISBN: 978-3-031-25056-9
eBook Packages: Computer ScienceComputer Science (R0)