Abstract
Reconstructing 3D objects from 2D images is both challenging for our brains and machine learning algorithms. To support this spatial reasoning task, contextual information about the overall shape of an object is critical. However, such information is not captured by established loss terms (e.g. Dice loss). We propose to complement geometrical shape information by including multi-scale topological features, such as connected components, cycles, and voids, in the reconstruction loss. Our method uses cubical complexes to calculate topological features of 3D volume data and employs an optimal transport distance to guide the reconstruction process. This topology-aware loss is fully differentiable, computationally efficient, and can be added to any neural network. We demonstrate the utility of our loss by incorporating it into SHAPR, a model for predicting the 3D cell shape of individual cells based on 2D microscopy images. Using a hybrid loss that leverages both geometrical and topological information of single objects to assess their shape, we find that topological information substantially improves the quality of reconstructions, thus highlighting its ability to extract more relevant features from image datasets.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Expert readers may recognise that cubical complexes are related to meshes and simplicial complexes but use squares instead of triangles as their building blocks.
- 2.
We use the subscript f to indicate the corresponding likelihood function; we will drop this for notational convenience when discussing general properties.
- 3.
We dropped all hyperparameters of the loss term for notational clarity.
- 4.
References
Biewald, L.: Experiment tracking with Weights and Biases (2020). https://www.wandb.com/
Carrière, M., Chazal, F., Glisse, M., Ike, Y., Kannan, H., Umeda, Y.: Optimizing persistent homology based functions. In: Proceedings of the 38th International Conference on Machine Learning, pp. 1294–1303 (2021)
Choy, C.B., Xu, D., Gwak, J.Y., Chen, K., Savarese, S.: 3D-R2N2: a unified approach for single and multi-view 3D object reconstruction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 628–644. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_38
Clough, J., Byrne, N., Oksuz, I., Zimmer, V.A., Schnabel, J.A., King, A.: A topological loss function for deep-learning based image segmentation using persistent homology. IEEE Trans. Pattern Anal. Mach. Intell. (2020)
Cohen-Steiner, D., Edelsbrunner, H., Harer, J.: Extending persistence using Poincaré and Lefschetz duality. Found. Comput. Math. 9(1), 79–103 (2009)
Cohen-Steiner, D., Edelsbrunner, H., Harer, J., Mileyko, Y.: Lipschitz functions have \(\rm L _p\)-stable persistence. Found. Comput. Math. 10(2), 127–139 (2010)
Edelsbrunner, H., Letscher, D., Zomorodian, A.J.: Topological persistence and simplification. Discrete Comput. Geom. 28(4), 511–533 (2002)
Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3D object reconstruction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Flamary, R., et al.: POT: python optimal transport. J. Mach. Learn. Res. 22(78), 1–8 (2021)
Gkioxari, G., Malik, J., Johnson, J.: Mesh R-CNN. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019)
Hensel, F., Moor, M., Rieck, B.: A survey of topological machine learning methods. Front. Artif. Intell. 4, 681108 (2021)
Hofer, C.D., Graf, F., Rieck, B., Niethammer, M., Kwitt, R.: Graph filtration learning. In: Proceedings of the 37th International Conference on Machine Learning (ICML), pp. 4314–4323 (2020)
Hu, X., Li, F., Samaras, D., Chen, C.: Topology-preserving deep image segmentation. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Hu, X., Wang, Y., Fuxin, L., Samaras, D., Chen, C.: Topology-aware segmentation using discrete Morse theory. In: International Conference on Learning Representations (2021)
Lesnick, M., Wright, M.: Interactive visualization of 2-D persistence modules arXiv:1512.00180 (2015)
Minaee, S., Boykov, Y., Porikli, F., Plaza, A., Kehtarnavaz, N., Terzopoulos, D.: Image segmentation using deep learning: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 44(7), 3523–3542 (2021)
Moor, M., Horn, M., Rieck, B., Borgwardt, K.: Topological autoencoders. In: Proceedings of the 37th International Conference on Machine Learning (ICML), pp. 7045–7054 (2020)
Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32, pp. 8024–8035 (2019)
Poulenard, A., Skraba, P., Ovsjanikov, M.: Topological function optimization for continuous shape matching. Comput. Graphics Forum 37(5), 13–25 (2018)
Rieck, B., Leitte, H.: Exploring and comparing clusterings of multivariate data sets using persistent homology. Comput. Graphics Forum 35(3), 81–90 (2016)
Rieck, B., Yates, T., Bock, C., Borgwardt, K., Wolf, G., Turk-Browne, N., Krishnaswamy, S.: Uncovering the topology of time-varying fMRI data using cubical persistence. In: Advances in Neural Information Processing Systems, vol. 33, pp. 6900–6912 (2020)
Shit, S., et al.: clDice - a novel topology-preserving loss function for tubular structure segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16560–16569 (2021)
Simionato, G., et al.: Red blood cell phenotyping from 3D confocal images using artificial neural networks. PLoS Comput. Biol. 17(5), 1–17 (2021)
Skraba, P., Turner, K.: Wasserstein stability for persistence diagrams. arXiv:2006.16824 (2020)
Wagner, H., Chen, C., Vuçini, E.: Efficient computation of persistent homology for cubical data. In: Peikert, R., Hauser, H., Carr, H., Fuchs, R. (eds.) Topological Methods in Data Analysis and Visualization II: Theory, Algorithms, and Applications, pp. 91–106. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-23175-9_7
Waibel, D.J.E., Kiermeyer, N., Atwell, S., Sadafi, A., Meier, M., Marr, C.: SHAPR - an AI approach to predict 3D cell shapes from 2D microscopic images bioRxiv:2021.09.29.462353 (2021)
Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.-G.: Pixel2Mesh: generating 3D mesh models from single RGB images. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 55–71. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_4
Acknowledgements
We thank Lorenz Lamm, Melanie Schulz, Kalyan Varma Nadimpalli, and Sophia Wagner for their valuable feedback to this manuscript. The authors also are indebted to Teresa Heiss for discussions on the topological changes induced by downsampling volume data.
Funding
Carsten Marr received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 Research and Innovation Programme (Grant Agreement 866411).
Author information
Authors and Affiliations
Contributions
DW and BR implemented code and conducted experiments. DW, BR, and CM wrote the manuscript. DW created figures and BR the main portrayal of results. SA and MM provided the 3D nuclei dataset. BR supervised the study. All authors have read and approved the manuscript.
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Waibel, D.J.E., Atwell, S., Meier, M., Marr, C., Rieck, B. (2022). Capturing Shape Information with Multi-scale Topological Loss Terms for 3D Reconstruction. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13434. Springer, Cham. https://doi.org/10.1007/978-3-031-16440-8_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-16440-8_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16439-2
Online ISBN: 978-3-031-16440-8
eBook Packages: Computer ScienceComputer Science (R0)
