Skip to main content
SpringerLink
Log in

Differential Oriented Image Foresting Transform and Its Applications to Support High-level Priors for Object Segmentation

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

Image foresting transform (IFT) is a graph-based framework to develop image operators based on optimum connectivity between a root set and the remaining nodes, according to a given path-cost function. Oriented image foresting transform (OIFT) was proposed as an extension of some seeded IFT-based segmentation methods to directed graphs, enabling them to support the processing of global object properties, such as connectedness, shape constraints, boundary polarity, and hierarchical constraints, allowing their customization to a given target object. OIFT lies in the intersection of generalized graph cut and general fuzzy connectedness frameworks, inheriting their properties. Its returned segmentation is optimal, with respect to an appropriate graph cut measure, among all segmentations satisfying the given constraints. In this work, we propose differential oriented image foresting transform, which allows multiple OIFT executions for different root sets, making the processing time proportional to the number of modified nodes. Experimental results show considerable efficiency gains over the sequential flow of OIFTs in image segmentation, while maintaining a good treatment of tie zones. We also demonstrate that the differential flow makes it feasible to incorporate the prior knowledge about the maximum allowable size for the segmented object, thus avoiding false positive errors in the segmentation of multi-dimensional images. We also propose an algorithm to efficiently create a hierarchy map that encodes area-constrained OIFT results for all possible thresholds, facilitating the quick selection of the object of interest.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and materials

Not applicable.

Code Availability

Our code is available upon request to the corresponding author.

Notes

  1. Concerning the computational complexity, OIFT can be implemented in \(\mathcal{O}((M + N)\log N)\) (linearithmic time), where N is the number of vertices in the graph and M is the number of arcs, if a binary heap is used for its priority queue. This computational complexity can be improved to \(\mathcal{O}(M+N\times K)\), when the weights are integers in a small interval of size K, by using bucket sorting. The graph cut computational complexity is \(\mathcal{O}(\sqrt{M}*N^2)=O(N^{2.5})\) for a sparse graph, which is more than quadratic-time using a push-relabel based on the highest label node selection rule [7].

  2. As will be shown, the sign is used for the OIFT path-cost functions to indicate that they are related to the optimality of the outer-cut boundary (see Theorem 1), since it resembles an arc pointing to the exterior of an object.

  3. Note that in Line 10 of Algorithm 4, a new seed s is selected from \({{\,\textrm{bd}\,}}(\mathcal{O}^i_1)\) such that \(E^{\{b\}} \le E^{\{s\}}\) for any \(b \in {{\,\textrm{bd}\,}}(\mathcal{O}^i_1)\), which implies that \(E^{\mathcal{S}^i_0 \cup \{b\}} \le E^{\mathcal{S}^i_0 \cup \{s\}}\) according to Proposition 2, where \(\mathcal{S}^i_0\) denotes the background seeds of the ith iteration. Therefore, the new set of background seeds \(\mathcal{S}^{i+1}_0 = \mathcal{S}^{i}_0 \cup \{s\}\) is the one leading to the highest possible energy.

  4. We used a robot user [37] to simulate user interaction by placing brush strokes automatically to iteratively perform the segmentation task. At each iteration, the robot user selects a new corrective seed in the largest connected component of mislabeled pixels, placed at a point farthest from the boundary of the component, in order to imitate the behavior of a real user.

  5. In the first interaction of the robot user, a seed is selected for the object to start the process. The background seed is not counted as a user interaction as we use a fixed set of background seeds at the image border for all images.

References

  1. Miranda, P.A.V., Mansilla, L.A.C.: Oriented image foresting transform segmentation by seed competition. IEEE Trans. Image Process. 23(1), 389–398 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  2. Bejar, H.H.C., Miranda, P.A.V.: Oriented relative fuzzy connectedness: Theory, algorithms, and its applications in hybrid image segmentation methods. EURASIP J. Image Video Process. 2015(21) (2015)

  3. Ciesielski, K.C., Udupa, J.K., Falcão, A.X., Miranda, P.A.V.: A unifying graph-cut image segmentation framework: algorithms it encompasses and equivalences among them. In: Proc. of SPIE on Med. Imaging: Image Process., vol. 8314 (2012)

  4. Ciesielski, K.C., Udupa, J.K., Saha, P.K., Zhuge, Y.: Iterative relative fuzzy connectedness for multiple objects with multiple seeds. Comput. Vis. Image Underst. 107(3), 160–182 (2007)

    Article  Google Scholar 

  5. Cousty, J., Bertrand, G., Najman, L., Couprie, M.: Watershed cuts: minimum spanning forests and the drop of water principle. IEEE Trans. Pattern Anal. Mach. Intell. 31(8), 1362–1374 (2008)

    Article  Google Scholar 

  6. Boykov, Y., Funka-Lea, G.: Graph cuts and efficient N-D image segmentation. Intl. Jrnl. of Comp. Vision 70(2), 109–131 (2006)

    Article  Google Scholar 

  7. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1124–1137 (2004)

    Article  MATH  Google Scholar 

  8. Ciesielski, K.C., Herman, G.T., Kong, T.Y.: General theory of fuzzy connectedness segmentations. J. Math. Imaging Vision 55(3), 304–342 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  9. de Moraes Braz, C., Miranda, P.A.V., Ciesielski, K.C., Cappabianco, F.A.M.: Optimum cuts in graphs by general fuzzy connectedness with local band constraints. J. Math. Imaging Vis. 62, 659–672 (2020)

  10. Tavares, A.C.M., Bejar, H.H.C., Miranda, P.A.V.: Seed robustness of oriented image foresting transform: core computation and the robustness coefficient. In: Angulo, J., Velasco-Forero, S., Meyer, F. (eds.) Math. Morphol. Appl. Signal Image Process., pp. 119–130. Springer, Cham (2017)

    Google Scholar 

  11. Perret, B., Cousty, J., Tankyevych, O., Talbot, H., Passat, N.: Directed connected operators: asymmetric hierarchies for image filtering and segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1162–1176 (2015)

    Article  Google Scholar 

  12. Choi, J., Ueda, E.K., Duran, G.C., Miranda, P.A.V., Tsuzuki, M.S.G.: Automatic lung segmentation with seed generation and ROIFT algorithm for the creation of anatomical atlas. In: ICGG 2022 - Proceedings of the 20th International Conference on Geometry and Graphics, pp. 636–647. Springer, Cham (2023)

  13. Mansilla, L.A.C., Miranda, P.A.V., Cappabianco, F.A.M.: Oriented image foresting transform segmentation with connectivity constraints. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 2554–2558 (2016)

  14. Mansilla, L.A.C., Miranda, P.A.V.: Oriented image foresting transform segmentation: Connectivity constraints with adjustable width. In: 29th SIBGRAPI Conference on Graphics, Patterns and Images, pp. 289–296 (2016)

  15. de Moraes Braz, C.: Graph-based segmentation with local band constraints. In: Couprie, M., Cousty, J., Kenmochi, Y., Mustafa, N. (eds.) Discrete Geometry for Computer Imagery. Springer, Cham (2019)

  16. Mansilla, L.A.C., Miranda, P.A.V.: Image Segmentation by Oriented Image Foresting Transform with Geodesic Star Convexity. In: 15th Intl. Conference on Computer Analysis of Images and Patterns (CAIP), vol. 8047. York, UK, pp. 572–579 (2013)

  17. de Moraes Braz, C., Miranda, P.A.V.: Image segmentation by image foresting transform with geodesic band constraints. In: Image Processing (ICIP), 2014 IEEE International Conference On, pp. 4333–4337 (2014)

  18. Mansilla, L.A.C., Miranda, P.A.V.: Image Segmentation by Oriented Image Foresting Transform: Handling Ties and Colored Images. In: 18th International Conference on Digital Signal Processing, Greece, pp. 1–6 (2013)

  19. Leon, L.M.C., Ciesielski, K.C., Miranda, P.A.V.: Efficient hierarchical multi-object segmentation in layered graphs. Math. Morphol. Theory Appl. 5(1), 21–42 (2021). https://doi.org/10.1515/mathm-2020-0108

    Article  MATH  Google Scholar 

  20. Leon, L.M.C., Miranda, P.A.V.D.: Multi-object segmentation by hierarchical layered oriented image foresting transform. In: 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pp. 79–86 (2017)

  21. Lézoray, O., Grady, L.: Image Processing and Analysis with Graphs: Theory and Practice. CRC Press, California, USA (2012)

    MATH  Google Scholar 

  22. Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene labeling Trans. On Pattern Anal. Mach. Intell. 35, 1915–1929 (2013)

    Article  Google Scholar 

  23. Belém, F., Borlido, I., João, L., Perret, B., Cousty, J., Guimarães, S.J.F., Falcão, A.: Fast and effective superpixel segmentation using accurate saliency estimation. In: Baudrier, É., Naegel, B., Krähenbühl, A., Tajine, M. (eds.) Discrete Geometry and Mathematical Morphology, pp. 261–273. Springer, Cham (2022)

    Chapter  Google Scholar 

  24. Belém, F., Perret, B., Cousty, J., Guimarães, S.J.F., Falcão, A.: Efficient Multiscale Object-based Superpixel Framework. arXiv (2022). https://arxiv.org/abs/2204.03533

  25. Oliveira, D.E.C., Demario, C.L., Miranda, P.A.V.: Image segmentation by relaxed deep extreme cut with connected extreme points. In: Lindblad, J., Malmberg, F., Sladoje, N. (eds.) Discrete Geometry and Mathematical Morphology, pp. 441–453. Springer, Cham (2021)

    Chapter  Google Scholar 

  26. Wolf, S., Schott, L., Kothe, U., Hamprecht, F.: Learned Watershed: End-to-End Learning of Seeded Segmentation. In: International Conference on Computer Vision (ICCV), pp. 2030–2038 (2017)

  27. Sofiiuk, K., Petrov, I.A., Konushin, A.: Reviving iterative training with mask guidance for interactive segmentation. In: 2022 IEEE International Conference on Image Processing (ICIP), pp. 3141–3145 (2022)

  28. Braz, C.d.M., Santos, L.F.D., Miranda, P.A.V.: Graph-based image segmentation with shape priors and band constraints. In: Baudrier, É., Naegel, B., Krähenbühl, A., Tajine, M. (eds.) Discrete Geometry and Mathematical Morphology, pp. 287–299. Springer, Cham (2022)

  29. Falcão, A.X., Stolfi, J., Lotufo, R.A.: The image foresting transform: Theory, algorithms, and applications. IEEE TPAMI 26(1), 19–29 (2004)

    Article  Google Scholar 

  30. Falcão, A.X., Bergo, F.P.G.: Interactive volume segmentation with differential image foresting transforms. IEEE Trans on Medical Imaging 23(9), 1100–1108 (2004)

    Article  Google Scholar 

  31. Condori, M.A.T., Cappabianco, F.A.M., Falcão, A.X., Miranda, P.A.V.: An extension of the differential image foresting transform and its application to superpixel generation. J. Vis. Commun. Image Rep. 71, 102748 (2020)

    Article  Google Scholar 

  32. Vargas-Muñoz, J.E., Chowdhury, A.S., Alexandre, E.B., Galvão, F.L., Miranda, P.A.V., Falcão, A.X.: An iterative spanning forest framework for superpixel segmentation. IEEE Trans. Image Process. 28(7), 3477–3489 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  33. Galvão, F.L., Falcão, A.X., Chowdhury, A.S.: Risf: Recursive iterative spanning forest for superpixel segmentation. In: 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pp. 408–415 (2018)

  34. Alexandre, E.B., Chowdhury, A.S., Falcão, A.X., Miranda, P.A.V.: IFT-SLIC: A general framework for superpixel generation based on simple linear iterative clustering and image foresting transform. In: 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images, pp. 337–344 (2015)

  35. Condori, M.A.T., Miranda, P.A.V.: Differential oriented image foresting transform segmentation by seed competition. In: Baudrier, É., Naegel, B., Krähenbühl, A., Tajine, M. (eds.) Discrete Geometry and Mathematical Morphology, pp. 300–311. Springer, Cham (2022)

    Chapter  Google Scholar 

  36. Ciesielski, K.C., Falcão, A.X., Miranda, P.A.V.: Path-value functions for which dijkstra’s algorithm returns optimal mapping. J. Math. Imaging Vis. 60(7), 1025–1036 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  37. Gulshan, V., Rother, C., Criminisi, A., Blake, A., Zisserman, A.: Geodesic star convexity for interactive image segmentation. In: Proceeding of Computer Vision and Pattern Recognition, pp. 3129–3136 (2010)

Download references

Acknowledgements

Thanks are due to Conselho Nacional de Desenvolvimento Científico e Tecnológico – CNPq – (Grant 407242/2021-0, 313087/2021-0, 465446/2014-0, 166631/2018-3), CAPES (88887.136422/2017-00) and FAPESP (2014/12236-1, 2014/50937-1).

Funding

This work was supported by the Conselho Nacional de Desenvolvimento Científico e Tecnológico—CNPq—(Grant 407242/2021-0, 313087/2021-0, 465446/2014-0, 166631/2018-3), CAPES (88887.136422/2017-00) and FAPESP (2014/12236-1, 2014/50937-1).

Author information

Authors and Affiliations

  1. Institute of Mathematics and Statistics, University of São Paulo, R. do Matão 1010, São Paulo, SP, 05508–090, Brazil

    Marcos A. T. Condori & Paulo A. V. Miranda

Authors
  1. Marcos A. T. Condori
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paulo A. V. Miranda
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The authors contributed equally to this work.

Corresponding authors

Correspondence to Marcos A. T. Condori or Paulo A. V. Miranda.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Ethics Approval

This paper is an extended version of a conference paper presented at International Conference on Discrete Geometry and Mathematical Morphology (DGMM22), as an invited paper to a special issue of JMIV.

Consent to participate

This research did not directly involve human participants or animals. The medical images used in this paper were provided by the hospital of Unicamp and the University of Pennsylvania preserving the anonymity of their patients and subjects.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Condori, M.A.T., Miranda, P.A.V. Differential Oriented Image Foresting Transform and Its Applications to Support High-level Priors for Object Segmentation. J Math Imaging Vis 65, 802–817 (2023). https://doi.org/10.1007/s10851-023-01158-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-023-01158-7

Keywords

Search

Navigation