Skip to main content
SpringerLink
Log in

Multi-Output Tree Chaining: An Interpretative Modelling and Lightweight Multi-Target Approach

  • Published:
Journal of Signal Processing Systems Aims and scope Submit manuscript

Abstract

Multi-target regression (MTR) regards predictive problems with multiple numerical targets. To solve this, machine learning techniques can model solutions treating each target as a separated problem based only on the input features. Nonetheless, modelling inter-target correlation can improve predictive performance. When performing MTR tasks using the statistical dependencies of targets, several approaches put aside the evaluation of each pair-wise correlation between those targets, which may differ for each problem. Besides that, one of the main drawbacks of the current leading MTR method is its high memory cost. In this paper, we propose a novel MTR method called Multi-output Tree Chaining (MOTC) to overcome the mentioned disadvantages. Our method provides an interpretative internal tree-based structure which represents the relationships between targets denominated Chaining Trees (CT). Different from the current techniques, we compute the outputs dependencies, one-by-one, based on the Random Forest importance metric. Furthermore, we proposed a memory friendly approach which reduces the number of required regression models when compared to a leading method, reducing computational cost. We compared the proposed algorithm against three MTR methods (Single-target - ST; Multi-Target Regressor Stacking - MTRS; and Ensemble of Regressor Chains - ERC) on 18 benchmark datasets with two base regression algorithms (Random Forest and Support Vector Regression). The obtained results show that our method is superior to the ST approach regarding predictive performance, whereas, having no significant difference from ERC and MTRS. Moreover, the interpretative tree-based structures built by MOTC pose as great insight on the relationships among targets. Lastly, the proposed solution used significantly less memory than ERC being very similar in predictive performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12

Similar content being viewed by others

Notes

  1. http://mulan.sourceforge.net/datasets-mtr.html

  2. The source codes for MOTC and the other evaluated MTR methods are disponible in http://www.uel.br/grupo-pesquisa/remid/?page_id=145.

References

  1. Aho, T., Zenko, B., Dzeroski, S., Elomaa, T. (2012). Multi-target regression with rule ensembles. Journal of Machine Learning Research, 13, 2367–2407.

    MathSciNet  MATH  Google Scholar 

  2. Borchani, H., Varando, G., Bielza, C., Larrañaga, P. (2015). A survey on multi-output regression. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 5(5), 216–233.

    Google Scholar 

  3. Breiman, L. (2001). Random forests. Machine learning, 45.1, 5–32. https://doi.org/10.1017/CBO9781107415324.004.

    Article  MATH  Google Scholar 

  4. Brugger, D., Rosenstiel, W., Bogdan, M. (2011). Online SVR training by solving the primal optimization problem. Journal of Signal Processing Systems, 65(3), 391–402.

    Article  Google Scholar 

  5. Chen, H., & Ser, W. (2011). Sound source DOA estimation and localization in noisy reverberant environments using least-squares support vector machines. Journal of Signal Processing Systems, 63(3), 287–300.

    Article  Google Scholar 

  6. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1023/A:1022627411411.

    Article  MATH  Google Scholar 

  7. Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7, 1–30.

    MathSciNet  MATH  Google Scholar 

  8. Di Persio, L., & Honchar, O. (2016). Artificial neural networks architectures for stock price prediction: comparisons and applications. International Journal of Circuits, Systems and Signal Processing, 10, 403–413.

    Google Scholar 

  9. Drucker, H., Burges, C.J.C., Kaufman, L., Smola, A.J., Vapnik, V. (1997). Support vector regression machines. In Mozer, M.C., Jordan, M.I., Petsche, T. (Eds.) Advances in neural information processing systems (Vol. 9, pp. 155–161). MIT Press. http://papers.nips.cc/paper/1238-support-vector-regression-machines.pdf.

  10. Evgeniou, T., Figueiras-Vidal, A.R., Theodoridis, S. (2008). Emerging machine learning techniques in signal processing.

  11. Gama, J., & Brazdil, P. (2000). Cascade generalization. Machine Learning, 41(3), 315–343. https://doi.org/10.1023/A:1007652114878.

    Article  MATH  Google Scholar 

  12. Genuer, R., Poggi, J.M., Tuleau-Malot, C. (2010). Variable selection using random forests. Pattern Recognition Letters, 31(14), 2225–2236. https://doi.org/10.1016/j.patrec.2010.03.014. http://www.sciencedirect.com/science/article/pii/S0167865510000954.

    Article  Google Scholar 

  13. Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301), 13–30. https://doi.org/10.1080/01621459.1963.10500830. http://amstat.tandfonline.com/doi/abs/10.1080/01621459.1963.10500830.

    Article  MathSciNet  MATH  Google Scholar 

  14. Katagiri, S., Nakamura, A., Adali, T., Tao, J., Larsen, J., Tan, T. (2014). Guest editorial: Machine learning for signal processing. Journal of Signal Processing Systems, 74(3), 281–283. https://doi.org/10.1007/s11265-014-0871-6.

    Article  Google Scholar 

  15. Kocev, D., Džeroski, S., White, M.D., Newell, G.R., Griffioen, P. (2009). Using single- and multi-target regression trees and ensembles to model a compound index of vegetation condition. Ecological Modelling, 220(8), 1159–1168.

    Article  Google Scholar 

  16. Kocev, D., Vens, C., Struyf, J., Džeroski, S. (2007). Ensembles of multi-objective decision trees. In European conference on machine learning (pp. 624–631). Springer.

  17. Kocev, D., Vens, C., Struyf, J., Džeroski, S. (2013). Tree ensembles for predicting structured outputs. Pattern Recognition, 46(3), 817–833.

    Article  Google Scholar 

  18. Li, X., & Zheng, J. (2016). Active learning for regression with correlation matching and labeling error suppression. IEEE Signal Processing Letters, 23(8), 1081–1085.

    Article  Google Scholar 

  19. Lichman, M. (2013). UCI machine learning repository. http://archive.ics.uci.edu/ml.

  20. Mastelini, S.M., Santana, E.J., Cerri, R., Barbon, S. Jr. (2017). DSTARS: a multi-target deep structure for tracking asynchronous regressor stack. In Brazilian conference on intelligent systems. BRACIS 2017.

  21. Melki, G., Cano, A., Kecman, V., Ventura, S. (2017). Multi-target support vector regression via correlation regressor chains. Information Sciences, 415, 53–69.

    Article  MathSciNet  Google Scholar 

  22. Moyano, J.M., Gibaja, E.L., Ventura, S. (2017). An evolutionary algorithm for optimizing the target ordering in ensemble of regressor chains. In 2017 IEEE congress on evolutionary computation (CEC) (pp. 2015–2021). IEEE.

  23. Santana, E.J., Mastelini, S.M., Barbon, S. Jr. (2017). Deep regressor stacking for air ticket prices prediction. In Brazilian symposium of information systems (pp. 216–233). SBSI 2017.

  24. Sidike, P., Krieger, E., Alom, M.Z., Asari, V.K., Taha, T. (2017). A fast single-image super-resolution via directional edge-guided regularized extreme learning regression. In Signal, image and video processing (pp. 1–8).

  25. Spyromitros-Xioufis, E., Tsoumakas, G., Groves, W., Vlahavas, I. (2016). Multi-target regression via input space expansion: treating targets as inputs. Machine Learning, 104(1), 55–98.

    Article  MathSciNet  Google Scholar 

  26. Tsoumakas, G., Spyromitros-Xioufis, E., Vrekou, A., Vlahavas, I. (2014). Multi-target regression via random linear target combinations. In Joint european conference on machine learning and knowledge discovery in databases (pp. 225–240). Springer.

  27. Wang, Q., Wu, Y., Shen, Y., Liu, Y., Lei, Y. (2015). Supervised sparse manifold regression for head pose estimation in 3d space. Signal Processing, 112, 34–42.

    Article  Google Scholar 

  28. Watanabe, S., Nakamura, A., Juang, B.H.F. (2014). Structural bayesian linear regression for hidden Markov models. Journal of Signal Processing Systems, 74(3), 341–358.

    Article  Google Scholar 

  29. Zhang, W., Liu, X., Ding, Y., Shi, D. (2012). Multi-output LS-SVR machine in extended feature space. In CIMSA 2012 - 2012 IEEE Int. Conf. Comput. Int.ll. Meas. Syst. Appl. Proc. (pp. 130–144). https://doi.org/10.1109/CIMSA.2012.6269600.

Download references

Acknowledgements

The authors would like to thank CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior), CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico) and FAPESP (Fundação de Amparo à Pesquisa do Estado de São Paulo) for financial support.

Author information

Authors and Affiliations

  1. Computer Science Department, State University of Londrina. Rodovia Celso Garcia Cid, Km 380, s/n - Campus Universitário, Londrina, PR, 86057-970, Brazil

    Saulo Martiello Mastelini, Victor Guilherme Turrisi da Costa & Sylvio Barbon Jr.

  2. Electrical Engineering Department, State University of Londrina. Rodovia Celso Garcia Cid, Km 380, s/n - Campus Universitário, Londrina, PR, 86057-970, Brazil

    Everton Jose Santana

  3. Department of Computer Science, Federal University of São Carlos, Rodovia Washington Luís, km 235, São Carlos, SP, 13565-905, Brazil

    Felipe Kenji Nakano & Ricardo Cerri

  4. Instituto de Biociências, Letras e Ciências Exatas, Unesp - Univ Estadual Paulista (São Paulo State University), Rua Cristóvão Colombo 2265, Jd Nazareth, 15054-000, São José do Rio Preto, SP, Brazil

    Rodrigo Capobianco Guido

Authors
  1. Saulo Martiello Mastelini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Victor Guilherme Turrisi da Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Everton Jose Santana
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Felipe Kenji Nakano
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rodrigo Capobianco Guido
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ricardo Cerri
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Sylvio Barbon Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saulo Martiello Mastelini.

Appendices

Appendix A: Datasets Used in the Experiments

Table 4 Dataset’s characteristics: dataset name, number of examples, number of input variables, number of targets, and description.
Full size table

Appendix B: Obtained Condensed Chaining Tree Graphs and Target Labels

Table 5 Targets’ labels for datasets EDM, ENB and Jura.
Full size table
Table 6 Targets labels for datasets SCPF, SF1, SF2 and Slump.
Full size table
Table 7 Targets’ labels for datasets Andro, ATP1D and ATP7D.
Full size table
Table 8 Targets’ labels for dataset OES10.
Full size table
Table 9 Targets’ labels for dataset OES97.
Full size table
Table 10 Targets labels for datasets OSALES, RF1 and RF2.
Full size table
Table 11 Targets labels for dataset SCM1D, SCM20D and WQ.
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mastelini, S.M., da Costa, V.G.T., Santana, E.J. et al. Multi-Output Tree Chaining: An Interpretative Modelling and Lightweight Multi-Target Approach. J Sign Process Syst 91, 191–215 (2019). https://doi.org/10.1007/s11265-018-1376-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11265-018-1376-5

Keywords

Search

Navigation