Abstract
One of the methods used to evaluate the performance of ensemble classifiers is bias and variance analysis. In this chapter, we analyse bootstrap aggregating (Bagging) and Error Correcting Output Coding (ECOC) ensembles using a biasvariance framework; and make comparisons with single classifiers, while having Neural Networks (NNs) as base classifiers. As the performance of the ensembles depends on the individual base classifiers, it is important to understand the overall trends when the parameters of the base classifiers – nodes and epochs for NNs –, are changed. We show experimentally on 5 artificial and 4 UCI MLR datasets that there are some clear trends in the analysis that should be taken into consideration while designing NN classifier systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Allwein, E., Schapire, R., Singer, Y.: Reducing multiclass to binary: A unifying approach for margin classifiers. J. Machine Learning Research 1, 113–141 (2002)
Asuncion, A., Newman, D.J.: UCI machine learning repository, School of Information and Computer Science. University of California, Irvine (2007)
Breiman, L.: Arcing classifiers. The Annals of Stat. 26, 801–849 (1998)
Dietterich, T.G., Bakiri, G.: Solving multi-class learning problems via error-correcting output codes. J. Artif. Intell. Research 2, 263–286 (1995)
Domingos, P.: Why does bagging work? A Bayesian account and its implications. In: Heckerman, D., Mannila, H., Pregibon, D. (eds.) Proc. the 3rd Int. Conf. Knowledge Discovery and Data Mining, Newport Beach, CA, pp. 155–158. AAAI Press, Menlo Park (1997)
Domingos, P.: A unified bias-variance decomposition for zero-one and squared loss. In: Proc. the 17th Natl. Conf. Artif. Intell., Austin, TX, pp. 564–569. MIT Press, Cambridge (2000)
Escalera, S., Tax, D.M.J., Pujol, O., Radeva, P., Duin, R.P.W.: Subclass problem-dependent design for error-correcting output codes. IEEE Trans. Pattern Analysis and Machine Intell. 30, 1041–1054 (2008)
Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Saitta, L. (ed.) Proc. the 13th Int. Conf. Machine Learning, Bari, Italy, pp. 148–156. Morgan Kaufmann, San Francisco (1996)
Friedman, J.H.: On bias, variance, 0/1 loss and the curse of dimensionality. Data Mining and Knowledge Discovery 1, 55–77 (1997)
Geman, S., Bienenstock, E., Doursat, R.: Neural networks and the bias/variance dilemma. Neural Comp. 4, 1–58 (1992)
Heskes, T.: Bias/variance decomposition for likelihood-based estimators. Neural Comp. 10, 1425–1433 (1998)
James, G.M.: Majority vote classifiers: Theory and applications. PhD Thesis, Department of Statistics, University of Standford (1998)
James, G.: Variance and bias for general loss functions. Machine Learning 51, 115–135 (2003)
James, G.M., Hastie, T.: The error coding method and PICT’s. Comp. and Graph. Stat. 7, 377–387 (1998)
Kohavi, R., Wolpert, D.H.: Bias plus variance decomposition for zero-one loss functions. In: Saitta, L. (ed.) Proc. the 13th Int. Conf. Machine Learning, Bari, Italy, pp. 275–283. Morgan Kaufmann, San Francisco (1996)
Kong, E.B., Dietterich, T.G.: Error-correcting output coding corrects bias and variance. In: Prieditis, A., Russell, S.J. (eds.) Proc. the 12th Int. Conf. Machine Learning, Tahoe City, CA, pp. 313–321. ACM Press, New York (1995)
Schapire, R.E., Freund, Y., Bartlett, P., Lee, W.S.: Boosting the margin: a new explanation for the effectiveness of voting methods. The Annals of Stat. 26, 1651–1686 (1998)
Smith, R.S., Windeatt, T.: The bias variance trade-off in bootstrapped error correcting output code ensembles. In: Benediktsson, J.A., Kittler, J., Roli, F. (eds.) MCS 2009. LNCS, vol. 5519, pp. 1–10. Springer, Heidelberg (2009)
Tibshirani, R.: Bias, variance and prediction error for classification rules. Technical Report, University of Toronto, Toronto, Canada (1996)
Tumer, K., Ghosh, J.: Analysis of decision boundaries in linearly combined neural classifiers. Pattern Recogn. 29, 341–348 (1996)
Tumer, K., Ghosh, J.: Error correlation and error reduction in ensemble classifiers. Connection Science 8, 385–403 (1996)
Valentini, G., Dietterich, T.: Bias-variance analysis of support vector machines for the development of SVM-based ensemble methods. J. Machine Learning Research 5, 725–775 (2004)
Webb, G.I., Conilione, P.: Estimating bias and variance from data. School of Comp. Science and Software Engineering, Monash University (2005)
Windeatt, T., Ghaderi, R.: Coding and decoding strategies for multi-class learning problems. Inf. Fusion 4, 11–21 (2003)
Wolpert, D.H.: On bias plus variance. Neural Comp. 9, 1211–1244 (1996)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Zor, C., Windeatt, T., Yanikoglu, B. (2011). Bias-Variance Analysis of ECOC and Bagging Using Neural Nets. In: Okun, O., Valentini, G., Re, M. (eds) Ensembles in Machine Learning Applications. Studies in Computational Intelligence, vol 373. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22910-7_4
Download citation
DOI: https://doi.org/10.1007/978-3-642-22910-7_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-22909-1
Online ISBN: 978-3-642-22910-7
eBook Packages: EngineeringEngineering (R0)