Entropy coefficient of determination for generalized linear models

doi:10.1016/j.csda.2009.12.003

Computational Statistics & Data Analysis

Volume 54, Issue 5, 1 May 2010, Pages 1381-1389

https://doi.org/10.1016/j.csda.2009.12.003 Get rights and content

Abstract

The objective of the present paper is to propose a predictive power measure for generalized linear models (GLMs). First, basic predictive power measures for GLMs are compared with respect to some desirable properties. We propose a generalized coefficient of determination for GLMs, which is referred to as the entropy coefficient of determination (ECD). The advantage of the measure is discussed in the GLM framework. Second, the asymptotic properties of the maximum likelihood estimator of ECD are discussed. Third, ECD is applied to GLMs with polytomous response variables. Finally, discussions and a conclusion to this study are provided.

Introduction

Measurement of the predictive or explanatory power of GLMs (Nelder and Wedderburn, 1972, McCullagh and Nelder, 1989) is important for regression analysis as well as a model selection, i.e. determination of significant factors in regression models. GLMs include various useful regression models such as normal linear regression, logistic regression, and loglinear models, and are widely applied in practical data analyses. In normal linear regression, the coefficient of determination plays an important role in the measurement of predictive power; however regression models for non-normal responses especially polytomous ones need other measures for assessing the predictive powers. By using variation functions of response variables, generalized coefficient of determinations were proposed as proportions of the variation explained by explanatory variables (Efron, 1978, Agresti, 1986, Korn and Simon, 1991). Predictive power measures based on the likelihood function (Theil, 1970, Goodman, 1971) and entropy (Haberman, 1982) can be viewed as the above type ones. For logistic regression model, the squared sample correlation between responses and their conditional expectations and the sums-of-squares measure are considered as preferable predictive power measures (Mittlebock and Schemper, 1996, Ash and Shwarts, 1999). Zheng and Agresti (2000) recommended the correlation coefficient between responses and their conditional expectations as a predictive power measure for GLMs. The random component of the GLM is an exponential family distribution, and the covariance between the response variable and a canonical parameter is described with the Kullback–Leibler information that measures the difference between the model concerned and the independent (null) model (Eshima and Tabata, 2007). The desirable properties of the predictive power measures for GLMs may be (i) interpretability; (ii) being the multiple correlation coefficient or the coefficient of determination in normal linear regression models; (iii) entropy-based property; (iv) applicability to all GLMs; in addition to these, it may be appropriate for a measure to have the following property: (v) monotonicity in the complexity of the linear predictor.

The aim of the present paper is to propose a generalized coefficient of determination for GLMs. Section 2 proposes the entropy coefficient of determination as a predictive power measure for GLMs. Basic predictive power measures are compared with respect to desirable properties for assessing the effects of factors in GLMs. In Section 3, the asymptotic properties of the maximum likelihood estimator of ECD are discussed. Section 4 considers advantageous properties of ECD in GLMs with polytomous response variables. In Section 5, the ECD approach is applied to a logit model. Finally, in Section 6 discussions and a conclusion to this study are given.

Section snippets

Basic predictive power measures for GLMs

In the GLM framework, predictive power measures are compared, and a generalized coefficient of determination is proposed. Let $X = {(X_{1}, X_{2}, \dots, X_{p})}^{T}$ be a $p \times 1$ factor or explanatory variable vector; $Y$ be a response variable; and let $f (y | x)$ be the conditional probability or density function of $Y$ given $X = x$ . In GLMs, the function $f (y | x)$ is assumed to be the following exponential form: $f (y | x) = exp (\frac{y θ - b (θ)}{a (φ)} + c (y, φ)),$ where $θ$ and $φ$ are parameters, and $a (φ)$ , $b (θ)$ (>0) and $c (y, φ)$ are specific functions.

Asymptotic property of the ML estimator of ECD

A basic association measure, $m_{P P} (Y | X)$ , was considered in the case where $X$ is uniformly distributed or not random (Eshima and Tabata, 2007). In this section, more general cases are considered. Let $f (y)$ and $g (x)$ be the marginal density or probability function of $Y$ and $X$ , respectively. Then, the association measure is expressed as $m_{P P} (Y | X) = \iint f (y | x) g (x) log (\frac{f (y | x)}{f (y)}) d x d y + \iint f (y) g (x) log (\frac{f (y)}{f (y | x)}) d x d y .$ If $Y$ is discrete, the integral is replaced with the summation. If $X$ is not random and take values $x_{k}$

GLMs with polytomous response variables

Let $Y$ be a polytomous response variable with levels ${1, 2, \dots, K}$ and let $Y_{k} = {\begin{matrix} 1 & (Y = k) \\ 0 & (Y \neq k) . \end{matrix}$ Then, dummy variable vector $Y = {(Y_{1}, Y_{2}, \dots, Y_{K})}^{T}$ is identified with response $Y$ . Then, the random component of a GLM is described as follows: $f (y | x) = exp (\frac{y^{T} θ - b (θ)}{a (φ)} + c (y, φ)),$ where $θ = {(θ_{1}, θ_{2}, \dots, θ_{K})}^{T}$ . For model identification we set $θ_{K} = 0$ . For an appropriate link function, $θ$ is a function of explanatory variable vector $X$ . In this case, ECD is given by $ECD (X, Y) = \frac{\sum_{k = 1}^{K - 1} Cov (θ_{k}, Y_{k})}{\sum_{k = 1}^{K - 1} Cov (θ_{k}, Y_{k}) + a (φ)} .$

Theorem 3

For the GLM with a

Application to a generalized logit model

A baseline-category logit model is considered. Let $X_{1}$ and $X_{2}$ be categorical factors that take levels ${1, 2, \dots, I}$ and ${1, 2, \dots, J}$ , respectively, and let $Y$ be a categorical response variable with levels ${1, 2, \dots, K}$ . Let $X_{a i} = {\begin{matrix} 1 & (X_{a} = i) \\ 0 & (X_{a} \neq i) \end{matrix} (a = 1, 2) and Y_{k} = {\begin{matrix} 1 & (Y = k) \\ 0 & (Y \neq k) . \end{matrix}$ Then, dummy variable vectors $X_{1} = {(X_{11}, X_{12}, \dots, X_{1 I})}^{T}, X_{2} = {(X_{21}, X_{22}, \dots, X_{2 J})}^{T}$ and $Y = {(Y_{1}, Y_{2}, \dots, Y_{K})}^{T}$ are identified with factors $X_{1}, X_{2}$ and response $Y$ , respectively. From this, the systematic component of the baseline-category logit model is assumed as

Discussion

As an extension of the coefficient of determination, ECD has been proposed for GLMs. In the GLM framework, regression models are described with random, systematic and link components, and the explanatory variables are related to entropy of response variables. From this it may be suitable to propose an entropy-based predictive power measure for GLMs. In these days, GLMs are widely applied to practical data analyses, and especially canonical link GLMs play an important role in regression

Acknowledgements

The authors would like to thank the referees and the editor for their useful comments and suggestions. The first version of this paper was drastically improved. This research was supported by Grant-in-aid for Scientific Research 18500216, Ministry of Education, Culture, Sports, Science and Technology of Japan.

References (14)

N. Eshima et al.
Entropy correlation coefficient for measuring predictive power of generalized linear models
Statistics & Probability Letters
(2007)
A. Agresti
Applying $R^{2}$ -type measures to ordered categorical data
Technometrics
(1986)
A. Agresti
Categorical Data Analysis
(2002)
A. Ash et al.
$R^{2}$ : A useful measure of model performance with predicting a dichotomous outcome
Statistics in Medicine
(1999)
B. Efron
Regression and ANOVA with zero-one data: Measures of residual variation
Journal of the American Statistical Association
(1978)
L.A. Goodman
The analysis of multinomial contingency tables: Stepwise procedures and direct estimation methods for building models for multiple classifications
Technometrics
(1971)
S.J. Haberman
Analysis of dispersion of multinomial responses
Journal of the American Statistical Association
(1982)

There are more references available in the full text version of this article.

Cited by (24)

Sentiment dynamics and volatility: A study based on GARCH-MIDAS and machine learning
2024, Finance Research Letters
This work investigates the relationship between investor sentiment and volatility of stock indexes. A sentiment proxy is constructed via a machine learning approach from the consumer confidence indexes of four countries. Granger causality tests highlight the influence of sentiment on volatility. This impact is quantified via GARCH-MIDAS models that, retaining variables in their sampling frequency, allow the estimation of the long-run volatility without information loss. Sentiment is finally used to predict long-run volatility. Thus, further insights into the relationship between investor sentiment and return volatility are provided, helping investors to stabilize the former and contain its effect on market uncertainty.
Feature selection based on the best-path algorithm in high dimensional graphical models
2023, Information Sciences
This paper proposes a new algorithm for an automatic feature selection procedure in High Dimensional Graphical Models. The algorithm, called Best-Path Algorithm (BPA), rests on a filter method and performs feature selection based on mutual information. Over the last years, filter methods have been successfully employed to reduce the size of the input dataset and retain, at the same time, the relevant feature information for modeling and classification problems. However, the extant filter algorithms are mostly heuristic or require high computational effort. The BPA overcomes these drawbacks by taking advantage of the links between variables brought to the fore by the Edwards's algorithm. Once the High Dimensional Graphical Model, depicting the probabilistic structure of the variables, is determined, the BPA selects the best subset of features by analyzing its path-steps. The path-step that includes the variables with the most predictive power for the target one is then determined via the computation of the entropy correlation coefficient. This index, being based on the notion of (symmetric) Kullback-Leibler divergence, is closely connected to the mutual information that the path-step variables share with that of interest. The BPA application to simulated and real-word benchmark datasets highlights its potential and greater effectiveness compared to alternative extant methods.
Regression correlation coefficient for a Poisson regression model
2016, Computational Statistics and Data Analysis
This study examines measures of predictive power for a generalized linear model (GLM). Although many measures of predictive power for GLMs have been proposed, most have limitations. Hence, we focus on the regression correlation coefficient (RCC) (Zheng and Agresti, 2000), which satisfies the four requirements of (i) interpretability, (ii) applicability, (iii) consistency, and (iv) affinity. The RCC is a population value that is defined by the correlation between a response variable and the conditional expectation of the response variable. Its sample value is defined by the sample correlation between the observed response values and estimated values of the response variable. For an arbitrary GLM, we do not always have an explicit form of the RCC. However, for a Poisson regression model, assuming that the predictor variables have a multivariate normal distribution, we can find the explicit form of the RCC (true value). Therefore, it is possible to compare the estimators (sample values) of the RCC in terms of bias and RMSE (root of the mean square error) by using the true value. Furthermore, by using the explicit form, we propose a new estimator of the RCC for the Poisson regression model. We then compare the new estimator with the sample correlation estimator, the jack-knife estimator, and the leave-one-out cross validation estimator in terms of bias and RMSE. The leave-one-out cross validation estimator has large negative bias and large RMSE. Although the remaining three estimators show similar behavior for a large sample size, for a small sample size the new estimator shows the best behavior in terms of bias and RMSE.
Goodness of fit in restricted measurement error models
2016, Journal of Multivariate Analysis
The restricted measurement error model is employed when certain study variables are not observable by direct measurement and if some information about the unknown regression coefficients is available a priori. In this study, we present a method for checking the goodness of fit in the restricted measurement error model. We obtain the goodness-of-fit statistics based on the concept of coefficient of determination and their asymptotic distributions are derived. The results of simulations are also presented to demonstrate the finite sample behaviour of the estimators.
Coefficient of determination for multiple measurement error models
2014, Journal of Multivariate Analysis
The coefficient of determination ( $R^{2}$ ) is used for judging the goodness of fit in a linear regression model. It is the square of the multiple correlation coefficient between the study and explanatory variables based on the sample values. It gives valid results only when the observations are correctly observed without any measurement error. The conventional $R^{2}$ provides invalid results in the presence of measurement errors in the data because the sample $R^{2}$ becomes an inconsistent estimator of its population counterpart which is the square of the population multiple correlation coefficient between the study and explanatory variables. The goodness of fit statistics based on the variants of $R^{2}$ for multiple measurement error models have been proposed in this paper. These variants are based on the utilization of the two forms of additional information from outside the sample. The two forms are the known covariance matrix of measurement errors associated with the explanatory variables and the known reliability matrix associated with the explanatory variables. The asymptotic properties of the conventional $R^{2}$ and the proposed variants of $R^{2}$ like goodness of fit statistics have been studied analytically and numerically.
Three predictive power measures for generalized linear models: The entropy coefficient of determination, the entropy correlation coefficient and the regression correlation coefficient
2011, Computational Statistics and Data Analysis
Citation Excerpt :
In this paper, the coefficient is referred to as the regression correlation coefficient (RCC). The entropy correlation coefficient (ECC; Eshima and Tabata, 2007) and the entropy coefficient of determination (ECD; Eshima and Tabata, 2010) were proposed with consideration of their desirable properties: (i) interpretability; (ii) being the multiple-correlation coefficient or the coefficient of determination in normal linear regression models; (iii) the entropy-based property; and (iv) applicability to all GLMs. The aim of the present paper is to compare three predictive power measures, ECC, ECD and RCC, in theoretical aspects, and the utility of ECC and ECD is demonstrated, with ECC and ECD being applied to GLMs with polytomous responses.
In this paper, three predictive power measures for generalized linear models (GLMs) are compared, and the utility of the entropy correlation coefficient (ECC) and the entropy coefficient of determination (ECD) is demonstrated. First, ECC, ECD and the regression correlation coefficient (RCC) are briefly explained. Second, relationships of the three measures are discussed, and the necessary and sufficient condition under which ECC and RCC are equal is deduced. Third, ECC and ECD are discussed for GLMs with canonical links and polytomous response variables, and an analysis of the effects of factors in GLMs is given. Finally, a discussion of the conclusions of this study is provided.

View all citing articles on Scopus

View full text

Entropy coefficient of determination for generalized linear models

Abstract

Introduction

Section snippets

Basic predictive power measures for GLMs

Asymptotic property of the ML estimator of ECD

GLMs with polytomous response variables

Application to a generalized logit model

Discussion

Acknowledgements

Statistics & Probability Letters

Applying R2-type measures to ordered categorical data

Technometrics

Categorical Data Analysis

R2: A useful measure of model performance with predicting a dichotomous outcome

Statistics in Medicine

Regression and ANOVA with zero-one data: Measures of residual variation

Journal of the American Statistical Association

The analysis of multinomial contingency tables: Stepwise procedures and direct estimation methods for building models for multiple classifications

Technometrics

Analysis of dispersion of multinomial responses

Journal of the American Statistical Association

Applying $R^{2}$ -type measures to ordered categorical data

$R^{2}$ : A useful measure of model performance with predicting a dichotomous outcome