A modified score function estimator for multinomial logistic regression in small samples

doi:10.1016/S0167-9473(01)00048-2

Computational Statistics & Data Analysis

Volume 39, Issue 1, 28 March 2002, Pages 57-74

https://doi.org/10.1016/S0167-9473(01)00048-2 Get rights and content

Abstract

Logistic regression modelling of mixed binary and continuous covariates is common in practice, but conventional estimation methods may not be feasible or appropriate for small samples. It is well known that the usual maximum likelihood estimates (MLEs) of the log-odds-ratio parameters are biased in finite samples, and there is a non-zero probability that an MLE is infinite, i.e., does not exist. In this paper, we extend the approach proposed by Firth (Biometrika 80 (1993) 27) for bias reduction of MLEs in exponential family models to the multinomial logistic regression model, and consider general regression covariate types. The method is based on a suitable modification of the score function that removes first order bias. We apply the method in the analysis of two datasets: one is a study of disease prognosis and the other is a disease prevention trial. In a series of simulation studies in small samples, the modified-score estimates for binomial and trinomial logistic regressions had mean bias closer to zero and smaller mean squared error than other approaches. The modified-score estimates have properties that make them attractive for routine application in logistic regressions of binary and continuous covariates, including the advantage that they can be obtained in samples in which the MLEs are infinite.

Introduction

Methods for logistic regression modelling of nominal categorical responses based on the multinomial logistic likelihood are now generally available in standard statistical packages, and have been applied in the analysis of case-control studies with multiple case or multiple control groups, and in randomized trials and cross-sectional surveys with categorical responses. One of the concerns of investigators is the valid estimation of model parameters in the finite sample sizes encountered in practice. In finite samples, the usual maximum likelihood estimates (MLEs) of the log odds ratios are biased, and the bias increases as the ratio of the number of observations to the number of parameters (n-to-p ratio) decreases (Cordeiro and McCullagh, 1991; Bull et al., 1997). This is of particular concern when there are several response categories and multiple covariates because the number of parameters can become large.

We consider an alternative estimation method for small samples based on a modification of the score function that removes first order bias and is equivalent to penalizing the likelihood by Jeffreys’ prior (Firth, 1993). We extend the modified score function method to the multinomial logistic regression model with nominal response categories, and compare the modified estimates to the usual MLEs and to the MLEs corrected by an estimate of the asymptotic bias. As the sample size increases, the modified-score estimates become equivalent to the usual MLEs. As systematic small sample comparisons of this approach have not been reported previously for binomial or multinomial logistic regression models, we also present a Monte Carlo simulation study in which we compare the mean bias and mean squared error (MSE) of the modified estimates to the MLEs, and to the MLEs corrected by the estimated asymptotic bias. Considering the same series of logistic regression models studied previously (Bull et al., 1997), we find that the modified-score estimates are competitive and often superior to the other approaches.

Section snippets

Methods for small-sample analysis

The small-sample properties of the logistic regression MLEs can be improved by the general approach of Cox and Snell (1968) which uses higher order terms in a Taylor series expansion of the log-likelihood to approximate the asymptotic bias and obtain bias-corrected MLEs (Anderson and Richardson, 1979; Schaefer, 1983; Copas, 1988; Cordeiro and McCullagh, 1991; Bull et al., 1997). When the magnitude of the linear predictor is small, Cordeiro and McCullagh (1991) showed that the effect of bias

Usual maximum likelihood estimation with bias correction

We consider a multicategory outcome y that is a multinomial variable with J+1 categories. For each category j (j=1,…,J), there is a regression function in which the log odds of response in category j, relative to category 0, is a linear function of regression parameters and a vector $x$ of p covariates (including a constant): $log {prob(y=j | x)/prob(y=0 | x)}= β_{j}^{T} x$ . We let $y_{i}$ be a J×1 vector of indicators for the observed response category for observation i, with the corresponding J×1 vector of

Applications of small-sample estimation

The first application is a study of clinical factors that relate to the presence or absence of nodal involvement in patients with prostate cancer; 20 of the 53 patients have nodal involvement (Brown, 1980; Cox and Snell, 1989). Table 1 presents the estimates for a logistic regression model with six covariates: four binary indicators (X-ray, stage, grade, interaction of stage and grade) and two continuous variables (acid, age). In this dataset, the ratio of the number of observations to the

Design

The Monte Carlo study evaluated the MPLEs with respect to mean bias and MSE and compared them to the usual MLEs and the BCMLEs in multiple logistic regressions that included both binary and continuous covariates. We also calculated the mean bias and MSE for the MPLEs in all datasets, including those in which one or more of the MLEs did not exist. As in our previous study, we conducted three series of simulations to investigate the effects of sample size (200,100,75,50), type of covariates

Discussion

The applications and the Monte Carlo study demonstrate several advantages for the modified score function estimator. In the applications, it is apparent that the shrinkage effect of the modified score function operated more strongly when it was needed, i.e., when there was a large association and the ratio of observations to parameters was small, but was minimal for parameter estimates that did not require bias reduction. It was effectively applied in both very small samples and in large

Acknowledgements

During this study, S.B. Bull was a National Health Research Scholar of the National Health Research and Development Program of Health and Welfare Canada. This research was supported by the Natural Sciences and Engineering Research Council of Canada. Thanks to J.P. Lewinger for assistance in preparing the applications and for helpful discussions.

References (39)

M.A. Blajchman et al.
For the Canadian post-transfusion hepatitis prevention study group. Post-transfusion hepatitis: impact of the non-A non-B hepatitis surrogate tests
Lancet
(1995)
M. Zelen
Multinomial response models
Comput. Statist. Data Anal.
(1991)
A. Albert et al.
On the existence of maximum likelihood estimates in logistic regression models
Biometrika
(1984)
A. Albert et al.
Multivariate Interpretation of Clinical Laboratory Data.
(1984)
J.M. Alho
On the computation of likelihood and score test based confidence intervals in generalized linear models
Statist. Med.
(1992)
J.A. Anderson et al.
Logistic discrimination and bias correction in maximum likelihood estimation
Technometrics
(1979)
Aptech Systems Incorporated, 1990. The GAUSS System, Version 2.0, Kent,...
B.W. Brown
Prediction analyses for binary data.
S.B. Bull et al.
Two-step jackknife bias reduction for logistic regression MLEs
Commun. Statist.—Simulation Comput.
(1994)
S.B. Bull et al.
Jackknife bias reduction for polychotomous logistic regression
Statist. Med.
(1997)

J.B. Copas

Binary regression models for contaminated data, with discussion

J. Royal Statist. Soc. Ser. B

(1988)

G.M. Cordeiro et al.

Bias correction in generalized linear models

J. Royal Statist. Soc. Ser. B

(1991)

D.R. Cox et al.

A general definition of residuals

J. Royal Statist. Soc. Ser. B

(1968)

D.R. Cox et al.

The Analysis of Binary Data.

(1989)

Cytel Software Corporation, 1992. LogXact-Turbo: A Software Package for Exact and Asymptotic Logistic Regression,...

D.E. Duffy et al.

On the small sample properties of norm-restricted maximum likelihood estimators for logistic regression models

Commun. Statist.—Theory Meth.

(1989)

V.T. Farewell

Jackknife estimation with structured data

Biometrika

(1978)

D. Firth

Generalized linear models and Jeffreys priors: an iterative weighted least-squares approach.

D. Firth

Bias reduction, the Jeffreys prior and GLIM.

Cited by (71)

The analysis of influences of attitudes on mode choice under highly unbalanced mode share patterns.
2020, Journal of Choice Modelling
The aim of the study is to examine the potential effect of attitudes towards physical activity on bus utility in the context of a rural area where studies have shown that the level and opportunity for physical activity are generally low. The need to analyze attitudes remains a strong motivation for the application of integrated choice and latent variable models. As such, we integrated attitudes towards physical activity with two specific attitudes towards bus and car use in a binary logit mode choice model between car and bus, looking at the area of Asuke, Japan. As we found data separation due to highly unbalanced mode shares in the input data, the choice models were estimated with the likelihood function penalized. While only a few parameters were found to be significant, arguably as a result of the unbalanced mode share pattern, attitude variables were almost unaffected by the data separation phenomenon. Whereas maximum likelihood estimates do not exist in the presence of data separation, the employed penalized maximum likelihood estimator was demonstrated to be a solution to this problem. Thus, we suggest that checking for data separation in the case of highly unbalanced mode share patterns is important and if data separation exits, penalizing the likelihood functions can be a solution rather than excluding out some irrelevant variables to avoid data separation. Overall, we found that attitude towards physical activity had a significant effect on bus utility, suggesting that policymakers could use this factor to connect transport and health policies.
Stress appraisals influence athletic performance and psychophysiological response during 16.1 km cycling time trials
2020, Psychology of Sport and Exercise
We examined how stress appraisals were associated with emotions, coping behaviours, as well as subjective and objective measures of performance.
Prospective field- and laboratory-based studies.
In Study 1, 192 athletes completed process-oriented psychometrics pertaining to the aforementioned constructs throughout a sporting competition. Study 2 utilised an experimental design to assess the causal influence of stress appraisals on performance, cortisol, and psychological variables. Thirty gender-matched athletes were randomly assigned to either a stress appraisal (e.g., challenge, threat, benefit, or harm/loss) or the control group. Participants completed three 16.1 km cycling time trials (TT) on a cycle ergometer, with their appropriate stress appraisal engendered via falsified performance feedback throughout the final TT. Salivary cortisol samples and psychometrics (e.g., appraisals, emotions, and coping) were collected before and after each TT.
The results of Study 1 revealed a sequential link between challenge stress appraisals and perceived goal attainment via pleasant emotions and task-oriented coping behaviours. Threat stress appraisals inversely related to goal attainment via unpleasant emotions and both distraction- and disengagement-oriented coping. In Study 2, no significant psychophysiological or performance differences were found across genders. The temporal orientation of stress appraisals influenced objective cycling TT performance. Benefit and harm/loss stress appraisals significantly facilitated or inhibited performance, respectively. Cortisol spikes were observed in the stress appraisal group’s threat, challenge, and benefit, with a decline detected within the harm/loss group. Whilst the process of winning is physiologically stressful, the fear of defeat may be more stressful than losing itself.
Stress appraisals influence subjective and objective performance, as well as neuroendocrine and psychological responses to stress.
Using Firth's method for model estimation and market segmentation based on choice data
2019, Journal of Choice Modelling
Citation Excerpt :
As we show in this paper, a major advantage of the method is that it allows fitting a MNL model to individual-level data, and subsequently, exploring the heterogeneity in the respondents' preferences and segmenting the market. Bull et al. (2002) were the first to propose Firth's method to estimate the MNL model, but they applied it to small sample clinical trials outside a choice modeling context, and did not consider individual-level data. Firth's method was originally developed as a general bias reducing technique in the context of ML estimation, but it was also shown to provide finite parameter estimates in the case of separation (see, for binomial and trinomial logistic regression on clinical data, Bull et al. (2002), Heinze and Schemper (2002) and Heinze (2006)).
Using maximum likelihood (ML) estimation for discrete choice modeling of small datasets causes two problems. The first problem is that the data may exhibit separation, in which case the ML estimates do not exist. Also, provided they exist, the ML estimates are biased. In this paper, we show how to adapt Firth's penalized likelihood estimation for use in discrete choice modeling. A powerful advantage of Firth's estimation is that, unlike ML estimation, it provides useful estimates in the case of data separation. For aggregates of six or more respondents, Firth estimates have negligible bias. For preference estimates on an individual level, Firth estimates show little bias as long as each person evaluates a sufficient number of choice sets. Additionally, Firth's individual-level estimation makes it possible to construct an empirical distribution of the respondents' preferences without imposing any a priori population distribution and to effectively predict people's choices and detect market segments. Segment recovery may even be better when individual-level estimates are obtained using Firth's method instead of hierarchical Bayes estimation under a normal prior. We base all findings on data from a stated choice study on various forms of employee compensation.
Neonatal Sepsis of Early Onset, and Hospital-Acquired and Community-Acquired Late Onset: A Prospective Population-Based Cohort Study
2018, Journal of Pediatrics
Citation Excerpt :
We then fitted a multivariable model containing all predictors simultaneously. Because the distribution of certain variables was unbalanced between groups, we used Firth penalized maximum likelihood bias reduction method for multinomial regression39 and Cox regression40 and calculated corresponding CIs. We considered only the first episode of each patient in regression analyses to avoid conditionality between repeated sepsis episodes in the same patient.
To assess the epidemiology of blood culture-proven early- (EOS) and late-onset neonatal sepsis (LOS).
All newborn infants admitted to tertiary care neonatal intensive care units in Switzerland and presenting with blood culture-proven sepsis between September 2011 and December 2015 were included in the study. We defined EOS as infection occurring <3 days after birth, and LOS as infection ≥3 days after birth. Infants with LOS were classified as having community-acquired LOS if onset of infection was ≤48 hours after admission, and hospital-acquired LOS, if onset was >48 hours after admission. Incidence was estimated based on the number of livebirths in Switzerland and adjusted for the proportion of admissions at centers participating in the study.
We identified 444 episodes of blood culture-proven sepsis in 429 infants; 20% of cases were EOS, 62% hospital-acquired LOS, and 18% community-acquired LOS. The estimated national incidence of EOS, hospital-acquired LOS, and community-acquired LOS was 0.28 (95% CI 0.23-0.35), 0.86 (0.76-0.97), and 0.28 (0.23-0.34) per 1000 livebirths. Compared with EOS, hospital-acquired LOS occurred in infants of lower gestational age and was more frequently associated with comorbidities. Community-acquired LOS was more common in term infants and in male infants. Mortality was 18%, 12%, and 0% in EOS, hospital-acquired LOS, and community-acquired LOS, and was higher in preterm infants, in infants with septic shock, and in those requiring mechanical ventilation.
We report a high burden of sepsis in neonates with considerable mortality and morbidity. EOS, hospital-acquired LOS, and community-acquired LOS affect specific patient subgroups and have distinct clinical presentation, pathogens and outcomes.
Latent Class Analysis of Online Platforms for Partner-Seeking and Sexual Behaviors Among Men Who Have Sex with Men from Central Kentucky
2024, AIDS and Behavior
Penalized maximum likelihood inference under the mixture cure model in sparse data
2023, Statistics in Medicine

View all citing articles on Scopus

View full text

A modified score function estimator for multinomial logistic regression in small samples

Abstract

Introduction

Section snippets

Methods for small-sample analysis

Usual maximum likelihood estimation with bias correction

Applications of small-sample estimation

Design

Discussion

Acknowledgements

Lancet

Comput. Statist. Data Anal.

On the existence of maximum likelihood estimates in logistic regression models

Biometrika

Multivariate Interpretation of Clinical Laboratory Data.

On the computation of likelihood and score test based confidence intervals in generalized linear models

Statist. Med.

Logistic discrimination and bias correction in maximum likelihood estimation

Technometrics

Prediction analyses for binary data.

Two-step jackknife bias reduction for logistic regression MLEs

Commun. Statist.—Simulation Comput.

Jackknife bias reduction for polychotomous logistic regression

Statist. Med.

Binary regression models for contaminated data, with discussion

J. Royal Statist. Soc. Ser. B

Bias correction in generalized linear models

J. Royal Statist. Soc. Ser. B

A general definition of residuals

J. Royal Statist. Soc. Ser. B

The Analysis of Binary Data.

On the small sample properties of norm-restricted maximum likelihood estimators for logistic regression models

Commun. Statist.—Theory Meth.

Jackknife estimation with structured data

Biometrika

Generalized linear models and Jeffreys priors: an iterative weighted least-squares approach.

Bias reduction, the Jeffreys prior and GLIM.