An Entropy-Based Approach for Measuring Factor Contributions in Factor Analysis Models

Abstract

In factor analysis, factor contributions of latent variables are assessed conventionally by the sums of the squared factor loadings related to the variables. First, the present paper considers issues in the conventional method. Second, an alternative entropy-based approach for measuring factor contributions is proposed. The method measures the contribution of the common factor vector to the manifest variable vector and decomposes it into contributions of factors. A numerical example is also provided to demonstrate the present approach.

1. Introduction

Factor analysis is a statistical method for extracting simple structures to explain inter-relations between manifest and latent variables. The origin dates back to the works of [1], and the single factor model was extended to the multiple factor model [2]. These days, factor analysis is widely applied in behavioral sciences [3]; hence, it is important to interpret the extracted factors and is critical to explain how such factors influence manifest variables, that is, measurement of factor contribution. Let $X_i$ be manifest variables; ${\xi }_j$ latent variables (common factors); ${\epsilon }_i$ unique factors related to $X_i$; and let ${\lambda }_{ij}$ be factor loadings that are weights of factors ${\xi }_j$ to explain $X_i$. Then, the factor analysis model is given as follows:

$$X_i={\sum }_{j=1}^m{\lambda }_{ij}{\xi }_j+{\epsilon }_i, i=1,2,\dots ,p,$$

(1)

where

$$\begin{gathered} \mathrm{E}\left(X_i\right)=\mathrm{E}\left({\xi }_j\right)=\mathrm{E}\left({\epsilon }_i\right)=0, \mathrm{var}\left({\xi }_j\right)=1,\mathrm{cov}\left({\xi }_j,{\epsilon }_i\right)=0, \\ \mathrm{cov}\left({\epsilon }_i,{\epsilon }_k\right)=0 \mathrm{for} i\ne k\mathrm{and}\mathrm{var}\left({\epsilon }_i\right)={\sigma }_i^2>0 \end{gathered}$$

For the simplicity of discussion, common factors ${\xi }_j$are assumed to be mutually independent in this section, that is, we first consider an orthogonal factor analysis model. In the conventional approach, the contribution of factor ${\xi }_j$ to all manifest variables$X_i$, $C_j$, is defined as follows:

$$C_j={\sum }_{i=1}^p\mathrm{cov}{\left(X_i,{\xi }_j\right)}^2={\sum }_{i=1}^p{\lambda }_{ij}^2$$

(2)

The above definition of factor contributions is based on the following decomposition of the total of variances of the observed variables$X_i$ [4] (p. 59):

$${\sum }_{i=1}^p\mathrm{var}\left(X_i\right)={\sum }_{j=1}^m{\sum }_{i=1}^p{\lambda }_{ij}^2+{\sum }_{i=1}^p{\sigma }_i^2$$

What physical meaning does the above quantity have? Applying it to the manifest variables observed, however, such a decomposition leads to scale-variant results. For this reason, factor contribution is usually considered on the standardized versions of manifest variables$X_i$. What does it mean to measure factor contributions by (2)? For standardized manifest variables$X_i$, we have

$${\lambda }_{ij}=\mathrm{cor}\left(X_i,{\xi }_j\right)$$

(3)

Then, (2) is the sum of the coefficients of determination for all standardized manifest variables$X_i$with respect to a single latent variable${\xi }_j$. The squared correlation coefficients (3), that is, $\mathrm{cor}{\left(X_i,{\xi }_j\right)}^2$, are the ratios of explained variances of a manifest variable$X_i$, and in this sense, they can be interpreted as the contributions (effects) of factors ${\xi }_j$ to the manifest variable $X_i$. Although, what does the sum of these with respect to all manifest variables $X_i$, that is, (2), mean? The conventional method may be intuitively reasonable for measuring factor contributions; however, we think it is sensible to propose a method measuring factor contributions as the effects of factors on the manifest variable vector $\mathit{X}=\left(X_1,X_2,\dots ,X_p\right)$, which are interpretable and have a theoretical basis. There is no research on this topic as far as we have searched. The present paper provides an entropy-based solution to the problem. Entropy is a useful concept to measure the uncertainty in the systems of random variables and sample spaces [5] and it can be applied to measure multivariate dependences of random variables [6,7].

This paper proposes an entropy-based method for measuring factor contributions of ${\xi }_j$ to the manifest variable vector $\mathit{X}=\left(X_1,X_2,\dots ,X_p\right)$ concerned, which can treat not only orthogonal factors, but also oblique cases. The present paper has five sections in addition to this section. In Section 2, the conventional method for measuring factor contributions is reviewed. Section 3 considers the factor analysis model in view of entropy and makes a preliminary discussion on measurement of factor contribution. In Section 4, an entropy-based path analysis is applied as a tool to measure factor contributions. Contributions of factors ${\xi }_j$are defined by the total effects of the factors on the manifest variable vector, and the contributions are decomposed into those to manifest variables and subsets of manifest variables. Section 5 illustrates the present method using a numerical example. Finally, in Section 6, some conclusions are provided.

2. Relative Factor Contributions in the Conventional Method

In the conventional approach, for the orthogonal factor model (1), the contribution ratio of ${\xi }_j$ is defined by

$${\mathrm{RC}}_j=\frac{C_j}{{{\displaystyle \sum }}_{l=1}^m{C}_l}=\frac{{{\displaystyle \sum }}_{i=1}^p{\lambda }_{ij}^2.}{{{\displaystyle \sum }}_{l=1}^m{{\displaystyle \sum }}_{k=1}^p{\lambda }_{kl}^2}$$

(4)

The above measure is referred to as the factor contribution ratio in the common factor space. Let $R_i$ be the multiple correlation coefficient of latent variable vector $\mathit{\xi }={\left({\xi }_1,{\xi }_2,\dots ,{\xi }_m\right)}^T$ and manifest variable$X_i$. Then, for standardized manifest variable$X_i$, we have

$$R_i^2={\sum }_{j=1}^m{\lambda }_{ij}^2$$

(5)

The above quantity can be interpreted as the effect (explanatory power) of latent variable vector $\mathit{\xi }=\left({\xi }_j\right)$on manifest variable$X_i$; however, the denominator of (4) is the sum of those effects (5) and there is no theoretical basis to interpret it. Another contribution ratio of${\xi }_j$is referred to as that in the whole space of $\mathit{X}=\left(X_i\right)$, and is defined by

$${\widetilde{\mathrm{RC}}}_j=\frac{C_j}{{{\displaystyle \sum }}_{i=1}^p\mathrm{var}\left(X_i\right)}=\frac{{{\displaystyle \sum }}_{k=1}^p{\lambda }_{kj}^2}{{{\displaystyle \sum }}_{k=1}^p\left({{\displaystyle \sum }}_{l=1}^m{\lambda }_{kl}^2+{\sigma }_k^2\right)}$$

(6)

If the manifest variables are standardized, we have

$${\widetilde{\mathrm{RC}}}_j=\frac{C_j}{p}=\frac{{{\displaystyle \sum }}_{k=1}^p{\lambda }_{kj}^2}{p}$$

Here, there is an issue similar to (4), because the denominator in (6) does not express the variation of the manifest variable vector $\mathit{X}=\left(X_i\right)$. Indeed, it is the sum of the variances of manifest variables and does not include covariances between them. In the next section, the factor analysis model (1) is reconsidered in the framework of generalized linear models (GLMs), and the effects (contributions) of latent variables ${\xi }_j$ on the manifest variable vector $\mathit{X}=\left(X_i\right)$, that is, factor contributions, are discussed through entropy [8].

3. Factor Analysis Model and Entropy

It is assumed that factors ${\epsilon }_i$ and ${\xi }_j$ are normally distributed, and the factor analysis model (1) is reconsidered in the GLM framework. Let $\Lambda =\left({\lambda }_{ij}\right)$ be a $p\times m$ factor loading matrix; let $\Phi$ be an $m\times m$ correlation matrix of common factor vector $\mathit{\xi }={\left({\xi }_1,{\xi }_2,\dots ,{\xi }_m\right)}^T$; and let $\Omega$ be the$p\times p$ variance-covariance matrix of unique factor vector $\epsilon ={\left({\epsilon }_1,{\epsilon }_2,\dots ,{\epsilon }_p\right)}^T$. The conditional density function of $\mathit{X}$ given$\mathit{\xi }$, $f\left(\mathit{x}|\mathit{\xi }\right)$, is normal with mean $\Lambda \mathit{\xi }$ and variance matrix$\Omega$, and is given as follows:

$$f\left(\mathit{x}|\mathit{\xi }\right)=\frac{1}{{\left(2\pi \right)}^{\frac{p}{2}}{\left|\Omega \right|}^{\frac{1}{2}}}\exp \left(\frac{{\mathit{X}}^T\tilde{\Omega }\Lambda \mathit{\xi }-\frac{1}{2}{\mathit{\xi }}^T{\Lambda }^T{\tilde{\Omega }}^2\Lambda \mathit{\xi }}{\left|\Omega \right|}-\frac{\frac{1}{2}{\mathit{X}}^T\tilde{\Omega }X}{\left|\Omega \right|}\right)$$

where $\tilde{\Omega }$ is the cofactor matrix of $\Omega$. Let $f\left(\mathit{x}\right)$ and $g\left(\mathit{\xi }\right)$ be the marginal density functions of $\mathit{X}$ and $\mathit{\xi }$, respectively. Then, a basic predictive power measure for GLMs [9] is based on the Kullback–Leibler information [6], and applying it to the above model, we have

$$\mathrm{KL}\left(\mathit{X}, \mathit{\xi }\right)=\iint f\left(\mathit{x}|\mathit{\xi }\right)g\left(\mathit{\xi }\right)\log \frac{f\left(\mathit{x}|\mathit{\xi }\right)}{f\left(\mathit{x}\right)}d\mathit{x}d\mathit{\xi }+\iint f\left(\mathit{x}\right)g\left(\mathit{\xi }\right)\log \frac{f\left(\mathit{x}\right)}{f\left(\mathit{x}|\mathit{\xi }\right)}d\mathit{x}d\mathit{\xi }=\frac{\mathrm{tr}\tilde{\Omega }\Lambda \Phi {\Lambda }^T}{\left|\Omega \right|}$$

(7)

The above measure was derived from a discussion on log odds ratios in GLMs [9], and is scale-invariant with respect to manifest variables $X_i$. The numerator of (7) is the explained entropy of $\mathit{X}$ by $\mathit{\xi }$, and the denominator is the dispersion of the unique factors in entropy, that is, the generalized variance of $\epsilon ={\left({\epsilon }_1,{\epsilon }_2,\dots ,{\epsilon }_p\right)}^T$. Thus, (7) expresses the total effect (contribution) of factor vector $\mathit{\xi }=\left({\xi }_j\right)$on manifest variable vector $\mathit{X}=\left(X_i\right)$ in entropy, and is denoted by $\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)$ in the present paper. The entropy coefficient of determination (ECD) is calculated as follows [9]:

$$\mathrm{ECD}\left(\mathit{X}, \mathit{\xi }\right)=\frac{\mathrm{tr}\tilde{\Omega }\Lambda \Phi {\Lambda }^T}{\mathrm{tr}\tilde{\Omega }\Lambda \Phi {\Lambda }^T+\left|\Omega \right|}$$

(8)

The denominator of the above measure is interpreted as the variation of manifest variable vector $\mathit{X}=\left(X_i\right)$ in entropy and the numerator is the explained variation of random vector $\mathit{X}$ in entropy. In this sense, ECD (8) is the factor contribution ratio of $\mathit{\xi }=\left({\xi }_j\right)$ for the whole entropy space of $\mathit{X}=\left(X_i\right)$, and it expresses the standardized total effect of $\mathit{\xi }={\left({\xi }_1,{\xi }_2,\dots ,{\xi }_m\right)}^T$ on the manifest variable vector $\mathit{X}={\left(X_1,X_2,\dots ,X_p\right)}^T$, which is denoted by $e_T\left(\mathit{\xi }\to \mathit{X}\right)$ [8,10]. As for (6), in the present paper, the ECD is denoted by $\widetilde{\mathrm{RC}}\left(\mathit{\xi }\to \mathit{X}\right)$, that is, the relative contribution of factor vector $\mathit{\xi }$for the whole space of manifest variable vector $\mathit{X}$ in entropy.

Remark 1.

Let $\Sigma$ be the $p\times p$ variance-covariance matrix of manifest variable vector $\mathit{X}={\left(X_1,X_2,\dots ,X_p\right)}^T$ and let $\Phi$ be the $m\times m$ correlation matrix of$\mathit{\xi }$. Then, we have

$$\Sigma = \Lambda \Phi {\Lambda }^T+\Omega$$

(9)

For assessing the goodness-of-fit of the models, the following overall coefficient of determination (OCD) is suggested ([11], p. 60) on the basis of (9):

$$\mathrm{OCD}\left(\mathit{X}, \mathit{\xi }\right)=1-\frac{\left|\Omega \right|}{\left|\Sigma \right|}\left(=\frac{\left|\Sigma \right|-\left|\Omega \right|}{\left|\Sigma \right|}\right)$$

Determinant $\left|\Omega \right|$ is the generalized variance of unique factor vector $\epsilon ={\left({\epsilon }_1,{\epsilon }_2,\dots ,{\epsilon }_p\right)}^T$ and $\left|\Sigma \right|$ is that of manifest variable vector $\mathit{X}={\left(X_1,X_2,\dots ,X_p\right)}^T$. Then, OCD is interpreted as the ratio of the explained generalized variance of manifest variable vector $\mathit{X}={\left(X_1,X_2,\dots ,X_p\right)}^T$ by common factor vector $\mathit{\xi }={\left({\xi }_1,{\xi }_2,\dots ,{\xi }_m\right)}^T$in the p-dimensional Euclidian space. On the other hand, from (8), it follows that

$$\mathrm{ECD}\left(\mathit{X}, \mathit{\xi }\right)=1-\frac{\left|\Omega \right|}{tr\tilde{\Omega }\Lambda \Phi {\Lambda }^T+\left|\Omega \right|}$$

Hence, ECD is viewed as the ratio of the explained variation of the manifest variable vector in entropy.

Cofactor matrix $\tilde{\Omega }$is diagonal and the $\left(i,i\right)$ elements are ${\displaystyle \prod }_{k\ne i}{\sigma }_k^2, i=1,2,\dots ,p$. If common factors are statistically independent, it follows that

$$\begin{gathered} \mathrm{tr}\tilde{\Omega }\Lambda \Phi {\Lambda }^T={{\displaystyle \sum }}_{i=1}^p{\displaystyle \prod }_{k\ne i}{\sigma }_k^2{{\displaystyle \sum }}_{j=1}^m{\lambda }_{ij}^2 \\ ={{\displaystyle \sum }}_{i=1}^p{{\displaystyle \sum }}_{j=1}^m{\lambda }_{ij}^2{\displaystyle \prod }_{k\ne i}{\sigma }_k^2 \end{gathered}$$

Thus, (7) is decomposed as

$$\mathrm{KL}\left(\mathit{X}, \mathit{\xi }\right)={\sum }_{i=1}^p{\sum }_{j=1}^m\frac{{\lambda }_{ij}^2}{{\sigma }_i^2}$$

As detailed below, in the present paper, the contribution of factor ${\xi }_j$ to $\mathit{X}$, $\mathrm{C}\left({\xi }_j\to \mathit{X}\right)$, is defined by

$$\mathrm{C}\left({\xi }_j\to \mathit{X}\right)={\sum }_{i=1}^p\frac{{\lambda }_{ij}^2}{{\sigma }_i^2}$$

(10)

Remark 2.

The above contribution is different from the conventional definition of factor contribution (2); unless ${\sigma }_i^2=1, i=1,2,\dots ,p$. In this sense, we may say that the standardization of manifest variables in entropy is obtained by setting all the unique factor variances to one.

In the next section, the contributions (effects) of factors${\xi }_j$to manifest variable vector$\mathit{X}$ are discussed in a general framework through an entropy-based path analysis [8].

4. Measurement of Factor Contribution Based on Entropy

A path diagram for the factor analysis model is given in Figure 1, in which the single-headed arrows imply the directions of effects of factors and the double-headed curved arrows indicate the associations between the related variables. In this section, common factors are assumed to be correlated, that is, we consider an oblique case, and an entropy-based path analysis [8] is applied to make a general discussion in the measurement of factor contributions.

Theorem 1.

In the factor analysis model (1),

$$\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)={\sum }_{i=1}^p\mathrm{KL}\left(X_i,\mathit{\xi }\right)$$

Proof. 

Let $f_i\left(x_i|\mathit{\xi }\right)$ be the conditional density functions of manifest variables $X_i$, given factor vector $\mathit{\xi }$; let $f_i\left(x_i\right)$ be the marginal density functions of $X_i$; let $f\left(\mathit{x}\right)$ be the marginal density function of $\mathit{X}$; and let $g\left(\mathit{\xi }\right)$ be the marginal density function of common factor vector$\mathit{\xi }$. As the manifest variables are conditionally independent, given factor vector $\mathit{\xi }$, the conditional density function of $\mathit{X}$ is

$$f\left(\mathit{x}|\mathit{\xi }\right)={{\displaystyle \prod }}_{i=1}^p{f}_i\left(x_i|\mathit{\xi }\right)$$

From (7), we have

$$\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)= \iint {{\displaystyle \prod }}_{i=1}^p{f}_i\left(x_i|\mathit{\xi }\right)g\left(\mathit{\xi }\right)\log \frac{{{\displaystyle \prod }}_{k=1}^p{f}_k\left(x_k|\mathit{\xi }\right)}{f\left(\mathit{x}\right)}d\mathit{x}d\mathit{\xi }+ \iint f\left(\mathit{x}\right)g\left(\mathit{\xi }\right)\log \frac{f\left(\mathit{x}\right)}{{{\displaystyle \prod }}_{k=1}^p{f}_k\left(x_k|\mathit{\xi }\right)}d\mathit{x}d\mathit{\xi }$$

$$\begin{array}{l}=\iint \left({{\displaystyle \prod }}_{i=1}^p{f}_i\left(x_i|\mathit{\xi }\right)g\left(\mathit{\xi }\right)-f\left(\mathit{x}\right)g\left(\mathit{\xi }\right)\right)\log {{\displaystyle \prod }}_{k=1}^p{f}_k\left(x_k|\mathit{\xi }\right)d\mathit{x}d\mathit{\xi }\\ =\iint {{\displaystyle \prod }}_{i=1}^p{f}_i\left(x_i|\mathit{\xi }\right)g\left(\mathit{\xi }\right)\log \frac{{{\displaystyle \prod }}_{k=1}^p{f}_k\left(x_k|\mathit{\xi }\right)}{{{\displaystyle \prod }}_{k=1}^p{f}_k\left(x_k\right)}d\mathit{x}d\mathit{\xi }\\ +\iint f\left(\mathit{x}\right)g\left(\mathit{\xi }\right)\log \frac{{{\displaystyle \prod }}_{k=1}^p{f}_k\left(x_k\right)}{{{\displaystyle \prod }}_{k=1}^p{f}_k\left(x_k|\mathit{\xi }\right)}d\mathit{x}d\mathit{\xi }\end{array}$$

$$\begin{gathered} ={{\displaystyle \sum }}_{k=1}^p\iint {{\displaystyle \prod }}_{i=1}^p{f}_i\left(x_i|\mathit{\xi }\right)g\left(\mathit{\xi }\right)\log \frac{f_k\left(x_k|\mathit{\xi }\right)}{f_k\left(x_k\right)}d\mathit{x}d\mathit{\xi }+{{\displaystyle \sum }}_{k=1}^p\iint f\left(\mathit{x}\right)g\left(\mathit{\xi }\right)\log \frac{f_k\left(x_k\right)}{f_k\left(x_k|\mathit{\xi }\right)}d\mathit{x}d\mathit{\xi } \\ ={{\displaystyle \sum }}_{k=1}^p\iint f_k\left(x_k|\mathit{\xi }\right)g\left(\mathit{\xi }\right)\log \frac{f_k\left(x_k|\mathit{\xi }\right)}{f_k\left(x_k\right)}d{x}_{k}d\mathit{\xi } \\ +{{\displaystyle \sum }}_{k=1}^p\iint f_k\left(x_k\right)g\left(\mathit{\xi }\right)\log \frac{f_k\left(x_k\right)}{f_k\left(x_k|\mathit{\xi }\right)}d{x}_{k}d\mathit{\xi } \end{gathered}$$

$$={{\displaystyle \sum }}_{i=1}^p\left(\iint f_i\left(x_i|\mathit{\xi }\right)g\left(\mathit{\xi }\right)\log \frac{f_i\left(x_i|\mathit{\xi }\right)}{f_i\left(x_i\right)}d{x}_{i}d\mathit{\xi }+\iint f_i\left(x_i\right)g\left(\mathit{\xi }\right)\log \frac{f_i\left(x_i\right)}{f_i\left(x_i|\mathit{\xi }\right)}d{x}_{i}d\mathit{\xi }\right)={{\displaystyle \sum }}_{i=1}^{p}KL\left(X_i,\mathit{\xi }\right)$$

 ☐

In model (1) with correlation matrix $\Phi =\left({\phi }_{ij}\right)$, we have

$$\mathrm{KL}\left(X_i,\mathit{\xi }\right)=\frac{{{\displaystyle \sum }}_{k=1}^m{{\displaystyle \sum }}_{l=1}^m{\lambda }_{ik}{\phi }_{kl}{\lambda }_{il}}{{\sigma }_i^2}$$

The above quantity is referred to as the contribution of $\mathit{\xi }$ to $X_i$, and is denoted as $\mathrm{C}\left(\mathit{\xi }\to X_i\right)$. Let $R_i$ be the multiple correlation coefficient of$X_{i }$and $\mathit{\xi }=\left({\xi }_j\right)$. Then,

$$\mathrm{C}\left(\mathit{\xi }\to X_i\right)=\frac{R_i^2}{1-R_i^2}\left(=\mathrm{KL}\left(X_i, \mathit{\xi }\right)\right)$$

(11)

From Theorem 1, we then have

$$\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)={\sum }_{i=1}^p\frac{R_i^2}{1-R_i^2}\left(=\mathrm{KL}\left(\mathit{X}, \mathit{\xi }\right)\right)$$

(12)

Hence, Theorem 1 gives the following decomposition of the contribution of $\mathit{\xi }$ on $X$ into those on the single manifest variables $X_i$ (11):

$$\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)={\sum }_{i=1}^p\mathrm{C}\left(\mathit{\xi }\to X_i\right)$$

(13)

Remark 3.

Notice that in the denominator of (4), the total contribution of all factors ${\xi }_i$ is simply defined as the total sum assessed:

$${\sum }_{l=1}^m{C}_l={\sum }_{i=1}^p{R}_i^2$$

On the other hand, in the present approach, the total effect (contribution) of factor vector $\mathit{\xi }$ on manifest variable vector $\mathit{X}$ is decomposed into those of manifest variables $X_i, \left(12\right) \mathrm{and} \left(13\right)$.

Let ${\mathit{X}}_{sub}$ be any sub-vector of manifest variable vector $\mathit{X}={\left(X_1,X_2,\dots ,X_p\right)}^T$. Then, the contribution of factor vector $\mathit{\xi }$ to ${\mathit{X}}_{sub}$ is defined by

$$\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{sub}\right)=\mathrm{KL}\left({\mathit{X}}_{sub},\mathit{\xi }\right)$$

From Theorem 1, we have the following corollary.

Corollary 1.

Let${\mathit{X}}_{\left(1\right)}={\left(X_{i_1},X_{i_2},\dots ,X_{i_q}\right)}^T$and${\mathit{X}}_{\left(2\right)}={\left(X_{j_1},X_{j_2},\dots ,X_{j_{p-q}}\right)}^T$be a decomposition of manifest variable vector$\mathit{X}={\left(X_1,X_2,\dots ,X_p\right)}^T$, where $q<p$. Then, for factor analysis model (1), it follows that

$$\begin{gathered} \mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)=\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(1\right)}\right)+\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(2\right)}\right) \\ \mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(1\right)}\right)={{\displaystyle \sum }}_{k=1}^q\mathrm{C}\left(\mathit{\xi }\to X_{i_k}\right), \mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(2\right)}\right)={{\displaystyle \sum }}_{k=1}^{p-q}\mathrm{C}\left(\mathit{\xi }\to X_{j_k}\right) \end{gathered}$$

Proof: 

From a similar discussion to the proof of Theorem 1, we have

$$\begin{gathered} \mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)=\mathrm{KL}\left({\mathit{X}}_{\left(1\right)},\mathit{\xi }\right)+\mathrm{KL}\left({\mathit{X}}_{\left(2\right)},\mathit{\xi }\right) \\ \mathrm{KL}\left({\mathit{X}}_{\left(1\right)},\mathit{\xi }\right)={\sum }_{k=1}^q\mathrm{KL}\left(X_{i_k},\mathit{\xi }\right), \mathrm{KL}\left({\mathit{X}}_{\left(2\right)},\mathit{\xi }\right)={\sum }_{k=1}^{p-q}\mathrm{KL}\left(X_{j_k},\mathit{\xi }\right) \end{gathered}$$

Hence, the corollary follows.

Next, the standardized total effects of single factors ${\xi }_j$on manifest variable vector$\mathit{X}$, that is, $e_T\left({\xi }_j\to \mathit{X}\right)$, are calculated [8,10]. Let ${\mathit{\xi }}^{/j}={\left({\xi }_1,{\xi }_2,\dots ,{\xi }_{j-1},{\xi }_{j+1},\dots ,{\xi }_m\right)}^T$; $f\left(\mathit{x},{\mathit{\xi }}^{/j}|{\xi }_j\right)$ be the conditional density function of $\mathit{X}$ and${\mathit{\xi }}^{/j}$ given${\xi }_j$; $f\left(\mathit{x}|{\xi }_j\right)$be the conditional density function of$\mathit{X}$ given${\xi }_j$; $g\left({\mathit{\xi }}^{/j}|{\xi }_j\right)$be the conditional density function of${\mathit{\xi }}^{/j}$given${\xi }_j$; and $g_j\left({\xi }_j\right)$ be the marginal density function of ${\xi }_j$. Then, we have

$$\begin{array}{l}\mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{/j}|{\xi }_j\right)=\iint f\left(\mathit{x},\mathit{\xi }\right)\log \frac{f\left(\mathit{x},{\mathit{\xi }}^{/j}|{\xi }_j\right)}{f\left(\mathit{x}|{\xi }_j\right)g\left({\mathit{\xi }}^{/j}|{\xi }_j\right)}d\mathit{x}d{\mathit{\xi }}^{/j}d{\xi }_j\\ +\iint f\left(\mathit{x}|{\xi }_j\right)g\left({\mathit{\xi }}^{/j}|{\xi }_j\right)g_j\left({\xi }_j\right)\log \frac{f\left(\mathit{x}|{\xi }_j\right)g\left({\mathit{\xi }}^{/j}|{\xi }_j\right)}{f\left(\mathit{x},{\mathit{\xi }}^{/j}|{\xi }_j\right)}d\mathit{x}d{\mathit{\xi }}^{/j}d{\xi }_j=\frac{\mathrm{tr}\tilde{\Omega }\Lambda \mathrm{cov}\left(\mathit{\xi },\mathit{X}|{\xi }_j\right)}{\left|\Omega \right|}\end{array}$$

where $\mathrm{cov}\left(\mathit{\xi },\mathit{X}|{\xi }_j\right)$is a $m\times p$ covariance matrix given ${\xi }_j$, of which the $\left(k,i\right)$ elements are $\mathrm{cov}\left({\xi }_k,X_i|{\xi }_j\right)$. The standardized total effect$e_T\left({\xi }_j\to \mathit{X}\right)$ is given by

$$e_T\left({\xi }_j\to \mathit{X}\right)=\frac{\mathrm{KL}\left(\mathit{X}, \mathit{\xi }\right)-\mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{/j}|{\xi }_j\right)}{\mathrm{KL}\left(\mathit{X}, \mathit{\xi }\right)+1}=\frac{\mathrm{tr}\tilde{\Omega }\Lambda \left(\mathrm{cov}\left(\mathit{\xi },\mathit{X}\right)-\mathrm{cov}\left(\mathit{\xi },\mathit{X}|{\xi }_j\right)\right)}{ \mathrm{tr}\tilde{\Omega }\Lambda \mathrm{cov}\left(\mathit{\xi },\mathit{X}\right)+\left|\Omega \right|}$$

The standardized total effect$e_T\left({\xi }_j\to \mathit{X}\right)$ [8] is interpreted as the contribution ratio of factor${\xi }_j$in the whole entropy space of$\mathit{X}$, and in the present paper, it is denoted by $\widetilde{RC}\left({\xi }_j\to \mathit{X}\right)$. The contribution of factor${\xi }_j$ measured in entropy is defined by

$$\mathrm{C}\left({\xi }_j\to \mathit{X}\right)=\mathrm{KL}\left(\mathit{X}, \mathit{\xi }\right)-\mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{/j}|{\xi }_j\right)=\frac{\mathrm{tr}\tilde{\Omega }\Lambda \mathrm{cov}\left(\mathit{\xi },\mathit{X}\right)}{\left|\Omega \right|}-\frac{\mathrm{tr}\tilde{\Omega }\Lambda \mathrm{cov}\left(\mathit{\xi },\mathit{X}|{\xi }_j\right)}{\left|\Omega \right|}$$

As for (6), the relative contribution of factor${\xi }_j$ on $\mathit{X}$ is given by

$$\mathrm{RC}\left({\xi }_j\to \mathit{X}\right)=\frac{\widetilde{\mathrm{RC}}\left({\xi }_j\to \mathit{X}\right)}{\widetilde{\mathrm{RC}}\left(\mathit{\xi }\to \mathit{X}\right)}=\frac{\mathrm{C}\left({\xi }_j\to \mathit{X}\right)}{\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)}$$

Concerning factor contributions of ${\xi }_j$ on the single manifest variables $X_i$, that is, $\mathrm{C}\left({\xi }_j\to X_i\right)$, the following theorem can be stated.

Theorem 2.

In the factor analysis model (1),

$$\mathrm{C}\left({\xi }_j\to \mathit{X}\right)={\sum }_{i=1}^p\mathrm{C}\left({\xi }_j\to X_i\right)$$

Proof: 

From Theorem 1, it follows that

$$\mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{/j}|{\xi }_j\right)={\sum }_{i=1}^p\mathrm{KL}\left(X_i,{\mathit{\xi }}^{/j}|{\xi }_j\right)$$

Then, we have

$$\mathrm{C}\left({\xi }_j\to X_i\right)=\mathrm{KL}\left(X_i,\mathit{\xi }\right)-\mathrm{KL}\left(X_i,{\mathit{\xi }}^{/j}|{\xi }_j\right)$$

and,

$$\begin{array}{l}\mathrm{C}\left({\xi }_j\to \mathit{X}\right)=\mathrm{KL}\left(\mathit{X}, \mathit{\xi }\right)-\mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{/j}|{\xi }_j\right)\\ ={\sum }_{i=1}^p\mathrm{KL}\left(X_i,\mathit{\xi }\right)-{\sum }_{i=1}^p\mathrm{KL}\left(X_i,{\mathit{\xi }}^{/j}|{\xi }_j\right)\\ ={\sum }_{i=1}^p\left(KL\left(X_i,\mathit{\xi }\right)-\mathrm{KL}\left(X_i,{\mathit{\xi }}^{/j}|{\xi }_j\right)\right)={\sum }_{i=1}^p\mathrm{C}\left({\xi }_j\to X_i\right)\end{array}$$

From the above theorem, we have the following corollary.

Corollary 2.

Let${\mathit{X}}_{\left(1\right)}={\left(X_{i_1},X_{i_2},\dots ,X_{i_q}\right)}^T$and${\mathit{X}}_{\left(2\right)}={\left(X_{j_1},X_{j_2},\dots ,X_{j_{p-q}}\right)}^T$bedecomposition of manifest variable vector$\mathit{X}={\left(X_1,X_2,\dots ,X_p\right)}^T$, where$q<p$.

$$\begin{gathered} \mathrm{C}\left({\xi }_j\to \mathit{X}\right)=\mathrm{C}\left({\xi }_j\to {\mathit{X}}_{\left(1\right)}\right)+\mathrm{C}\left({\xi }_j\to {\mathit{X}}_{\left(2\right)}\right) \\ \mathrm{C}\left({\xi }_j\to {\mathit{X}}_{\left(1\right)}\right)={{\displaystyle \sum }}_{k=1}^q\mathrm{C}\left({\xi }_j\to X_{i_k}\right), \mathrm{C}\left({\xi }_j\to {\mathit{X}}_{\left(2\right)}\right)={{\displaystyle \sum }}_{k=1}^{p-q}\mathrm{C}\left({\xi }_j\to X_{j_k}\right) \end{gathered}$$

Proof: 

From a similar discussion in the proof of Theorem 2, the corollary follows. ☐

Remark 4.

Let ${\mathit{X}}_{sub}$ be any sub-vector of manifest variable vector $\mathit{X}={\left(X_1,X_2,\dots ,X_p\right)}^T$. By substituting $\mathit{X}$ for ${\mathit{X}}_{sub}$ in the above discussion, $\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{sub}\right)$, $\mathrm{C}\left({\xi }_j\to {\mathit{X}}_{sub}\right)$, $\widetilde{\mathrm{RC}}\left({\xi }_j\to {\mathit{X}}_{sub}\right)$, and $\mathrm{RC}\left({\xi }_j\to {\mathit{X}}_{sub}\right)$ can be defined.

For orthogonal factor analysis models, the following theorem holds true.

Theorem 3.

In factor analysis model (1), if common factors ${\xi }_j$ are statistically independent, then

$$\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)={\sum }_{j=1}^m{\sum }_{i=1}^p\mathrm{C}\left({\xi }_j\to X_i\right).$$

Proof: 

From model (1), we have

$$\mathrm{C}\left({\xi }_j\to X_i\right)=\mathrm{KL}\left(X_i,\mathit{\xi }\right)-\mathrm{KL}\left(X_i,{\mathit{\xi }}^{/j}|{\xi }_j\right)=\frac{{\lambda }_{ij}^2}{{\sigma }_i^2}$$

This completes the theorem. ☐

From the above discussion, if common factors ${\xi }_j$ are statistically independent, (10) is derived. Moreover, we have

$$\widetilde{\mathrm{RC}}\left({\xi }_j\to \mathit{X}\right)=\frac{\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)-\mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{/j}|{\xi }_j\right)}{\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)+1}=\frac{{{\displaystyle \sum }}_{i=1}^p\frac{{\lambda }_{ij}^2}{{\sigma }_j^2}}{\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)+1}$$

This measure is the relative contribution ratio of ${\xi }_j$for the variation of $\mathit{X}$ in entropy. The relative contributions of ${\xi }_j$ on $\mathit{X}$ in entropy are calculated as follows:

$$\mathrm{RC}\left({\xi }_j\to \mathit{X}\right)=\frac{\mathrm{C}\left({\xi }_j\to \mathit{X}\right)}{\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)}=\frac{{{\displaystyle \sum }}_{i=1}^p\frac{{\lambda }_{ij}^2}{{\sigma }_i^2}}{{{\displaystyle \sum }}_{j=1}^m{{\displaystyle \sum }}_{i=1}^p\frac{{\lambda }_{ij}^2}{{\sigma }_i^2}}$$

Remark 5.

It is difficult to use OCD for assessing factor contributions, because $\left|\Sigma \right|$ cannot be decomposed as in the above discussion.

5. Numerical Example

In order to illustrate the present method, we use the data shown in

Table 1. Data for illustrating factor analysis.

Subject	$\mathbf{Japanese} {\mathit{X}}_1$	$\mathbf{English} {\mathit{X}}_2$	$\mathbf{Social}{\mathit{X}}_3$	$\mathbf{Mathematics} {\mathit{X}}_4$	$\mathbf{Science}{\mathit{X}}_5$
1	64	65	83	69	70
2	54	56	53	40	32
3	80	68	75	74	84
4	71	65	40	41	68
5	63	61	60	56	80
6	47	62	33	57	87
7	42	53	50	38	23
8	54	17	46	58	58
9	57	48	59	26	17
10	54	72	58	55	30
11	67	82	52	50	44
12	71	82	54	67	28
13	53	67	74	75	53
14	90	96	63	87	100
15	71	69	74	76	42
16	61	100	92	53	58
17	61	69	48	63	71
18	87	84	64	65	53
19	77	75	78	37	44
20	57	27	41	54	30

[12]. In this table, manifest variables $X_1,X_2, \mathrm{and} X_3$ are subjects in liberal arts and variables $X_4 \mathrm{and} X_5$ are those in sciences. First, orthogonal factor analysis (varimax method by S-PLUS ver. 8.2) is applied to the data and the results are illustrated in

Table 2. Factor loadings of orthogonal factor analysis (${\chi }^2=0.55, df=1, P=0.45).$

	${\mathit{X}}_{\mathbf{1}}$	${\mathit{X}}_{\mathbf{2}}$	${\mathit{X}}_{\mathbf{3}}$	${\mathit{X}}_{\mathbf{4}}$	${\mathit{X}}_{\mathbf{5}}$
${\xi }_1$	0.60	0.75	0.65	0.32	0.00
${\xi }_2$	0.39	0.24	0.00	0.59	0.92
uniqueness	0.50	0.38	0.58	0.55	0.16

. From the estimated factor loadings, the first factor is interpreted as an ability relating to liberal arts, and the second factor as that for sciences. According to the factor contributions $\mathrm{C}\left({\xi }_j\to \mathit{X}\right)$ shown in

Table 3. Factor contributions based on entropy (orthogonal case).

	${\mathit{\xi }}_1$	${\mathit{\xi }}_2$	Total
$\mathrm{C}\left({\xi }_j\to \mathit{X}\right)$	3.11	6.23	$9.34=C\left(\mathit{\xi }\to \mathit{X}\right)$
$\widetilde{\mathrm{RC}}\left({\xi }_j\to \mathit{X}\right)$	0.30	0.60	$0.90=\widetilde{RC}\left(\mathit{\xi }\to \mathit{X}\right)$
$\mathrm{RC}\left({\xi }_j\to \mathit{X}\right)$	0.33	0.67	$1$

, the contribution of factor ${\xi }_2$ is about twice as big than that of factor ${\xi }_1$ from a view point of entropy, and from the relative contributions $\widetilde{\mathrm{RC}}\left({\xi }_j\to \mathit{X}\right)$, about 30% of variation of manifest variable vector $\mathit{X}$ in entropy is explained by factor ${\xi }_1$ and about 60% by factor ${\xi }_2$. The relative contribution $\widetilde{\mathrm{RC}}\left(\mathit{\xi }\to \mathit{X}\right)$ in Table 3 implies about 90% of the entropy of manifest variable vector $\mathit{X}$ is explained by the two factors. On the other hand, in the conventional method, the measured factor contributions of ${\xi }_1$ and ${\xi }_2$, that is, $C_j$, are almost equal (

Table 4. Factor contributions with the conventional method.

	${\mathit{\xi }}_1$	${\mathit{\xi }}_2$	Total
$C_j$	1.44	1.39	2.83
${\widetilde{\mathrm{RC}}}_j$	0.29	0.28	0.57
${\mathrm{RC}}_j$	0.51	0.49	1

). As discussed in the present paper, the conventional method is intuitive and does not have any logical foundation for multidimensionally measuring contributions of factors to manifest variable vectors.

Table 5. Decomposition of factor contribution $\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)$ into $\mathrm{C}\left({\xi }_j\to X_i\right).$

	${\mathit{X}}_1$	${\mathit{X}}_2$	${\mathit{X}}_3$	${\mathit{X}}_4$	${\mathit{X}}_5$	$\mathbf{Total}=\mathbf{C}\left({\mathit{\xi }}_{\mathit{j}}\to \mathit{X}\right)$
${\xi }_1$	0.72	1.49	0.72	0.19	0.00	3.11
${\xi }_2$	0.30	0.15	0	0.63	5.14	6.23
total$=\mathrm{C}\left(\mathit{\xi }\to X_i\right)$	1.01	1.64	0.72	0.82	5.14	9.34

decomposes “the contribution of $\mathit{\xi }$ to $\mathit{X}$” into components $C\left({\xi }_j\to X_i\right)$. The contribution of ${\xi }_2$ to $X_5$ is prominent compared with the other contributions.

From the discussion in the previous section, the contributions of factors are flexibly calculated. For example, it is reasonable to divide the manifest variable vector into ${\mathit{X}}_{\left(1\right)}=\left(X_1,X_2,X_3\right)$ and ${\mathit{X}}_{\left(2\right)}=\left(X_4,X_5\right)$, because the first sub-vector is related to the liberal arts and the second one to the sciences. First, the contributions of ${\xi }_1$ and ${\xi }_2$ to ${\mathit{X}}_{\left(1\right)}$ are calculated according to the present method, and the details are given as follows:

$$\begin{gathered} \mathrm{C}\left({\xi }_1\to {\mathit{X}}_{\left(1\right)}\right)=\mathrm{C}\left({\xi }_1\to X_1\right)+\mathrm{C}\left({\xi }_1\to X_2\right)+\mathrm{C}\left({\xi }_1\to X_3\right)=0.72+1.49+0.72=2.93 \\ \mathrm{C}\left({\xi }_2\to {\mathit{X}}_{\left(1\right)}\right)=0.30+0.15+0.00=0.45 \end{gathered}$$

$$\begin{gathered} \mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(1\right)}\right)=\mathrm{C}\left({\xi }_1\to {\mathit{X}}_{\left(1\right)}\right)+ \mathrm{C}\left({\xi }_2\to {\mathit{X}}_{\left(1\right)}\right)=2.93+0.45=3.38 \\ \widetilde{\mathrm{RC}}\left(\mathit{\xi }\to {\mathit{X}}_{\left(1\right)}\right)=\frac{\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(1\right)}\right)}{\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(1\right)}\right)+1}=\frac{3.38}{3.38+1}=0.77 \end{gathered}$$

(14)

$$\widetilde{\mathrm{RC}}\left({\xi }_1\to {\mathit{X}}_{\left(1\right)}\right)=\frac{\mathrm{C}\left({\xi }_1\to {\mathit{X}}_{\left(1\right)}\right)}{\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(1\right)}\right)+1}=\frac{2.93}{3.38+1}=0.67$$

(15)

$$\widetilde{\mathrm{RC}}\left({\xi }_2\to {\mathit{X}}_{\left(1\right)}\right)=\frac{0.45}{3.38+1}=0.10$$

(16)

$$\mathrm{RC}\left({\xi }_1\to {\mathit{X}}_{\left(1\right)}\right)=\frac{\mathrm{C}\left({\xi }_1\to {\mathit{X}}_{\left(1\right)}\right)}{\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(1\right)}\right)}=\frac{2.93}{2.93+0.45}=0.87$$

(17)

$$\mathrm{RC}\left({\xi }_2\to {\mathit{X}}_{\left(1\right)}\right)=\frac{0.45}{2.93+0.45}=0.13$$

(18)

From (14), 77% of the entropy of manifest variable sub-vector ${\mathit{X}}_{\left(1\right)}$ are explained by the two factors, in which 67% of that are explained by factor ${\xi }_1$ (15) and 10% by factor ${\xi }_2 \left(16\right)$. From the relative contributions (17) and (18), 87% of the total contribution of the two factors are made by factor ${\xi }_1$ and 13% by factor ${\xi }_2$.

On the other hand, the contributions of ${\xi }_1$ and ${\xi }_2$ on ${\mathit{X}}_{\left(2\right)}=\left(X_4,X_5\right)$ are calculated as follows:

$$\begin{gathered} \mathrm{C}\left({\xi }_1\to {\mathit{X}}_{\left(2\right)}\right)=\mathrm{C}\left({\xi }_2\to X_4\right)+\mathrm{C}\left({\xi }_2\to X_5\right)=0.19+0.00=0.19 \\ \mathrm{C}\left({\xi }_2\to {\mathit{X}}_{\left(2\right)}\right)=0.63+5.14=5.77 \end{gathered}$$

$$\begin{gathered} \mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(2\right)}\right)=\mathrm{C}\left({\xi }_1\to {\mathit{X}}_{\left(2\right)}\right)+ \mathrm{C}\left({\xi }_2\to {\mathit{X}}_{\left(2\right)}\right)=0.19+5.77=5.96 \\ \widetilde{\mathrm{RC}}\left(\mathit{\xi }\to {\mathit{X}}_{\left(2\right)}\right)=\frac{\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(2\right)}\right)}{\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(2\right)}\right)+1}=\frac{5.96}{5.96+1}=0.86 \end{gathered}$$

(19)

$$\widetilde{\mathrm{RC}}\left({\xi }_1\to {\mathit{X}}_{\left(2\right)}\right)=\frac{\mathrm{C}\left({\xi }_1\to {\mathit{X}}_{\left(2\right)}\right)}{\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(2\right)}\right)+1}=\frac{0.19}{5.96+1}=0.03$$

(20)

$$\widetilde{\mathrm{RC}}\left({\xi }_2\to {\mathit{X}}_{\left(2\right)}\right)=\frac{5.77}{5.96+1}=0.83$$

(21)

$$\mathrm{RC}\left({\xi }_1\to {\mathit{X}}_{\left(2\right)}\right)=\frac{\mathrm{C}\left({\xi }_1\to {\mathit{X}}_{\left(2\right)}\right)}{\mathrm{C}\left(\mathit{\xi }\to {\mathit{X}}_{\left(2\right)}\right)}=\frac{0.19}{5.96}=0.03$$

(22)

$$\mathrm{RC}\left({\xi }_2\to {\mathit{X}}_{\left(2\right)}\right)=\frac{5.77}{5.96}=0.97$$

(23)

From (19), 86% of entropy of manifest variable sub-vector ${\mathit{X}}_{\left(2\right)}$ is explained by the two factors, in which 3% of the entropy are explained by factor ${\xi }_1$ (20) and 83% by factor ${\xi }_2$ (21). The contribution ratios of the factors to sub-vector ${\mathit{X}}_{\left(2\right)}$ are calculated in (22) and (23). Ninety-seven percent of the entropy was made by factor ${\xi }_2$.

Second, factor contributions in an oblique case are calculated. The estimated factor loadings and the correlation matrix of factors based on the covarimin method are shown in

Table 6. Factor loadings of oblique factor analysis (${\chi }^2=0.55, df=1, P=0.45).$

	${\mathit{X}}_1$	${\mathit{X}}_2$	${\mathit{X}}_3$	${\mathit{X}}_4$	${\mathit{X}}_5$
${\xi }_1$	0.59	0.77	0.68	0.29	0
${\xi }_2$	0.24	0.00	−0.12	0.52	0.92
uniqueness	0.50	0.41	0.58	0.55	0.16

and

Table 7. Correlation matrix of factors.

	${\mathit{\xi }}_1$	${\mathit{\xi }}_2$
${\xi }_1$	1	0.315
${\xi }_2$	0.315	1

, respectively. Based on factor loadings in Table 6, factor ${\xi }_1$ is interpreted as an ability for subjects in the liberal arts and factor ${\xi }_2$ as an ability for subjects in sciences. The results are similar to those in the orthogonal case mentioned above, because the correlation between the factors is not strong.

Table 8. Decomposition of factor contribution $\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)$ into $\mathrm{C}\left({\xi }_j\to X_i\right)$ (oblique case).

	${\mathit{X}}_1$	${\mathit{X}}_2$	${\mathit{X}}_3$	${\mathit{X}}_4$	${\mathit{X}}_5$	$\mathbf{Total}=\mathbf{C}\left({\mathit{\xi }}_{\mathit{j}}\to \mathit{X}\right)$
${\xi }_1$	0.90	1.44	0.70	0.37	0.54	3.95
${\xi }_2$	0.37	0.14	0.01	0.68	5.43	6.65
$\mathrm{C}\left(\mathit{\xi }\to X_i\right)$	1.01	1.44	0.73	0.82	5.43	$\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)=9.43$

shows the decomposition of $\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)$ based on Theorems 1 and 2. In this case, it is noted that $\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)\ne \mathrm{C}\left({\xi }_1\to \mathit{X}\right)+\mathrm{C}\left({\xi }_2\to \mathit{X}\right)$; however, $\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)={\sum }_{i=1}^5\mathrm{C}\left(\mathit{\xi }\to X_i\right)$. According to the table, the contributions of ${\xi }_1$ and ${\xi }_2$ to sub-vectors of manifest variable vector $\mathit{X}$ can also be calculated as in the above orthogonal factor analysis.

Table 9. Factor contributions based on entropy (oblique case).

	${\mathit{\xi }}_1$	${\mathit{\xi }}_2$	Effect of $\mathit{\xi }$ on $\mathit{X}$
$\mathrm{C}\left({\xi }_j\to \mathit{X}\right)$	3.95	6.65	$\mathrm{C}\left(\mathit{\xi }\to \mathit{X}\right)=9.43$
$\widetilde{\mathrm{RC}}\left({\xi }_j\to \mathit{X}\right)$	0.38	0.64	$\widetilde{\mathrm{RC}}\left(\mathit{\xi }\to \mathit{X}\right)=0.90$
$\mathrm{RC}\left({\xi }_j\to \mathit{X}\right)$	0.42	0.71

illustrates the contributions of factors on manifest variable vector$\mathit{X}$. Factor ${\xi }_1$ explains 42% of the entropy of $\mathit{X}$ and factor ${\xi }_2$ explains 71%.

6. Discussion

For orthogonal factor analysis models, the conventional method measures factor contributions (effects) by the sums (totals) of squared factor loadings related to the factors (2); however, there is no logical foundation for how they can be interpreted. It is reasonable to measure factor contributions as the effects of factors on the manifest variable vector concerned. The present paper has proposed a method of measuring factor contributions through entropy, that is, applying an entropy-based path analysis approach. The method measures the contribution of factor vector $\mathit{\xi }$ to manifest variable vector $\mathit{X}$ and decomposes it into those of factors ${\xi }_j$ to manifest variables $X_i$ and/or those to sub-vectors of $\mathit{X}$. Comparing (2) and (10), for standardization of unique factor variances ${\sigma }_i^2=1$, the present method equals to the conventional method. As discussed in this paper, the present method can be employed in oblique factor analysis as well, and it has been illustrated in a numerical example. The present method has a theoretical basis for measuring factor contributions in a framework of entropy, and it is a novel approach for factor analysis. The present paper confines itself to the usual factor analysis model. A more complicated model with a mixture of normal factor analysis models [13] is excluded, and a further study is needed to apply the entropy-based method to the model.