Information–theoretic approach to blind separation of sources in non-linear mixture
Introduction
One of the important issues in signal processing is to find a set of statistically independent components from the observed data by linear or non-linear transformation. Most of the blind separation algorithms are based on the theory of the independent component analysis (ICA) [8]when the mixture model is linear. The idea of the ICA is to find the independent components in the mixture of the statistically independent source signals by optimizing some criteria. Some blind separation algorithms such as those in Refs. 1, 2, 5, 6, 7have the equivariant property when there is no noise in the observation of the mixture. However, for the non-linear mixture model, the linear ICA theory is not applicable and the equivariant property does not hold. The blind separation algorithms for the linear mixture model generally fail to extract the independent sources in non-linear mixture.
The self-organizing map (SOM) has been used to extract sources in non-linear mixture 9, 11, 12. It is a model-free method but suffers from (a) the exponential growth of the network complexity and (b) interpolation error in recovering continuous sources.
An extension of ICA to the separation sources in non-linear mixture is to employ a non-linear function to transform the mixture such that the outputs become statistically independent. However, without limiting the function class for de-mixing transforms, this extension may give statistically independent outputs which tell nothing about sources because any random variable with a continuous distribution function can be transformed to a uniformly distributed random variable and multiple uniform random variables are always independent. This transformation is not unique. To limit a function class for de-mixing functions is equivalent to assume some knowledge about the mixing function. We employ a two-layer perceptron, a parametric model, as a de-mixing system. A two-layer perceptron was also used in Ref. [4]to separate sources in non-linear mixture by the gradient descent method to minimize the mutual information (measure of dependence) of the outputs. Comparing our approach with the approach in Ref. [4], this paper includes the following innovations:
- 1.
on-line learning algorithms derived by maximizing entropy and minimizing mutual information,
- 2.
the learning algorithms consisting of (a) the learning equations for the hidden-output layer which are the same as those in 2, 3, 6, 15for linear de-mixing system and (b) novel learning equations for input-hidden layer and threshold for the hidden neurons,
- 3.
using the natural gradient descent method to optimize cost functions (entropy or mutual information) without constraints.
A short version of this paper appeared in Ref. [17]. The non-linear mixture model considered there and in this paper contains crossing non-linearities in the mixture. This model is more general than that in Refs. 10, 14where the non-linear mixture is obtained by operating non-linear functions componentwise on the linear mixture.
Section snippets
Mixture models and de-mixing systems
Let us consider unknown source signals , which are mutually independent and stationary. It is assumed that each source has a zero mean and moments of any orders and at most one source is Gaussian.
Before we describe the non-linear mixture model, let us briefly review the linear mixture modelwhere is an unknown non-singular mixing matrix, and . In this paper, the transpose operation is denoted by (·)T. A linear de-mixing system
Information back-propagation approach
We have explained why the minimum mutual information (MMI) approach can be applied for extracting independent sources in the non-linear mixture when the inverse of the non-linear mixing function is approximated by the two-layer perceptron. We can also apply the maximum entropy (ME) approach in Ref. [3]to train the de-mixing system. When the mixture model is linear, the ME approach is justified by its relation with the MMI approach [16]. The two approaches are generally equivalent around the
The information BP vs the error BP
Blind separation of sources is an unsupervized learning problem. We employ a two-layer perceptron as the de-mixing system to extract the sources in the non-linear mixture. However, we cannot use the well-known error BP method to train this multilayer network since the desired signals are not accessible. Instead, we use information criteria such as entropy and mutual information as error signals to derive the information BP algorithms. The block diagram of the entropy BP is plotted in Fig. 2.
Simulation
Assume that the mixing function in Fig. 1 is a two-layer perceptron of 5 input neurons and 5 hidden neurons and 5 output neurons:where the source vectorconsisting of four modulated signals and one random source b(t) uniformly distributed in [−1,1]. and are two 5×5 mixing matrices:
Conclusions
Assuming the inverse of the non-linear mixing function can be approximated by a two-layer perceptron we employ a two-layer perceptron as a de-mixing system to extract sources in the non-linear mixture. The two blind separation algorithms are derived by using two approaches: minimum entropy and maximum mutual information. The two algorithms are called the entropy BP algorithm and the mutual information BP algorithm or information BP algorithms altogether. The information (entropy or mutual) BP
References (17)
Blind separation of sources: a non-linear neural algorithm
Neural Networks
(1992)Independent component analysis, a new concept
Signal Process.
(1994)- S. Amari, A. Cichocki, H.H. Yang, Recurrent neural networks for blind separation of sources, in: Proc. NOLTA 1995, vol....
- S. Amari, A. Cichocki, H.H. Yang, A new learning algorithm for blind signal separation, in: D.S. Touretzky, M.C. Mozer,...
- et al.
An information-maximisation approach to blind separation and blind deconvolution
Neural Comput.
(1995) - J.-F. Cardoso, B. Laheld, Equivariant adaptive source separation, IEEE Trans. Signal Process. 44 (12) (December 1996)...
- A. Cichocki, R. Unbehauen, Robust neural networks with on-line learning for blind identification and blind separation...
- A. Cichocki, R. Unbehauen, L. Moszczyński, E. Rummert, A new on-line adaptive learning algorithm for blind separation...
Cited by (124)
Color calibration and correction applying linear interpolation technique for color fringe projection system
2016, OptikCitation Excerpt :However, their approach requires physically switch of the lenses, which then makes the measurement speed be largely limited. In [22], the IBP (Information Back Propagation) [23] method was proposed to implement color correction. And a decoupling network was built to invert the color leaking process and reduce the color crosstalk between each channel.
The velocity measurement of two-phase flow based on particle swarm optimization algorithm and nonlinear blind source separation
2012, Chinese Journal of Chemical EngineeringNonlinear mixtures
2010, Handbook of Blind Source SeparationModern Signal Processing
2022, Modern Signal ProcessingOverview of industrial process monitoring methods based on independent component analysis and its extended model
2022, Kongzhi yu Juece/Control and DecisionImplementation of Blind Source Separation Based on Improved Variable Step-Size Natural Gradient Algorithm
2022, 2022 7th International Conference on Signal and Image Processing, ICSIP 2022