Elsevier

Automatica

Volume 40, Issue 4, April 2004, Pages 575-589
Automatica

On the ill-conditioning of subspace identification with inputs

https://doi.org/10.1016/j.automatica.2003.11.009Get rights and content

Abstract

There is experimental evidence that the performance of standard subspace algorithms from the literature (e.g. the N4SID method) may be surprisingly poor in certain experimental conditions. This happens typically when the past signals (past inputs and outputs) and future input spaces are nearly parallel. In this paper we argue that the poor behavior may be attributed to a form of ill-conditioning of the underlying multiple regression problem, which may occur for nearly parallel regressors. An elementary error analysis of the subspace identification problem, shows that there are two main possible causes of ill-conditioning. The first has to do with near collinearity of the state and future input subspaces. The second has to do with the dynamical structure of the input signal and may roughly be attributed to “lack of excitation”. Stochastic realization theory constitutes a natural setting for analyzing subspace identification methods. In this setting, we undertake a comparative study of three widely used subspace methods (N4SID, Robust N4SID and PO-MOESP). The last two methods are proven to be essentially equivalent and the relative accuracy, regarding the estimation of the (A,C) parameters, is shown to be the same.

Introduction

Subspace methods for the identification of linear systems have been object of much research in the last 10 years. In particular subspace methods for time series (no observable inputs) have been thoroughly analyzed in the literature (Van Overschee & De Moor, 1993; Bauer, 2002; Lindquist & Picci, 1996) and it seems fair to say that they are perhaps the most efficient and accurate methods available today for multivariable time series identification. The situation is not the same for identification of systems with observable inputs. Although it is generally admitted that subspace methods with inputs offer substantial advantages over traditional PEM identification (Ljung, 1997), especially for identification of multivariable systems, there is experimental evidence that standard subspace methods (e.g., the N4SID method) perform poorly in certain experimental conditions, in particular when the past signals (past inputs and outputs) and future input spaces are nearly parallel (Chiuso & Picci, 1999; Kawauchi, Chiuso, Katayama, & Picci, 1999). Subspace methods operating on joint input–output data involve the solution of a multiple regression problem and the reason for the poor behavior may be attributed to ill-conditioning of the regression problem, which may occur when the regressors are nearly parallel. Although this phenomenon is well-known in multivariate statistics (Stewart, 1987; Belsley, 1991), it does not seem to have been noticed and analyzed in the subspace identification literature. The study of numerical conditioning of the regression problem, besides the effect of numerical roundoff errors, which are clearly irrelevant in the present context, yields information on the sensitivity of the parameter and transfer function estimates to noise in the data, which is instead important to assess the performance of identification methods. It should be intuitively clear, and it will be demonstrated formally in the companion paper (Chiuso & Picci, 2004b), that bad numerical conditioning generally implies a large variance of the estimates. We believe that this aspect should be taken into account in the design and comparison of subspace algorithms.

In this paper, which expands on work presented in previous conference papers (Chiuso & Picci, 1999; Kawauchi et al., 1999), we shall provide an elementary error analysis of some well-known subspace methods. We shall see that the sensitivity to random errors of a subspace method can be measured by the condition number of the conditional cross-covariance matrices Σx̂x̂|u+, and Σu+u+|x̂ of the state, given the future inputs, and of the future inputs given the current state, which will be introduced later in the paper. We shall relate the conditioning of these two matrices to the “near parallelism” of the state and future input spaces and thereby see that there are two main possible causes of ill-conditioning. The first is due to the cross-correlation between the state and future inputs, while the second is inherent in the dynamical structure of the input signal (and there is not much one can do about it, if identification has to be based on experiments performed during normal operation of the plant). Here we make contact and extend the analysis of the papers (Jansson and Wahlberg 1997, Jansson and Wahlberg 1998); where it has been shown that the singularity of Σxx|u+ is a cause of lack of consistency of subspace methods with inputs. In a sense, we study also the effects of near-singularity of this matrix.

A main motivation of this paper is to compare the performance of some widely used subspace methods from the literature. To this purpose, we undertake a comparative analysis of the conditioning of N4SID, “Robust” N4SID and PO-MOESP. The analysis is first directed to recasting the various algorithms into a common setting using ideas from stochastic realization theory. A result of this analysis is that, at least for the estimation of the (A,C) parameters, the “Robust” N4SID and PO-MOESP methods are equivalent. As expected, the original N4SID of (Van Overschee & De Moor, 1994), is generally worse.

The structure of the paper is as follows:

  • In Section 2 we review the basic ideas of subspace identification, discuss the finite-interval stochastic realization problem, describe the basic “ideal” Kalman filter model which should be used in identification with inputs and discuss some difficulties which prevent constructing the state from finite input–output data. This explains why there is a multitude of subspace identification methods with inputs and indicates a common background for their analysis.

  • In Section 3 we do some error analysis, comparing the conditioning of the identification (regression) problem based on the “ideal model”, with the regression problem occurring in the N4SID method.

  • In Section 4 we compare the “Robust N4SID” and PO-MOESP methods. We prove that these two methods produce exactly the same estimates of (A,C) and hence the same conditioning analysis holds for these two methods.

  • Section 5 contains some conclusions.

As is well-known, numerical conditioning analysis deals in a sense with “worst case” situations and one may wonder what is the practical relevance of the results of this paper in terms of statistical accuracy (e.g., variance) of the estimates. This point is answered in the companion paper (Chiuso & Picci, 2004b), where we introduce asymptotic variance formulas for the (A,C) and (B,D) parameter estimates. From these formulas, the statistical meaning of the analysis of this paper emerges very clearly.

Section snippets

A review of subspace identification

Let{ut0,…,ut,…},{yt0,…,yt,…},utRp,ytRmbe observed input–output trajectories of an unknown system, which we want to identify. For the moment we shall pretend that the trajectories are infinitely long. We shall assume that the data are sample paths of a pair of zero-mean second-order stationary true random processes y={y(t)}, u={u(t)} having a rational spectral density; in other words, data (2.1) are generated by a linear stochastic system of the formx(t+1)=Ax(t)+Bu(t)+Gw(t),y(t)=Cx(t)+Du(t)+Jw

Error analysis of subspace methods

Ideally, the first step of subspace identification should be of constructing the state of a transient Kalman-filter type realization (2.16). Naturally, with only finite data available, the state vector must be approximated by a finite tail matrix and will be affected by random errors. Hence the parameter estimates, obtained by regressing the next state and output variables on the estimated state (and on the observed input) will also be affected by errors. In this section we shall show that the

Conditioning of the robust N4SID and of MOESP-type methods

In this section we shall see that the computation of the asymptotic estimates of (A,C) in the “robust” N4SID method of Van Overschee and De Moor (1996) and in the so-called PO-MOESP method of Verhaegen, 1994, is described, except for a change of basis in the state space, by the same formulas found for N4SID. Therefore the same conditioning analysis which applies to N4SID (in particular Proposition 7) also applies to these two methods.

In the process of doing this, we shall actually establish

Conclusions

In this paper we have presented an error analysis which applies to some commonly used subspace identification methods with inputs. We have shown that in presence of collinearity of the regressors these methods may lead to inaccurate estimates of the system parameters (and of the relative transfer function). We have also demonstrated that some of the most well-known algorithms in the literature (in particular robust N4SID and PO-MOESP) are equivalent as far as estimation of the matrices (A,C) is

Alessandro Chiuso received his D.Ing. degree Cum Laude in 1996, and the Ph.D. degree in Systems Engineering in 2000 both from the University of Padova. In 1998/99 he was a Visiting Research Scholar with the Electronic Signal and Systems Research Laboratory (ESSRL) at Washington University, St. Louis. From March 2000 to July 2000 he has been Visiting Post-Doctoral (EU-TMR) fellow with the Division of Optimization and System Theory, Department of Mathematics, KTH, Stockholm, Sweden. Since March

References (34)

  • D. Bauer et al.

    Some facts about the choice of the weighting matrices in larimore type of subspace algorithm

    Automatica

    (2001)
  • Bauer, D., Deistler, M., & Scherrer, W. (2000). On the impact of weighting matrices in subspace algorithms. In...
  • D.A. Belsley

    Conditioning diagnostics, collinearity and weak data regression

    (1991)
  • Caines, P. E., & Chan, C. W. (1976). Estimation, identification and feedback. In: R. Mehra, D. Lainiotis (Eds.), System...
  • Chiuso, A. (2000). Geometric methods for subspace identification. Ph.D. thesis. Dept. of Electronics and informatics,...
  • Chiuso, A., & Picci, G. (1999). Subspace identification by orthogonal decomposition. In Proceedings of the 14th IFAC...
  • Chiuso, A., & Picci, G. (2000). Error analysis of certain subspace methods. In Proceedings of IFAC international...
  • Cited by (56)

    • An iterative state-space identification method with data correlation for MIMO systems with measurement noise

      2022, Journal of the Franklin Institute
      Citation Excerpt :

      Subspace methods made relevant contributions starting from the work of Van Overschee and De Moor [29,30], with a highlight to the N4SID as the principal subspace identification method. However, some drawbacks of subspace methods were appointed as in the work of Chiuso et al. [31], mentioning collinearity of the regressors that may lead to inaccurate estimates of the system parameters. Our main contribution here is the use of feedbacking the Markov parameters in the identification algorithm to minimize the noise effects, which is distinct from the works [16,17] where the closed-loop is in the system plant.

    • A novel subspace identification approach with passivity enforcement

      2021, Automatica
      Citation Excerpt :

      Despite its many strengths, several drawbacks have been recognized. In general, estimates from 4SIDs are not as accurate as those from prediction error methods, and under closed-loop conditions, some traditional 4SID algorithms are biased thus requiring further methods to cope with this problem (Chiuso & Picci, 2004a, 2004b). Subspace-based methods were originally proposed in the late 90s by Overschee and Moor (1991), then three different algorithms emerged: CVA (Larimore, 1996), MOESP (Verhaegen & Dewilde, 1992) and N4SID (Overschee & Moor, 1994), which were later unified in Overschee and Moor (1995).

    View all citing articles on Scopus

    Alessandro Chiuso received his D.Ing. degree Cum Laude in 1996, and the Ph.D. degree in Systems Engineering in 2000 both from the University of Padova. In 1998/99 he was a Visiting Research Scholar with the Electronic Signal and Systems Research Laboratory (ESSRL) at Washington University, St. Louis. From March 2000 to July 2000 he has been Visiting Post-Doctoral (EU-TMR) fellow with the Division of Optimization and System Theory, Department of Mathematics, KTH, Stockholm, Sweden. Since March 2001 he is Research Faculty (“Ricercatore”) with the Dept. of Information Engineering, University of Padova. In the summer 2001 he has been visiting researcher with the Dept. of Computer Science, University of California Los Angeles. His research interests are mainly in Identification and Estimation Theory, System Theory and Computer Vision.

    Giorgio Picci holds a full professorship with the University of Padova, Italy, Department of Information Engineering, since 1980. He graduated (cum laude) from the University of Padova in 1967 and since then has held several long-term visiting appointments with various American and European universities among which Brown University, M.I.T., the University of Kentucky, Arizona State University, the Center for Mathematics and Computer Sciences (C.W.I.) in Amsterdam, the Royal Institute of Technology, Stockholm Sweden, Kyoto University and Washington University in St. Louis, Mo.

    He has been contributing to Systems and Control theory mostly in the area of modeling, estimation and identification of stochastic systems and published over 100 papers and edited three books in this area. Since 1992 he has been active also in the field of Dynamic Vision and scene and motion reconstruction from monocular vision.

    He has been involved in various joint research projects with industry and state agencies. He is currently general coordinator of the italian national project New techniques for identification and adaptive control of industrial systems, funded by MIUR (the Italian ministery for higher education), has been project manager of the italian team for the Commission of the European Communities Network of Excellence System Identification (ERNSI) and is currently general project manager of the Commission of European Communities IST project RECSYS, in the fifth Framework Program.

    Giorgio Picci is a Fellow of the IEEE, past chairman of the IFAC Technical Committee on Stochastic Systems and a member of the EUCA council.

    This work has been supported in part by the ERNSI TMR network System Identification and by the national project Identification and adaptive control of industrial systems funded by MIUR. Part of this work has been done while the first author (A.C.) was a post-doctoral researcher with the Division of Optimization and Systems Theory, KTH, Stockholm, supported by ERNSI. This paper was recommended for publication in revised form by Associate Editor Brett Niness under the direction of Editor Torsten Söderström.

    View full text