On the equivalence of time and frequency domain maximum likelihood estimation☆
Introduction
Maximum Likelihood (ML) estimation methods have become a popular approach to dynamic system identification (Goodwin and Payne, 1977, Ljung, 1999, Söderström and Stoica, 1989). Different approaches have been proposed in the time and frequency domains (Ljung, 1993, Ljung, 2006, McKelvey, 2002, Pintelon and Schoukens, 2001, Pintelon and Schoukens, 2006). A commonly occurring question is how time and frequency domain versions of ML estimation are related. Some insights into the relationship between the methods have been given in the previous literature. However, to the best of the authors’ knowledge, there has not previously been a comprehensive account of the equivalence between the two approaches, in particular, for finite length data. For example, Ljung (1993), McKelvey (2002), McKelvey and Ljung (1997) have shown that, when working in the frequency domain, an extra term arises in the likelihood function that depends on the noise model. This term vanishes asymptotically for long data sets when considering uniformly spaced frequency points over the full bandwidth (see, for example, Pintelon & Schoukens, 2006).
In Pintelon and Schoukens (2006), Box–Jenkins identification has been analyzed. Extensions to identification in closed loop have also been presented (Pintelon & Schoukens, 2006). Also the case of reduced bandwidth estimation has been considered. A surprising result, in this context, is that, for processes operating in open loop, the commonly used frequency domain ML method requires exact knowledge of the noise model structure in order to obtain consistent estimates for the plant parameters. On the other hand, it is well known that the commonly used ML in the time domain (for systems driven by a quasi-stationary input and Gaussian white noise that are mutually uncorrelated) provides consistent estimates for the transfer function from input to output irrespective of possible under-modeling of the transfer function from noise to output (Ljung, 1999). This fact suggests that there could be key differences between the time and frequency domain approaches in the usual formats. In the current paper we will see that the apparent differences are a result of inconsistent formulations rather than fundamental issues between the use of time or frequency domain data. In particular, we establish in this paper that the domain chosen to describe the available data (i.e., time or frequency) does not change the result of the estimation problem (see also Ljung, 2006, Schoukens et al., 2004). Instead, it is the choice of the likelihood function, i.e., which parameters are to be estimated and what data is assumed available, that leads to perceived differences in the estimation problems. This issue has previously been highlighted for time domain methods in the statistics literature where, for example, the way in which initial conditions are considered defines different likelihood functions and, thus, different estimation problems (see e.g. Pollock, 1999, chapter 22). More specifically, for dynamic system identification, the time domain likelihood function is different depending on the assumptions made regarding the initial state (), e.g.
- (T1)
is assumed to be zero,
- (T2)
is assumed as a deterministic parameter to be estimated, or
- (T3)
is assumed to be a random vector.
- (F1)
is assumed to be zero (equivalent to assuming periodicity in the state)
- (F2)
is estimated as a deterministic parameter (as in, e.g., Agüero et al., 2007, Pintelon and Schoukens, 2006), or
- (F3)
is considered as a hidden random variable.
Section snippets
Time domain model and data
We consider the following Single-Input Single-Output (SISO) linear system model: where and are the (sampled time domain) input and output signals, respectively, and is zero mean Gaussian noise with variance . and are rational functions in the forward shift operator . We also assume that: (i) and are stable, with no poles on the unit circle; (ii) is stable (i.e., is minimum phase, with no zeros on the unit circle); and (iii)
Frequency domain model and data
In this section we consider the situation where the data has been transformed from time to frequency domain. We review the statistical properties of transformed data when using the DFT. Later, in Section 3.2, we examine the impact that this transformation has on maximum likelihood estimation.
To transfer the estimation problem into the frequency domain we apply the discrete Fourier transform (DFT) to the data (see e.g. Hannan, 1970, chapter V): where ,
General considerations
We first show that the values of the time and frequency domain likelihood estimates are equal when consistent parameter definitions are used.
Theorem 17 Given a dynamic system model and a set of output measurements, the estimates obtained by maximizing the likelihood function are the same irrespective of the representation of the data, either in time or frequency domain. That iswhereis a vector that contains the parameters to be estimated and equality holds with
Numerical examples
In this section we present numerical examples to highlight the estimates obtained using time and frequency domain maximum likelihood. We consider a simple model expressed in state space form: where is a zero mean Gaussian noise sequence with unit variance, is a zero mean Gaussian noise sequence with variance , and is the state vector. The true parameters are , , and . We consider an initial state .
The system was simulated over
Conclusions
This paper has considered the equivalence between maximum likelihood estimation in the time and frequency domains for finite length data. It has been shown that both approaches are equivalent provided consistent assumptions are made regarding the unknown parameters in the problem. Also, it is apparent from the development presented here, that it is unsurprising that the methods lead to different results when inconsistent assumptions are made.
Acknowledgements
We thank the Associate Editor and the reviewers for their helpful comments and suggestions.
Juan Carlos Agüero was born in Osorno, Chile. He obtained the professional title of Ingeniero civil electrónico and a Master of engineering degree from the Universidad Técnica Federico Santa María (Chile) in 2000, and a Ph.D. from The University of Newcastle (Australia) in 2006. He gained industrial experience from 1997 to 1998 in the copper mining industry at El Teniente, Codelco (Chile). He is currently working as a Research Academic in The University of Newcastle (Australia). His research
References (42)
- et al.
Robust identification of process models from plant data
Journal of Process Control
(2008) - et al.
Box–jenkins identification revisited–part I: Theory
Automatica
(2006) - Agüero, J. C., Yuz, J. I., & Goodwin, G. C. (2007). Frequency domain identification of MIMO state space models using...
Matrix mathematics: Theory, facts, and formulas with application to linear systems theory
(2005)Time series: Data analysis and theory
(2001)- et al.
Functions of a complex variables: Theory and technique
(1966) System identification—General aspects and structure
Stochastic processes
(1953)Real discrete fourier transform, IEEE Transactions on Acoustics
Speech and Signal Processing
(1985)- Goodwin, G. C. (1985). Some observations on robust estimation and control. In 7th IFAC symposium on identification and...
Dynamic system identification: Experiment design and data analysis
Multiple time series
Lagged regression with unknown lags
Journal of the Royal Statistical Society. Series B (Methodological)
Linear regression in the frequency domain
International Economic Review
Capacity of the discrete-time Gaussian channel with intersymbol interference
IEEE Transactions on Information Theory
Stochastic processes and filtering theory
Linear systems
System identification: Theory for the user
Frequency domain versus time domain methods in system identification — revisited
Lecture notes in control and information sciences
Cited by (60)
Maximum Likelihood estimation for non-minimum-phase noise transfer function with Gaussian mixture noise distribution
2022, AutomaticaCitation Excerpt :The authors in Hsu and Jay Breidt (2009) showed that, when initial conditions are included as parameters to be estimated, the estimation accuracy is better than when initial conditions are fixed to zero (conditional ML estimation). This issue is due to the fact that the likelihood function depends on how the initial conditions are considered; thus, different ML estimation problems can be defined (see e.g. Agüero, Yuz, Goodwin, & Delgado, 2010; Pollock, 1999). In addition, a Monte Carlo method with importance sampling is used to numerically compute a surrogate function (E-step) of the likelihood function.
Parametric identification of a linear time invariant model for a subglottal system
2021, IFAC-PapersOnLineA framework for high-resolution frequency response measurement and parameter estimation in microscale impedance applications
2019, Measurement: Journal of the International Measurement Confederation
Juan Carlos Agüero was born in Osorno, Chile. He obtained the professional title of Ingeniero civil electrónico and a Master of engineering degree from the Universidad Técnica Federico Santa María (Chile) in 2000, and a Ph.D. from The University of Newcastle (Australia) in 2006. He gained industrial experience from 1997 to 1998 in the copper mining industry at El Teniente, Codelco (Chile). He is currently working as a Research Academic in The University of Newcastle (Australia). His research interest is in System Identification.
Juan I. Yuz was born in Valparaíso, Chile, in 1975. He received his professional title of Ingeniero Civil Electrónico and Master degree in Electronic Engineering from Universidad Técnica Federico Santa María (UTFSM) in 2001, obtaining the Best Electronic Engineering Student Award. He received his Ph.D. in Electrical Engineering from The University of Newcastle, Australia, in 2006. He currently holds a research position with the Automatic Control group at Departamento de Electrónica, UTFSM. His research interests include control and identification of sampled-data systems.
Graham Goodwin obtained B.Sc (Physics), B.E (Electrical Engineering), and Ph.D. from the University of New South Wales. He is currently Professor Laureate of Electrical Engineering at the University of Newcastle, Australia and is Director of the Australian Research Council Centre of Excellence for Complex Dynamic Systems and Control. He holds Honorary Doctorates from Lund Institute of Technology, Sweden and the Technion, Israel. He is the co-author of eight books, four edited books, and many technical papers. Graham is the recipient of Control Systems Society 1999 Hendrik Bode Lecture Prize, a Best Paper award from IEEE Transactions on Automatic Control, a Best Paper award from Asian Journal of Control, and 2 Best Engineering Text Book awards from the International Federation of Automatic Control in 1984 and 2005. In 2008 he received the Quazza Medal from the International Federation of Automatic Control. He is a Fellow of IEEE; an Honorary Fellow of Institute of Engineers, Australia; a Fellow of the International Federation of Automatic Control; a Fellow of the Australian Academy of Science; a Fellow of the Australian Academy of Technology, Science and Engineering; a Member of the International Statistical Institute; a Fellow of the Royal Society, London and a Foreign Member of the Royal Swedish Academy of Sciences.
Ramón A. Delgado was born in San Felipe, Chile, in 1984. He recently received his professional title of Ingeniero Civil Electrónico and Master degree in Electronic Engineering from Universidad Técnica Federico Santa María, Chile, in 2009. His research interests include control and system identification.
- ☆
Work supported by CDSC and FONDECYT-Chile through grant 11070158. The material in this paper was partially presented at 15th IFAC Symposium on System Identification, July 6–8, 2009, Saint Malo, France. This paper was recommended for publication in revised form by Associate Editor Johan Schoukens under the direction of Editor Torsten Söderström.