Elsevier

Signal Processing

Volume 64, Issue 3, 26 February 1998, Pages 371-382
Signal Processing

Multiple multivariate regression and global sequence optimization:: An application to large-scale models of radiation intensity

https://doi.org/10.1016/S0165-1684(97)00202-8Get rights and content

Abstract

We investigate the strengths and weaknesses of several neural network architectures applied to a large-scale thermodynamical application in which sequences of measurements from gas columns must be integrated to construct the columns' spectral radiation intensity profiles. This is a problem of interest for the aeronautical industry. The approaches proposed for its solution can be applied to a wide range of signal problems. Physical models often make use of a number of fitted functions as a simplified parametric base to approximate a high-dimensional nonlinear (and usually computationally intractable) function. Realistically, models of radiation contain thousands of fitted functions. The use of neural networks in applications of this scale are rare, and most effective techniques rely on cross-validation methods or involve other heavy computational overheads that are impracticable when a very large number of models need to be trained. We have employed here two different approaches: multiple multivariate regression, and global sequence minimization. The first approach shows that the integration of several nonlinear regression models into a single neural network may improve both generalization performance and speed of computation. For the former we propose a method of optimization by which we specialize our models globally, on typical sequences of input signals. We show how this does not degrade the overall accuracy but, rather, allows us to specialize our models.

Zusammenfassung

Wir untersuchen die Stärken und Schwächen verschiedener neuronaler Netzwerkarchitekturen für hochdimensionale thermodynamische Anwendungen. Dabei werden Meßreihen von Gassäulen vereint, um deren Strahlungsstärkenprofile zu rekonstruieren. Dieses Problem ist für die Luftfahrt von Interesse. Die vorgeschlagenen Lösungsansätze können auf eine Vielzahl von Problemen aus der Signalverarbeitung übertragen werden. Physikalische Modelle verwenden häufig mehrere angepaßte Funktionen als vereinfachte parametrische Basis, um eine hochdimensionale (und damit gewöhnlich mit dem Rechner schwer zu behandelnde) Funktion zu approximieren. Realistische Modelle für die Strahlung umfassen tausende angepaßter Funktionen. Für solche Anwendungen werden bisher neuronale Netzwerke nur selten verwendet. Die meisten effizienten Verfahren basieren auf `cross-valitation'-Methoden oder bringen in anderer Weise enormen rechentechnischen Aufwand mit sich, so daß sie nicht mehr anwendbar sind, wenn sehr viele Modelle Gebraucht werden. Wir haben hier zwei unterschiedliche Ansätze angewandt: mehrfache multivariate Regression und globale Minimierung der Abfolge. Der erste Ansatz zeigt daß eine Einbettung mehrerer nichtlinearer Regressionsmodelle in ein einziges neuronales Netzwerk sowohl die Generalisierungsfähigkeit als auch die Berechnungsgeschwindigkeit verbessern. Für den vorhergehenden schlagen wir eine Optimierungsmethode vor, bei der wir unsere Modelle global auf typische Abfolgen von Eingangssignalen spezialisieren. Wir zeigen daß dies nicht die Gesamtgenauigkeit verschlechtert sondern vielmehr eine Spezialisierung der Modelle erlaubt.

Résumé

On examine dans cet article les forces et faiblesses de différentes architectures de réseaux de neurones pour une application thermodynamique à grand échelle dans laquelle des séquences de mesures provenant de colonnes de gaz doivent être intégrées pour construire les profils d'intensité spectrale de radiation de ces colonnes. Ce problème est d'un grand intérêt pour l'industrie aéronautique. Les approches proposées pour le résoudre peuvent être appliquées à une grande variété de problèmes de traitement de signaux. Les modèles physiques font souvent usage d'un certain nombre de fonctions d'ajustement comme base paramétrique simplifiée pour approximer une fonction non linéaire de grande dimension (généralement intraitable d'un point de vue calcul). Des modèles réalistes de radiation contiennent des milliers de fonctions d'ajustement. L'utilisation de réseaux neuronaux dans ce genre d'applications est rare, et les techniques les plus efficaces reposent sur des méthodes de validation croisée ou impliquent d'autres techniques bien plus lourdes qui se révèlent souvent impratiquables quand un grand nombre de modèles doivent être entrainés. Deux approches différentes ont été employées ici: la régression multiple multivariée et la minimisation globale de séquence. La première fait apparaı̂tre que l'intégration de plusieurs modèles de régression non linéaires dans un simple réseau de neurones peut augmenter à la fois les performances générales et la vitesse de calcul. Pour cette approche on propose une méthode d'optimisation par laquelle on spécialise globalement les modèles sur des séquences typiques de signaux d'entrée. On montre de plus que la précision générale n'est pas dégradée mais qu'on peut au contraire spécialiser les modèles.

Introduction

The motivation behind the research presented in this paper was the development of a system for the fast and accurate computation of intensity of radiation profiles, observed through a column of absorbing gas. This is a problem of interest for the aeronautical industry, and is applied to a wide range of domains such as infrared monitoring or turboreactor design. There exists a method for the precise resolution of this thermodynamical problem [3], but unfortunately it is too computationally intensive, and cannot be used for interactive simulations or real-time applications. A number of approximated physical models exist and have been implemented to compute relatively accurate profiles in reasonable computational time 3, 6.

To implement such approximated models, however, it is necessary to determine a large number of application-dependant parameters, to which models are extremely sensible. This is usually an expensive and lengthy task, and demands great experience on the part of model constructors. Part of the difficulty is the nonlinear nature of the problem: physicists must carefully transform and partition data so that local linear regression models work.

In this implementation we utilize nonlinear multi-layer perceptrons (MLPs) to build accurate models which are constructed automatically, without the lengthy hand-tuning of traditional models. We will see that, while the automatically constructed MLP models cannot compete in accuracy with hand-crafted local linear models, they can be made sufficiently accurate and offer an alternative for applications in which the cost of hand-crafting them is prohibitive. This is not possible with traditional linear models. Because of the large number of regressors needed, this is in fact not possible for traditional neural network techniques either.

In the process of developing the thermodynamical application, we have obtained several results which we believe will be of general interest to developers of neural network applications. For example, a typical problem of nonlinear regression is the large number of parameters needed, in comparison to linear models. We show that it is possible to construct MLP architectures with equal or even less parameters than the original linear models. Specifically, we show how the integration of several correlated regressors into a single neural network may bring a drastic reduction in the number of network parameters while improving its generalization performance and reducing the computational costs of its utilization and development.

In the standard model, a large number of linear regressors (several thousands, in our application) are constructed to approximate independently small regions of the signal space. These regions correspond to sequences of points (segment descriptions) in typical gas columns. Regressors are then combined into a complex nonlinear model. In the second part of our work we propose a method of global sequence optimization, which goes beyond the standard thermodynamical model. We set out to train our models globally, taking into account the entire sequence of points in a column, with respect to the researched profile. This allows us to improve the generalization performance on a family of input sequences, biasing or `specializing' the model in a natural and easy to implement manner.

In Section 2we give a brief overview of the thermodynamical model in which the application is based. We will see that the main problem that needs to be solved is fitting a large number of functions which form a parametric base for the global solution of the problem, in order to integrate a function of these parameters over the signal sequence. In Section 3we discuss how ordinary least-square regression and neural networks (NNs) may be used to fit these functions. We will see that a straightforward substitution of linear models by NNs brings a drastic improvement in accuracy, but at a very high computational cost. In Section 4we discuss a method to reduce this cost while improving the generalization capacity of the NN model. This method consists in using a single NN to approximate simultaneously a family of parameter functions. Finally, in Section 5, we propose a two-stage optimization procedure which directly optimizes the global sequence instead of fitting independently the different parameter functions. This improves generalization and has the special interest of providing a subtle and easily implementable way to specialize the models after normal training is completed.

Section snippets

The physical problem

Below, we briefly introduce the physical problem, without going into details; these are not important for the comprehension of this work and are beyond the scope of the paper. We have provided a more thorough description of the physical model in Appendix A. For a detailed description the reader may refer to 3, 6.

Spectral radiation profiles offer a concise representation of a radiation source over a certain region of the spectrum. The radiation models allow to compute these profiles in any point

Independent regression framework

We dispose of a set of construction points obtained by the LBL model from which the transmissivity parameter functions are to be fitted. Formally, we dispose of a set of points (xs,ys)s=1…S which assign, to each segment description xs=(xs1,xs2,…,xsp), the values of the parameter functions at the segment, ys=(ys1,ys2,…,ysN). NN terminology denotes x and y as the input and desired patterns, respectively, while in statistics they are called predictor and response measurement vectors.

A simple way

Multiple function regression

Several ideas lead naturally to the implementation of multiple function regressors for our application. The combination of regressors, when correlated, may improve generalization. In the case of linear regression it can be shown that a weighted sum of a set of regressors gives a better (or at worst equal) solution than the independent regressors. Appropriate weights may be estimated from data [1]. There do not exist to our knowledge similar results in the case of non-linear function

Two-stage sequence optimization

Whether we employed linear or nonlinear regression methods, the models of 3 Independent regression framework, 4 Multiple function regressionwere concerned with the approximation of the parameter functions, without regard for the role each function plays in the spectral radiation model. However, we can see from , that not all the parameter functions are equally important for the intensity calculations. Depending on the column characteristics some parameters will have negligible effect.

Conclusions

We have presented an effective model of spectral luminance computation based on neural networks. This model transforms a sequence of column measurements into a spectral radiation intensity profile. The main characteristics of the problem is that several thousands of regressors have to be simultaneously estimated. This forbids the use of traditional estimation methods such as cross-validation or regularization. We have shown how to make the computations tractable by taking benefit from the

Acknowledgements

This material is based on work supported in part by the French Aeronautical Society SNECMA.

References (8)

  • Ph. Rivière et al.

    Correlated-k and fictitious gas methods for H2O near 2.7 μm

    J. Quant. Spetrosc. Radiat. Transfer

    (1992)
  • L. Breiman, J.H. Friedman, Predicting multivariate responses in multiple linear regression, Royal Statist. Soc., in...
  • R. Caruana

    Learning many related tasks at the same time with backpropagation

    Adv. Neural Inform. Systems

    (1994)
  • R.M. Goody et al.

    Atmospheric Radiation

    (1989)
There are more references available in the full text version of this article.

Cited by (0)

View full text