Elsevier

Computers & Chemical Engineering

Volume 106, 2 November 2017, Pages 464-479
Computers & Chemical Engineering

Data-driven robust optimization based on kernel learning

https://doi.org/10.1016/j.compchemeng.2017.07.004Get rights and content

Highlights

  • A new data-driven robust optimization method is proposed based on kernel learning.

  • A compact convex uncertainty set can be efficiently learnt from massive data.

  • Fraction of data coverage can be conveniently adjusted using only one parameter.

  • Computational tractability is well preserved by the robust counterpart problem.

  • The method reduces conservatism evidently and enjoys practical applicability.

Abstract

We propose piecewise linear kernel-based support vector clustering (SVC) as a new approach tailored to data-driven robust optimization. By solving a quadratic program, the distributional geometry of massive uncertain data can be effectively captured as a compact convex uncertainty set, which considerably reduces conservatism of robust optimization problems. The induced robust counterpart problem retains the same type as the deterministic problem, which provides significant computational benefits. In addition, by exploiting statistical properties of SVC, the fraction of data coverage of the data-driven uncertainty set can be easily selected by adjusting only one parameter, which furnishes an interpretable and pragmatic way to control conservatism and exclude outliers. Numerical studies and an industrial application of process network planning demonstrate that, the proposed data-driven approach can effectively utilize useful information with massive data, and better hedge against uncertainties and yield less conservative solutions.

Introduction

In science and engineering, optimization problems are inevitably encountered whenever one seeks to make decisions by maximizing/minimizing some certain criterion. However, real-world parameters are subject to randomness in various degrees, rendering deterministic optimization models unreliable in an uncertain environment (Sahinidis, 2004). It has been demonstrated that even a slight perturbation on parameters in an optimization problem could exert overwhelming effects on the computed optimal solutions, which results in suboptimality or even infeasibility of optimization problems (Mulvey et al., 1995). Motivated by the urgent requirement of handling uncertainties in decision making, stochastic optimization and robust optimization have received immense research attentions in recent decades (Kall et al., 1994, Ben-Tal et al., 2009, Birge and Louveaux, 2011, Bertsimas et al., 2011, Gabrel et al., 2014, Yuan et al., 2017), which hedge against uncertainties by bringing in conservatism. Stochastic optimization entails complete knowledge about the underlying probability distribution of uncertainties, which may be unrealistic in practice. As an effective alternative, robust optimization takes a deterministic and set-based way to model uncertainties, and balance between the modeling power and computational tractability, which has obtained considerable attentions recently in the realm of process systems engineering. Typical applications include process network design (Gong et al., 2016, Gong and You, 2017), supply chain management (Tong et al., 2014, Yue and You, 2016), and process scheduling (Shi and You, 2016).

A paramount ingredient in robust optimization is to construct an uncertainty set including probable realizations of uncertain parameters. The earliest attempts in formulating uncertainty sets date back to the 1970s, with the work of Soyster (1973), in which coefficients are perturbed by uncertainties distributed in a known box. Despite its computational convenience and guaranteed feasibility, the box uncertainty set tends to induce over-conservative decisions. Later, immense research effort has been made on devising more flexible robust models to ameliorate over-conservatism. Ellipsoidal uncertainty sets have been put forward independently by El Ghaoui et al., 1998, Ben-Tal and Nemirovski, 1998, Ben-Tal and Nemirovski, 1999, based on which the robust counterpart model simplifies to a conic quadratic problem in the presence of linear constraints. To enhance the modeling flexibilities, intersections of basic uncertainty sets have been designed, including the “interval + ellipsoidal” uncertainty set (Ben-Tal and Nemirovski, 2000) and the “interval + polyhedral” uncertainty set. Bertsimas and Sim (2004) robustify linear programs using a polyhedral uncertainty set adjustable with the so-called budget, which turns out to be identical to the “interval + polyhedral” uncertainty set. Interested readers are referred to review papers (Bertsimas et al., 2011, Gabrel et al., 2014) and the monograph (Ben-Tal et al., 2009) for a comprehensive overview of uncertainty sets-induced robust optimization methods.

Despite the burgeoning prevalence of uncertainty set-induced robust optimization, a potential limitation is that uncertainties in each dimension are assumed as being independently and symmetrically distributed. The issue of data correlation is first addressed by Bertsimas and Sim (2004), in which correlated uncertainties are disentangled by means of an underlying source of independent uncertainties, expressed as:a˜ij=aij+kKiη˜ikgkj.where {aij} stands for the nominal value, and {η˜ik} denotes the source of independent uncertainties. Ferreira et al. (2012) suggest determining the values of {a˜ij} and {gkj} by means of principal component analysis (PCA) and minimum power decomposition (MPD). Based on (1), classical symmetric uncertainty sets are generalized by Yuan et al. (2016), and explicit formulations of their robust counterparts are also provided. Jalilvand-Nejad et al. (2016) devise correlated polyhedral uncertainty sets using second-order statistical information from historical data. In regard to asymmetric uncertainties, Chen et al. (2007) adopt the forward and backward deviations to capture distributional asymmetry through the norm-induced uncertainty set while still preserving its tractability. In Natarajan et al. (2008), a modified value-at-risk (VaR) measure is developed and investigated by taking into account asymmetries in the distributions of portfolio returns.

A prominent and practical issue of uncertainty set-induced robust optimization is how to determine the set coefficients appropriately, which are in general assumed as known according to domain-specific knowledge. In the absence of first-principle knowledge, leveraging available historical data provides a practical way to characterize the distributional information. For instance, the lower and the upper bounds of an interval can be specified as the minimum and the maximum of observed data samples, albeit conservatively. More sophisticated approaches make use of variance and covariance of historical data, which renders the model statistically interpretable (Pachamanova, 2002, Bertsimas and Pachamanova, 2008, Ferreira et al., 2012). An alternative streamline of data-driven optimization is the distributionally robust optimization, which utilizes both data and hypothesis tests to construct the ambiguity set including at a high confidence level (Delage and Ye, 2010, Jiang and Guan, 2016, Bertsimas et al., 2017). However, one still needs to specify the type of hypothesis tests to yield a reliable solution, and the induced optimization problem is generally difficult to solve (Hanasusanto et al., 2017). For example, the moment-based hypothesis test typically leads to reformulations in terms of linear matrix inequalities and bilinear matrix inequalities (Delage and Ye, 2010, Zymler et al., 2013), which erect obstacles to further tackle mixed-integer and large-scale problems that are commonly encountered in process systems engineering. In this work, we hence focus on uncertainty set-based robust optimization with better applicability and implementation convenience.

In real-world applications, the underlying distribution of uncertainties may be intrinsically complicated and vary under different circumstances. When one is faced with high-dimensional uncertainties, it is rather challenging to choose the type of uncertainty sets by prior knowledge, tune the coefficients, and further evaluate its divergence with the true underlying distribution . The era of big data paves new way for decision-making under uncertainties by exploiting massive data available at hand (Bertsimas et al., 2011, Qin, 2014). A desirable uncertainty set shall flexibly adapt to the intrinsic structure behind data, thereby well characterizing and ameliorating the suboptimality of solutions. From a machine learning perspective, constructions of uncertainty sets based upon historical data can be viewed as an unsupervised learning problem. There have been a plethora of effective unsupervised learning models, for example, kernel density estimation (KDE) and support vector machines (SVM), which could provide powerful representations of data distributions (Bishop, 2006). In principle, one could resort to such machine learning tools to estimate data densities with sufficient accuracies; nevertheless, it remains a challenging task to formulate an appropriate uncertainty set for modeling robust optimization problems. This is mainly because complicating nonlinear items, such as the radial basis function (RBF) exp{−x2/2σ2} and the sigmoid function tanh(γx + r), dominate conventional machine learning models, which invariably prohibit an analytical treatment of robust optimization problems, especially a tractable robust counterpart reformulation. This may somewhat explain the scarce of applications of machine learning models in robust optimization all this time.

In this paper, we propose an effective data-driven approach for robust optimization that is tailored to uncertainty set constructions as well as computational implementations, thereby bridging machine learning and robust optimization directly. As an extended SVM technology, support vector clustering (SVC) has been extensively adopted to estimate the support of an unknown probability distribution from random data samples (Schölkopf et al., 1999; Müller et al., 2001). A particular merit of SVC is that it has theoretical underpinnings in statistical learning theory (Vapnik, 2013), and enjoys desirable generalization performance in face of real-world problems (Lee and Lee, 2005). Instead of using the ubiquitous RBF kernel involving intricate nonlinearities, a novel piecewise linear kernel, referred to as the generalized intersection kernel, is proposed in this work to formulate the SVC model, which entails solving a quadratic program (QP) only. Thanks to the kernel formulation, the SVC model could not only handle correlated uncertainties and lead to asymmetric uncertainty sets, but also enjoys an adaptive complexity, thereby featuring a nonparametric scheme. Most importantly, it leads to a convex polyhedral uncertainty set, thereby rendering the robust counterpart problem of the same type as the deterministic problem, which provides computational convenience. If the deterministic problem is an MILP, then the robust counterpart problem can be also cast as an MILP. In this way, a satisfactory trade-off can be achieved between modeling power of SVC and computational convenience of robust optimization. Moreover, we show that the parameters to be tuned bear explicit statistical implications, which allow one to easily control the conservatism as well as the complexity of the induced robust optimization problems. Numerical and application case studies are conducted to show the practicability and efficacy of the proposed data-driven approach in hedging against uncertainties and alleviating the conservatism of robust solutions.

The layout of this paper is organized as follows. Section 2 revisits the conventional uncertainty sets for robust optimization, and the SVC-based unsupervised learning schema. In Section 3, the piecewise linear kernel-based SVC is proposed, along with the induced data-driven uncertainty set. Its properties are also discussed from various aspects, including the parameter interpretation, asymptotic behavior, and computational tractability. Section 4 is devoted to a comprehensive numerical evaluation on the performance of the proposed approach compared with classical ones. In Section 5, case studies on an application of process network planning are performed, followed by concluding remarks in Section 6.

Section snippets

Uncertainty set induced robust optimization

We consider the following simple linear optimization problem:minxXcTxs.t.Axbwhere xn stands for the decision variables. Without loss of generality, we could assume that only the left-hand side (LHS) coefficients are contaminated by uncertainties. Specifically, we target on a single linear constraint, and suppress its dependence on the constraint index for notational brevity:ka˜kxkb.In most robust optimization literature, the uncertainty is often decomposed as (El Ghaoui et al., 1998,

Data-driven uncertainty set induced by generalized intersection kernel SVC

In this section, we first propose a new piecewise linear kernel for modeling SVC. Then we derive explicit formulations of the induced uncertainty sets, based on which the robust counterpart can be readily obtained. Well-established properties of SVC are exploited to facilitate practical use in robust optimization.

Computational studies

In this section, we first provide illustrative examples to demonstrate the properties of the data-driven SVC-based uncertainty set. Then we conduct comparative studies with existing methods to testify the performance of the induced robust optimization problem with numerical cases.

Application to robust planning of chemical process networks planning

In this section, we study the application of the proposed robust optimization framework to a multi-period process network planning problem. Chemical complexes are typically erected with a number of interconnected processes and versatile chemicals, which provide considerable operational flexibilities in that multiple manufacturing options are accessible for producing a certain chemical (Yue and You, 2013). Moreover, capacity expansions are allowed in each period to maximize the overall profit.

Concluding remarks

In this work, we propose a novel data-driven uncertainty set based on piecewise linear kernel learning for solving robust optimization problems. We propose a new piecewise linear kernel termed as the generalized intersection kernel, and integrate it with SVC for effective unsupervised learning of convex data support. The SVC-based uncertainty set inherits merits of nonparametric kernel learning methods, which could adapt to the complexity of data in an intelligent way. It allows the user to

Acknowledgements

C. Shang and F. You acknowledge financial support from the National Science Foundation (NSF) CAREER Award (CBET-1643244). X. Huang is supported in part by the National Natural Science Foundation of China under Grant 61603248.

References (57)

  • A. Ben-Hur et al.

    Support vector clustering

    J. Mach. Learn. Res.

    (2001)
  • A. Ben-Tal et al.

    Robust Optimization

    (2009)
  • A. Ben-Tal et al.

    Robust convex optimization

    Math. Oper. Res.

    (1998)
  • A. Ben-Tal et al.

    Robust solutions of uncertain linear programs

    Oper. Res. Lett.

    (1999)
  • A. Ben-Tal et al.

    Robust solutions of linear programming problems contaminated with uncertain data

    Math. Progr.

    (2000)
  • D. Bertsimas et al.

    Theory and applications of robust optimization

    SIAM Rev.

    (2011)
  • D. Bertsimas et al.

    Data-driven robust optimization

    Math. Progr.

    (2017)
  • D. Bertsimas et al.

    Robust multiperiod portfolio management in the presence of transaction costs

    Comput. Oper. Res.

    (2008)
  • D. Bertsimas et al.

    The price of robustness

    Oper. Res.

    (2004)
  • J.R. Birge et al.

    Introduction to Stochastic Programming

    (2011)
  • C.M. Bishop

    Pattern Recognition and Machine Learning

    (2006)
  • S. Boyd et al.

    Convex Optimization

    (2004)
  • X. Chen et al.

    A robust optimization perspective on stochastic programming

    Oper. Res.

    (2007)
  • E. Delage et al.

    Distributionally robust optimization under moment uncertainty with application to data-driven problems

    Oper. Res.

    (2010)
  • L. El Ghaoui et al.

    Robust solutions to uncertain semidefinite programs

    SIAM J. Optim.

    (1998)
  • R. Ferreira et al.

    Demand response models with correlated price data: a robust optimization approach

    Appl. Energy

    (2012)
  • V. Gabrel et al.

    Recent advances in robust optimization: an overview

    Eur. J. Oper. Res.

    (2014)
  • J. Gong et al.

    Unraveling optimal biomass processing routes from bioconversion product and process networks under uncertainty: an adaptive robust optimization approach

    ACS Sustain. Chem. Eng.

    (2016)
  • J. Gong et al.

    Optimal processing network design under uncertainty for producing fuels and value-added bioproducts from microalgae: two-stage adaptive robust mixed integer fractional programming model and computationally efficient solution algorithm

    AIChE J.

    (2017)
  • M. Grant et al.

    CVX: Matlab

    (2008)
  • Y.A. Guzman et al.

    New a priori and a posteriori probabilistic bounds for robust counterpart optimization: I. Unknown probability distributions

    Comput. Chem. Eng.

    (2016)
  • Y.A. Guzman et al.

    New a priori and a posteriori probabilistic bounds for robust counterpart optimization: II. A priori bounds for known symmetric and asymmetric probability distributions

    Comput. Chem. Eng.

    (2017)
  • G.A. Hanasusanto et al.

    Ambiguous joint chance constraints under mean and dispersion information

    Oper. Res.

    (2017)
  • X. Huang et al.

    Classification with truncated ℓ1 distance kernel

    IEEE Trans. Neural Netw. Learn. Syst.

    (2017)
  • A. Hyvärinen et al.
    (2004)
  • A. Jalilvand-Nejad et al.

    Robust optimization under correlated polyhedral uncertainty set

    Comput. Ind. Eng.

    (2016)
  • R. Jiang et al.

    Data-driven chance constrained stochastic program

    Math. Progr.

    (2016)
  • P. Kall et al.

    Stochastic Programming

    (1994)
  • Cited by (171)

    • Optimization with constraint learning: A framework and survey

      2024, European Journal of Operational Research
    View all citing articles on Scopus
    View full text