Skip to main content
SpringerLink
Log in

RBFOpt: an open-source library for black-box optimization with costly function evaluations

  • Full Length Paper
  • Published:
Mathematical Programming Computation Aims and scope Submit manuscript

Abstract

We consider the problem of optimizing an unknown function given as an oracle over a mixed-integer box-constrained set. We assume that the oracle is expensive to evaluate, so that estimating partial derivatives by finite differences is impractical. In the literature, this is typically called a black-box optimization problem with costly evaluation. This paper describes the solution methodology implemented in the open-source library RBFOpt, available on COIN-OR. The algorithm is based on the Radial Basis Function method originally proposed by Gutmann (J Glob Optim 19:201–227, 2001. https://doi.org/10.1023/A:1011255519438), which builds and iteratively refines a surrogate model of the unknown objective function. The two main methodological contributions of this paper are an approach to exploit a noisy but less expensive oracle to accelerate convergence to the optimum of the exact oracle, and the introduction of an automatic model selection phase during the optimization process. Numerical experiments show that RBFOpt is highly competitive on a test set of continuous and mixed-integer nonlinear unconstrained problems taken from the literature: it outperforms the open-source solvers included in our comparison by a large amount, and performs slightly better than a commercial solver. Our empirical evaluation provides insight on which parameterizations of the algorithm are the most effective in practice. The software reviewed as part of this submission was given the Digital Object Identifier (DOI) https://doi.org/10.5281/zenodo.597767.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Achterberg, T.: SCIP: solving constraint integer programs. Math. Program. Comput. 1(1), 1–41 (2009)

    Article  MathSciNet  Google Scholar 

  2. Audet, C., Dennis Jr., J.: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17(1), 188–217 (2004)

    Article  MathSciNet  Google Scholar 

  3. Audet, C., Kokkolaras, M., Le Digabel, S., Talgorn, B.: Order-based error for managing ensembles of surrogates in mesh adaptive direct search. J. Glob. Optim. 70(3), 645–675 (2018)

    Article  MathSciNet  Google Scholar 

  4. Baudoui, V.: Optimisation robuste multiobjectifs par modèles de substitution. Ph.D. thesis, University of Toulouse Paul Sabatier (2012)

  5. Björkman, M., Holmström, K.: Global optimization of costly nonconvex functions using radial basis functions. Optim. Eng. 1(4), 373–397 (2000)

    Article  MathSciNet  Google Scholar 

  6. Bonami, P., Biegler, L., Conn, A., Cornuéjols, G., Grossmann, I., Laird, C., Lee, J., Lodi, A., Margot, F., Sawaya, N., Wächter, A.: An algorithmic framework for convex mixed integer nonlinear programs. Discrete Optim. 5, 186–204 (2008)

    Article  MathSciNet  Google Scholar 

  7. Bussieck, M.R., Drud, A.S., Meeraus, A.: MINLPLib—a collection of test models for mixed-integer nonlinear programming. INFORMS J. Comput. 15(1), 114–119 (2003)

    Article  MathSciNet  Google Scholar 

  8. Byrd, R.H., Nocedal, J., Waltz, R.A.: KNITRO: an integrated package for nonlinear optimization. In: Di Pillo, G., Roma, M. (eds.) Large-Scale Nonlinear Optimization, pp. 35–59. Springer, New York (2006)

    Chapter  Google Scholar 

  9. Conn, A.R., Scheinberg, K., Toint, P.L.: Recent progress in unconstrained nonlinear optimization without derivatives. Math. Program. 79(1–3), 397–414 (1997). https://doi.org/10.1007/BF02614326

    Article  MathSciNet  MATH  Google Scholar 

  10. Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. MPS-SIAM Series on Optimization. Society for Industrial and Applied Mathematics, Philadelphia (2009)

    Book  Google Scholar 

  11. Costa, A., Di Buccio, E., Melucci, M., Nannicini, G.: Efficient parameter estimation for information retrieval using black-box optimization. IEEE Trans. Knowl. Data Eng. 30, 1240–1253 (2017)

    Article  Google Scholar 

  12. Costa, A., Nannicini, G., Schroepfer, T., Wortmann, T.: Black-box optimization of lighting simulation in architectural design. In: Cardin, M.A., Krob, D., Chuen, L., Tan, Y., Wood, K. (eds.) Designing Smart Cities: Proceedings of the First Asia-Pacific Conference on Complex Systems Design & Management, CSD&M Asia 2014, pp. 27–39. Springer (2015)

    Google Scholar 

  13. D’Ambrosio, C., Nannicini, G., Sartor, G.: MILP models for the selection of a small set of well-distributed points. Oper. Res. Lett. 45(1), 46–52 (2017)

    Article  MathSciNet  Google Scholar 

  14. Diaz, G.I., Fokour, A., Nannicini, G., Samulowitz, H.: An effective algorithm for hyperparameter optimization of neural networks. IBM J. Res. Dev. 61(4/5), 9-1 (2017)

    Article  Google Scholar 

  15. Dixon, L., Szego, G.: The global optimization problem: an introduction. In: Dixon, L., Szego, G. (eds.) Towards Global Optimization, pp. 1–15. North Holland, Amsterdam (1975)

    Google Scholar 

  16. Eriksson, D., Bindel, D., Shoemaker, C.: Surrogate optimization toolbox (pySOT) (2015). http://github.com/dme65/pySOT

  17. Fuerle, F., Sienz, J.: Formulation of the Audze–Eglais uniform latin hypercube design of experiments for constrained design spaces. Adv. Eng. Softw. 42(9), 680–689 (2011)

    Article  Google Scholar 

  18. Gablonsky, J., Kelley, C.: A locally-biased form of the DIRECT algorithm. J. Glob. Optim. 21(1), 27–37 (2001)

    Article  MathSciNet  Google Scholar 

  19. Gendreau, M., Potvin, J.Y. (eds.): Handbook of Metaheuristics, 2nd edn. Kluwer, Dordrecht (2010)

    MATH  Google Scholar 

  20. Glover, F., Kochenberger, G. (eds.): Handbook of Metaheuristics. Kluwer, Dordrecht (2003)

    MATH  Google Scholar 

  21. Gomes, C.P., Selman, B., Crato, N., Kautz, H.: Heavy-tailed phenomena in satisfiability and constraint satisfaction problems. J. Autom. Reason. 24(1–2), 67–100 (2000). https://doi.org/10.1023/A:1006314320276

    Article  MathSciNet  MATH  Google Scholar 

  22. Gutmann, H.M.: A radial basis function method for global optimization. J. Glob. Optim. 19, 201–227 (2001). https://doi.org/10.1023/A:1011255519438

    Article  MathSciNet  MATH  Google Scholar 

  23. Hart, W.E., Laird, C., Watson, J.P., Woodruff, D.L.: Pyomo—optimization Modeling in Python. Springer Optimization and Its Applications, vol. 67. Springer, Berlin (2012)

    Book  Google Scholar 

  24. Hart, W.E., Watson, J.P., Woodruff, D.L.: Pyomo: modeling and solving mathematical programs in Python. Math. Program. Comput. 3(3), 219–260 (2011). https://doi.org/10.1007/s12532-011-0026-8

    Article  MathSciNet  Google Scholar 

  25. Hemker, T.: Derivative free surrogate optimization for mixed-integer nonlinear black-box problems in engineering. Master’s thesis, Technischen Universität Darmstadt (2008)

  26. Holmström, K.: An adaptive radial basis algorithm (ARBF) for expensive black-box global optimization. J. Glob. Optim. 41(3), 447–464 (2008)

    Article  MathSciNet  Google Scholar 

  27. Holmström, K., Quttineh, N.H., Edvall, M.M.: An adaptive radial basis algorithm (ARBF) for expensive black-box mixed-integer constrained global optimization. Optim. Eng. 9(4), 311–339 (2008)

    Article  MathSciNet  Google Scholar 

  28. Huyer, W., Neumaier, A.: SNOBFIT—stable noisy optimization by branch and fit. ACM Trans. Math. Softw. 35(2), 1–25 (2008)

    Article  MathSciNet  Google Scholar 

  29. Ilievski, I., Akhtar, T., Feng, J., Shoemaker, C.A.: Efficient hyperparameter optimization of deep learning algorithms using deterministic RBF surrogates. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

  30. Jakobsson, S., Patriksson, M., Rudholm, J., Wojciechowski, A.: A method for simulation based optimization using radial basis functions. Optim. Eng. 11(4), 501–532 (2010)

    Article  MathSciNet  Google Scholar 

  31. Johnson, S.G.: The NLopt nonlinear-optimization package. http://ab-initio.mit.edu/nlopt

  32. Jones, D., Perttunen, C., Stuckman, B.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl. 79(1), 157–181 (1993)

    Article  MathSciNet  Google Scholar 

  33. Jones, D., Schonlau, M., Welch, W.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13(4), 455–492 (1998)

    Article  MathSciNet  Google Scholar 

  34. Kolda, T.G., Lewis, R.M., Torczon, V.J.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45(3), 385–482 (2003)

    Article  MathSciNet  Google Scholar 

  35. Le Digabel, S.: Algorithm 909: NOMAD: nonlinear optimization with the MADS algorithm. ACM Trans. Math. Softw. 37(4), 44:1–44:15 (2011). https://doi.org/10.1145/1916461.1916468

    Article  MathSciNet  MATH  Google Scholar 

  36. MINLP Library 2. http://www.gamsworld.org/minlp/minlplib2/html/

  37. Moré, J., Wild, S.M.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009)

    Article  MathSciNet  Google Scholar 

  38. Müller, J.: MISO: mixed-integer surrogate optimization framework. Optim. Eng. 1–27 (2015). https://doi.org/10.1007/s11081-015-9281-2

    Article  MathSciNet  Google Scholar 

  39. Müller, J., Paudel, R., Shoemaker, C.A., Woodbury, J., Wang, Y., Mahowald, N.: \(\text{ CH }_{4}\) parameter estimation in CLM4.5bgc using surrogate global optimization. Geosci. Model Dev. 8(10), 3285–3310 (2015). https://doi.org/10.5194/gmd-8-3285-2015

    Article  Google Scholar 

  40. Müller, J., Shoemaker, C.A.: Influence of ensemble surrogate models and sampling strategy on the solution quality of algorithms forcomputationally expensive black-box global optimization problems. J. Glob. Optim. 60(2), 123–144 (2014). https://doi.org/10.1007/s10898-014-0184-0

    Article  MATH  Google Scholar 

  41. Müller, J., Shoemaker, C.A., Piché, R.: SO-MI: a surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems. Comput. Oper. Res. 40(5), 1383–1400 (2013). https://doi.org/10.1016/j.cor.2012.08.022

    Article  MathSciNet  MATH  Google Scholar 

  42. Nelder, J.A., Mead, R.: A simplex method for function minimization. Comput. J. 7(4), 308–313 (1965)

    Article  MathSciNet  Google Scholar 

  43. Neumaier, A.: Neumaier’s collection of test problems for global optimization. http://www.mat.univie.ac.at/~neum/glopt/my_problems.html. Retrieved in May 2014

  44. Powell, M.: Recent research at Cambridge on radial basis functions. In: Müller, M.W., Buhmann, M.D., Mache, D.H., Felten, M. (eds.) New Developments in Approximation Theory. International Series of Numerical Mathematics, vol. 132, pp. 215–232. Birkhauser Verlag, Basel (1999)

    Chapter  Google Scholar 

  45. Powell, M.J.: The BOBYQA algorithm for bound constrained optimization without derivatives. Technical Report, Cambridge NA Report NA2009/06, University of Cambridge (2009)

  46. Regis, R., Shoemaker, C.: Improved strategies for radial basis function methods for global optimization. J. Glob. Optim. 37, 113–135 (2007). https://doi.org/10.1007/s10898-006-9040-1

    Article  MathSciNet  MATH  Google Scholar 

  47. Regis, R.G., Shoemaker, C.A.: A stochastic radial basis function method for the global optimization of expensive functions. INFORMS J. Comput. 19(4), 497–509 (2007). https://doi.org/10.1287/ijoc.1060.0182

    Article  MathSciNet  MATH  Google Scholar 

  48. Regis, R.G., Shoemaker, C.A.: A quasi-multistart framework for global optimization of expensive functions using response surface models. J. Glob. Optim. 56(4), 1719–1753 (2013)

    Article  MathSciNet  Google Scholar 

  49. Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56(3), 1247–1293 (2013)

    Article  MathSciNet  Google Scholar 

  50. Schoen, F.: A wide class of test functions for global optimization. J. Glob. Optim. 3(2), 133–137 (1993)

    Article  MathSciNet  Google Scholar 

  51. Törn, A., Žilinskas, A.: Global Optimization. Springer, Berlin (1987)

    MATH  Google Scholar 

  52. Wächter, A., Biegler, L.T.: On the implementation of a primal-dual interior point filter line search algorithm for large-scale nonlinear programming. Math. Program. 106(1), 25–57 (2006)

    Article  MathSciNet  Google Scholar 

  53. Wortmann, T., Costa, A., Nannicini, G., Schroepfer, T.: Advantages of surrogate models for architectural design optimization. Artif. Intell. Eng. Des. Anal. Manuf. 29(4), 471–481 (2015)

    Article  Google Scholar 

  54. Wortmann, T., Waibel, C., Nannicini, G., Evins, R., Schroepfer, T., Carmeliet, J.: Are genetic algorithms really the best choice for building energy optimization? In: Proceedings of the Symposium on Simulation for Architecture & Urban Design (SimAUD), pp. 51–58. SCS, Toronto, Canada (2017)

Download references

Acknowledgements

The authors are grateful for partial support by the SUTD-MIT International Design Center under grant IDG21300102. The research of A. C. was partially conducted at the Future Resilient Systems and the Future Cities Laboratory at the Singapore-ETH Centre (SEC). The SEC was established as a collaboration between ETH Zurich and National Research Foundation (NRF) Singapore (FI 370074011, FI 370074016) under the auspices of the NRF’s Campus for Research Excellence and Technological Enterprise (CREATE) program.

Author information

Authors and Affiliations

  1. Future Cities Laboratory Program, ETH Zurich, Singapore, Singapore

    Alberto Costa

  2. IBM T.J. Watson Research Center, Yorktown Heights, NY, USA

    Giacomo Nannicini

Authors
  1. Alberto Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giacomo Nannicini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giacomo Nannicini.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Costa, A., Nannicini, G. RBFOpt: an open-source library for black-box optimization with costly function evaluations. Math. Prog. Comp. 10, 597–629 (2018). https://doi.org/10.1007/s12532-018-0144-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12532-018-0144-7

Keywords

Mathematics Subject Classification

Search

Navigation