Dynamic load balancing for parallel interval-Newton using message passing

https://doi.org/10.1016/S0098-1354(02)00006-6Get rights and content

Abstract

Branch-and-prune and branch-and-bound techniques are commonly used for intelligent search in finding all solutions, or the optimal solution, within a space of interest. The corresponding binary tree structure provides a natural parallelism allowing concurrent evaluation of subproblems using parallel computing technology. Of special interest here are techniques derived from interval analysis, in particular an interval-Newton/generalized-bisection procedure. In this context, we discuss issues of load balancing and work scheduling that arise in the implementation of parallel interval-Newton on a cluster of workstations using message passing, and describe and analyze techniques for this purpose. Results using an asynchronous diffusive load balancing strategy show that a consistently high efficiency can be achieved in solving nonlinear equations, providing excellent scalability, especially with the use of a two-dimensional torus virtual network. The effectiveness of the approach used, especially in connection with a novel stack management scheme, is also demonstrated in the consistent superlinear speedups observed in performing global optimization.

Introduction

Process systems engineering involves the development and application of computer-aided, model-based techniques for the design, optimization, operation and control of chemical processing systems, as well as for the associated planning, scheduling and maintenance problems. In this field, two core numerical issues are the solution of nonlinear equation systems and of nonlinear programming (optimization) problems. However, neither of these problems can be solved with complete reliability using standard techniques. In this context, it has been shown (e.g. Balaji & Seader, 1995, Berner, McKinnon & Millar, 1999, Byrne & Bogle, 1996, Byrne & Bogle, 2000, Gau, Brennecke & Stadtherr, 2000, Gau & Stadtherr, 2000, Han, Manousiousthakis & Choi, 1997; Hua et al., 1996, Hua et al., 1998; Maier et al., 1998, Maier et al., 2000; Maier & Stadtherr, 2001, McKinnon, Millar & Mongeau, 1996, Schnepper & Stadtherr, 1996, Stadtherr, Schnepper & Brennecke, 1995, Stradi, Brennecke, Kohn & Stadtherr, 2001, Tessier, Brennecke & Stadtherr, 2000, Vaidyanathan & El-Halwagi, 1994, Xu, Scurto, Castier, Brennecke & Stadtherr, 2000) that methods based on interval mathematics, such as the interval-Newton technique (e.g. Hansen, 1992, Kearfott, 1996, Neumaier, 1990), can be used to solve the nonlinear equation solving and optimization problems with complete reliability, providing a mathematical and computational guarantee that either all roots are found in the nonlinear equation solving problem, or that the global optimum is found in the nonlinear programming problem. Unfortunately, though not unexpectedly, this guarantee of reliability may come at the cost of a high CPU time requirement, though it is often difficult to predict from problem characteristics, such as number of variables, when this is going to be the case. Fortunately, the interval-Newton method is implemented using branch-and-prune (BP) or branch-and-bound (BB) strategies that present a natural parallelism, and which therefore are particularly amenable to parallel processing. Thus, we seek to make effective use of high performance computing technology (parallel computing) to implement BP and BB strategies for reliably and efficiently solving problems in chemical process modeling and engineering.

Of particular interest here is the use of distributed parallel computing using a cluster of workstations (COW), in which multiple workstations on a network are used as a single parallel computing resource by employing message passing. This sort of parallel computing system has advantages since it is relatively inexpensive, and is based on widely available hardware. Thus, such an approach to parallel computing has become an important trend in providing high performance computing resources in science and engineering. In this paper, we focus specifically on issues of load balancing and scheduling that arise in the implementation of parallel BB and BP using message passing on a workstation cluster, and describe and analyze techniques for this purpose. In particular, we will consider load balancing and scheduling in the context of an interval-Newton/generalized-bisection (IN/GB) algorithm, details of which are given by Schnepper and Stadtherr (1996). Applications to problems arising in chemical process engineering are used to demonstrate the effectiveness of the approach used. These example problems are: (1) estimation of parameters in the Wilson equation using a maximum likelihood objective for a mixture of formic acid and water (Gau, 2001); (2) computation of mixture critical points from cubic equation-of-state models (Stradi et al., 2001); and (3) estimation of parameters in the Van Laar equation using an error-in-variables approach for a mixture of methanol and 1,2-dichloroethane (Esposito & Floudas, 1998, Gau & Stadtherr, 2000).

Section snippets

Background

BP and BB algorithms are general-purpose intelligent search techniques for finding all solutions, or the optimal solution, within a search space of interest, and have a wide range of applications. These techniques employ successive decomposition (tessellation) of the global problem into smaller disjoint or independent subproblems that are solved recursively until all solutions, or the optimal solution, are found. BB and BP methods have many important applications in engineering and science,

Dynamic load balancing

As noted above, since the subproblems to be solved are independent, the execution of interval-Newton techniques, whether BP or BB, on distributed parallel systems can clearly provide improvements in computational efficiency. And since, for some practical problems, the binary tree that needs to be searched may be quite large, there may in fact be a strong motivation for trying to exploit the opportunity for parallel computing. However, because of the irregular structure of the binary tree, doing

Implementation of dynamic load balancing algorithms

In this section, a sequence of three algorithms (SWS, SDLB and ADLB) is described for load balancing in a binary tree, with each algorithm in the sequence representing an improvement in principle over the previous one. The last method (ADLB) represents a synthesis of the most attractive and effective strategies adapted from previous research studies, and also incorporates some novel strategies in this context. Though ADLB is expected to be superior, we describe all three approaches, as this

Effect of virtual network

The virtual network used in the local coordination of neighbor processors for workload distribution and message propagation has an important effect on network communication overhead. Since more connections mean higher communication costs, the number of neighbors on the virtual network affects the message propagation speed across the set of processors. On one hand, having more connections among processors can reduce the message diffusion distance and help balance overloaded or underloaded

Stack management

Another issue of interest is how to improve the parallel search efficiency of interval BB for global optimization problems. As noted above, when a parallel BB approach is applied to solve global minimization problems, the amount of work required will typically differ depending on the number of processors used. This is because the number of processors used affects the timing with which best (least) upper bounds on the global minimum are found and propagated across the network to the entire

Concluding remarks

We have described here how load management strategies can be used for effectively solving interval BB and BP problems in parallel on a network-based COW. Of the dynamic load balancing algorithms considered, the best performance was achieved by the ADLB approach. This overlaps communication and computation by the use of the asynchronous nonblocking communication functions provided by MPI, and uses a type of diffusive load-adjusting scheme to prevents out-of-work idle states while keeping

Acknowledgements

This work has been supported in part by the donors of The Petroleum Research Fund, administered by the ACS, under Grant 35979-AC9, by the National Science Foundation Grant EEC97-00537-CRCD, by the Environmental Protection Agency Grant R826-734-01-0. Computational hardware was provided in part through US Army Research Office Grant DAAG55-98-1-0091.

References (83)

  • C.-Y. Gau et al.

    Reliable parameter estimation in VLE modeling

    Fluid Phase Equilibria

    (2000)
  • J.Z. Hua et al.

    Reliable prediction of phase stability using an interval-Newton method

    Fluid Phase Equilibria

    (1996)
  • J.L. Klepeis et al.

    Deterministic global optimization and torsion angle dynamics for molecular structure prediction

    Computers and Chemical Engineering

    (2000)
  • G.K. Kudva et al.

    A distributed control architecture for branch and bound calculations

    Computers & Chemical Engineering

    (1995)
  • R.W. Maier et al.

    Reliable computation of reactive azeotropes

    Computers & Chemical Engineering

    (2000)
  • C.M. McDonald et al.

    GLOPEQ: a new computational tool for the phase and chemical equilibrium problem

    Computers & Chemical Engineering

    (1997)
  • G. Mitra et al.

    A distributed processing algorithm for solving integer problems using a cluster of workstations

    Parallel Computing

    (1997)
  • J.F. Pekny et al.

    An exact parallel algorithm for scheduling when production costs depend on consecutive system states

    Computers & Chemical Engineering

    (1990)
  • J.F. Pekny et al.

    An exact algorithm for resource constrained sequencing with application to production scheduling under an aggregate deadline

    Computers & Chemical Engineering

    (1993)
  • R.A. Rushmeier

    Experiments with parallel branch-and-bound algorithms for the set covering problems

    Operations Research Letters

    (1993)
  • C.A. Schnepper et al.

    Robust process simulation using interval methods

    Computers & Chemical Engineering

    (1996)
  • E.M.B. Smith et al.

    A symbolic reformulation/spatial branch-and-bound algorithm for the global optimization of nonconvex MINLPs

    Computers & Chemical Engineering

    (1999)
  • S. Teng

    Adaptive parallel algorithms for integral knapsack problems

    Journal of Parallel and Distributed Computing

    (1990)
  • S.R. Tessier et al.

    Reliable phase stability analysis for excess Gibbs energy models

    Chemical Engineering Science

    (2000)
  • J. Troya et al.

    A study of parallel branch-and-bound algorithms with best-bound-first search

    Parallel Computing

    (1989)
  • C.S. Adjiman et al.

    Global optimization of mixed-integer nonlinear problems

    AIChE Journal

    (2000)
  • I.P. Androulakis et al.

    Distributed branch and bound algorithms for global optimization

  • I.P. Androulakis et al.

    Distributed decomposition based approaches in global optimization

  • Anstreicher, K.M., Brixius, N., Goux, J.-P., & Linderoth, J. (2000). Solving large quadratic assignment problems on...
  • G.V. Balaji et al.

    Application of interval-Newton method to chemical engineering problems

    AIChE Symposium Series

    (1995)
  • Y. Bard

    Nonlinear parameter estimation

    (1974)
  • S. Berner et al.

    A parallel algorithm for the global optimization of Gibbs free energy

    Annals of Operations Research

    (1999)
  • Blumofe, R.D., & Leiserson, C.E. (1994). Scheduling multithreaded computations by work stealing. In: S. Goldwasser...
  • R.D. Blumofe et al.

    Scheduling multithreaded computations by work stealing

    Journal of the ACM

    (1999)
  • R.P. Byrne et al.

    Global optimization of modular process flowsheets

    Industrial and Engineering Chemistry Research

    (2000)
  • J. Clausen et al.

    Implementations of parallel branch-and-bound algorithms-experience with the graph partitioning problem

    Annals of Operations Research

    (1991)
  • R. Correa et al.

    Parallel best-first branch-and-bound in discrete optimization: a framework

  • G. Cybenko

    Dynamic load balancing for distributed memory multiprocessors

    Journal of Parallel and Distributed Computing

    (1989)
  • D.L. Eager et al.

    Adapative load sharing in homogeneous distributed systems

    IEEE Transactions on Software Engineering

    (1986)
  • Epperly, T.G.W. (1995). PhD thesis, University of Wisconsin, Madison,...
  • W.R. Esposito et al.

    Global optimization in parameter estimation of nonlinear algebraic models via the error-in-variables approach

    Industrial and Engineering Chemistrty Research

    (1998)
  • Cited by (14)

    • Parallelization of B&BT Algorithms

      2019, Studies in Computational Intelligence
    • Branch-and-bound-type methods

      2019, Studies in Computational Intelligence
    View all citing articles on Scopus
    1

    Present address: LINDO Systems, Inc., 1415 North Dayton Street, Chicago, IL 60622, USA.

    View full text