Skip to main content
SpringerLink
Log in

A data-driven and model-based accelerated Hamiltonian Monte Carlo method for Bayesian elliptic inverse problems

  • Original Paper
  • Published:
Statistics and Computing Aims and scope Submit manuscript

Abstract

In this paper, we consider a Bayesian inverse problem modeled by elliptic partial differential equations (PDEs). Specifically, we propose a data-driven and model-based approach to accelerate the Hamiltonian Monte Carlo (HMC) method in solving large-scale Bayesian inverse problems. The key idea is to exploit (model-based) and construct (data-based) intrinsic approximate low-dimensional structure of the underlying problem which consists of two components—a training component that computes a set of data-driven basis to achieve significant dimension reduction in the solution space, and a fast solving component that computes the solution and its derivatives for a newly sampled elliptic PDE with the constructed data-driven basis. Hence we develop an effective data and model-based approach for the Bayesian inverse problem and overcome the typical computational bottleneck of HMC—repeated evaluation of the Hamiltonian involving the solution (and its derivatives) modeled by a complex system, a multiscale elliptic PDE in our case. Finally, we present numerical examples to demonstrate the accuracy and efficiency of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability and materials

The data generated and analysed during the current study are available from the corresponding author on reasonable request.

Code availability

The Python codes are published on GitHub.

Notes

  1. https://github.com/LSijing/Bayesian-pde-inverse-problem.

References

  • Abdulle, A., Barth, A., Schwab, C.: Multilevel Monte Carlo methods for stochastic elliptic multiscale PDEs. Multiscale Model. Simul. 11, 1033–1070 (2013)

    MathSciNet  MATH  Google Scholar 

  • Asokan, B.V., Zabaras, N.: A stochastic variational multiscale method for diffusion in heterogeneous random media. J. Comput. Phys. 218, 654–676 (2006)

    MathSciNet  MATH  Google Scholar 

  • Babuska, I., Nobile, F., Tempone, R.: A stochastic collocation method for elliptic partial differential equations with random input data. SIAM J. Numer. Anal. 45, 1005–1034 (2007)

    MathSciNet  MATH  Google Scholar 

  • Babuska, I., Tempone, R., Zouraris, G.: Galerkin finite element approximations of stochastic elliptic partial differential equations. SIAM J. Numer. Anal. 42, 800–825 (2004)

    MathSciNet  MATH  Google Scholar 

  • Bachmayr, M., Cohen, A., DeVore, R., Migliorati, G.: Sparse polynomial approximation of parametric elliptic PDEs, Part II: lognormal coefficients. ESAIM: Math. Model. Numer. Anal. 51, 341–363 (2017)

    MathSciNet  MATH  Google Scholar 

  • Barrault, M., Maday, Y., Nguyen, N.C., Patera, A.T.: An empirical interpolation method: application to efficient reduced-basis discretization of partial differential equations. C.R. Math. 339(9), 667–672 (2004)

    MathSciNet  MATH  Google Scholar 

  • Benner, P., Gugercin, S., Willcox, K.: A survey of projection-based model reduction methods for parametric dynamical systems. SIAM Rev. 57, 483–531 (2015)

    MathSciNet  MATH  Google Scholar 

  • Berkooz, G., Holmes, P., Lumley, J.L.: The proper orthogonal decomposition in the analysis of turbulent flows. Annu. Rev. Fluid Mech. 25(1), 539–575 (1993)

    MathSciNet  Google Scholar 

  • Beskos, A., Jasra, A., Muzaffer, E., Stuart, A.: Sequential Monte Carlo methods for Bayesian elliptic inverse problems. Stat. Comput. 25, 727–737 (2015)

    MathSciNet  MATH  Google Scholar 

  • Bryson, J., Zhao, H., Zhong, Y.: Intrinsic complexity and scaling laws: from random fields to random vectors. SIAM Multiscale Model. Simul. 17(1), 460–481 (2019)

    MathSciNet  MATH  Google Scholar 

  • Chung, E., Efendiev, Y., Leung, W., Zhang, Z.: Cluster-based generalized multiscale finite element method for elliptic PDEs with random coefficients. J. Comput. Phys. 371, 606–617 (2018)

    MathSciNet  MATH  Google Scholar 

  • Cohen, A., DeVore, R.: Approximation of high-dimensional parametric PDEs. Acta Numer. 24, 1–159 (2015)

    MathSciNet  MATH  Google Scholar 

  • Dashti, M., Stuart, A.: Uncertainty quantification and weak approximation of an elliptic inverse problem. SIAM J. Numer. Anal. 49, 2524–2542 (2011)

    MathSciNet  MATH  Google Scholar 

  • Duane, S., Kennedy, A.D., Pendleton, B.J., Roweth, D.: Hybrid Monte Carlo. Phys. Lett. B 195, 216–222 (1987)

    MathSciNet  Google Scholar 

  • Efendiev, Y., Kronsbein, C., Legoll, F.: Multilevel Monte Carlo approaches for numerical homogenization. Multiscale Model. Simul. 13, 1107–1135 (2015)

    MathSciNet  MATH  Google Scholar 

  • Ghanem, R., Spanos, P.: Stochastic Finite Elements: A Spectral Approach. Springer-Verlag, New York (1991)

    MATH  Google Scholar 

  • Girolami, M., Calderhead, B.: Riemann manifold Langevin and Hamiltonian Monte Carlo methods. J. R. Stat. Soc.: Series B (Stat. Methodol.) 73(2), 123–214 (2011)

    MathSciNet  MATH  Google Scholar 

  • Givoli, D.: A tutorial on the adjoint method for inverse problems. Comput. Methods Appl. Mech. Eng. 380, 113810 (2021)

    MathSciNet  MATH  Google Scholar 

  • Graham, I., Kuo, F., Nuyens, D., Scheichl, R., Sloan, I.: Quasi-Monte Carlo methods for elliptic PDEs with random coefficients and applications. J. Comput. Phys. 230, 3668–3694 (2011)

    MathSciNet  MATH  Google Scholar 

  • Graham, I.G., Kuo, F.Y., Nichols, J.A., Scheichl, R., Schwab, C., Sloan, I.H.: Quasi-Monte Carlo finite element methods for elliptic PDEs with lognormal random coefficients. Numer. Math. 131(2), 329–368 (2015)

    MathSciNet  MATH  Google Scholar 

  • Halko, N., Martinsson, P., Tropp, J.: Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53, 217–288 (2011)

    MathSciNet  MATH  Google Scholar 

  • He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778 (2016)

  • Hoang, V., Schwab, C.: N-term wiener chaos approximation rates for elliptic PDEs with lognormal Gaussian random inputs. Math. Models Methods Appl. Sci. 24, 797–826 (2014)

    MathSciNet  MATH  Google Scholar 

  • Holmes, P., Lumley, J., Berkooz, G.: Turbulence, Coherent Structures, Dynamical Systems and Symmetry. Cambridge University Press, Cambridge (1998)

    MATH  Google Scholar 

  • Hou, T., Liu, P., Zhang, Z.: A localized data-driven stochastic method for elliptic PDEs with random coefficients. Bull. Inst. Math. Acad. Sin. 1, 179–216 (2016)

    Google Scholar 

  • Hou, T., Ma, D., Zhang, Z.: A model reduction method for multiscale elliptic PDEs with random coefficients using an optimization approach. Multiscale Model. Simul. 17, 826–853 (2019)

    MathSciNet  MATH  Google Scholar 

  • Kaipio, J., Somersalo, E.: Statistical and Computational Inverse Problems, vol. 160. Springer, New York (2005)

    MATH  Google Scholar 

  • Kingma, D. P., Ba, J.: Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, (2014)

  • Lan, S.: Adaptive dimension reduction to accelerate infinite-dimensional geometric Markov Chain Monte Carlo. J. Comput. Phys. 392, 71–95 (2019)

    MathSciNet  MATH  Google Scholar 

  • Lan, S., Bui-Thanh, T., Christie, M., Girolami, M.: Emulation of higher-order tensors in mainifold Monte Carlo methods for Bayesian inverse problems. J. Comput. Phys. 308, 81–101 (2016)

    MathSciNet  MATH  Google Scholar 

  • Li, S., Zhang, Z., Zhao, H.: A data-driven approach for multiscale elliptic PDEs with random coefficients based on intrinsic dimension reduction. SIAM J. Multiscale Model. Simul. 18(3), 1242–1271 (2020)

    MathSciNet  MATH  Google Scholar 

  • Martin, J., Wilcox, L., Burstedde, C., Ghattas, O.: A stochastic Newton MCMC method for large-scale statistical inverse problems with application to seismic inversion. SIAM J. Sci. Comput. 34, A1460–A1487 (2012)

    MathSciNet  MATH  Google Scholar 

  • Mondal, A., Efendiev, Y., Mallick, B., Datta-Gupta, A.: Bayesian uncertainty quantification for flows in heterogeneous porous media using reversible jump Markov chain Monte Carlo methods. Adv. Water Resour. 33, 241–256 (2010)

    Google Scholar 

  • Natterer, F.: Adjoint methods as applied to inverse problems. Encyclopedia Appl. Comput. Math. 1, 33–36 (2015)

    Google Scholar 

  • Neal, R.M.: MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo 2, 2 (2011)

    MATH  Google Scholar 

  • Nobile, F., Tempone, R., Webster, C.: A sparse grid stochastic collocation method for partial differential equations with random input data. SIAM J. Numer. Anal. 46, 2309–2345 (2008)

    MathSciNet  MATH  Google Scholar 

  • Schwab, C., Todor, R.A.: Karhunen loève approximation of random fields by generalized fast multipole methods. J. Comput. Phys. 217(1), 100–122 (2006)

    MathSciNet  MATH  Google Scholar 

  • Sirovich, L.: Turbulence and the dynamics of coherent structures. I. Coherent structures. Q. Appl. Math. 45(3), 561–571 (1987)

    MathSciNet  MATH  Google Scholar 

  • Strathmann, H., Sejdinovic, D., Livingstone, S., Szabo, Z., Gretton, A.: Gradient-free Hamiltonian Monte Carlo with efficient kernel exponential families. In: Advances in Neural Information Processing Systems, pp 955–963 (2015)

  • Stuart, A.: Inverse problems: a Bayesian perspective. Acta Numer. 19, 451–559 (2010)

    MathSciNet  MATH  Google Scholar 

  • Wald, I., Havran, V.: On building fast kd-trees for ray tracing, and on doing that in O (N log N). In: 2006 IEEE Symposium on Interactive Ray Tracing, IEEE, pp 61–69 (2006)

  • Wan, J., Zabaras, N.: A probabilistic graphical model approach to stochastic multiscale partial differential equations. J. Comput. Phys. 250, 477–510 (2013)

    MathSciNet  MATH  Google Scholar 

  • Xiu, D., Karniadakis, G.: Modeling uncertainty in flow simulations via generalized polynomial chaos. J. Comput. Phys. 187, 137–167 (2003)

    MathSciNet  MATH  Google Scholar 

  • Zhang, C., Shahbaba, B., Zhao, H.: Hamiltonian Monte Carlo acceleration using surrogate functions with random bases. Stat. Comput. 27(6), 1473–1490 (2017)

    MathSciNet  MATH  Google Scholar 

  • Zhang, C., Shahbaba, B., Zhao, H.: Precomputing strategy for Hamiltonian Monte Carlo method base on regularity in parameter space. Comput. Stat. 32, 253–279 (2017)

    MATH  Google Scholar 

  • Zhang, C., Shahbaba, B., Zhao, H.: Variational Hamiltonian Monte Carlo via score matching. Bayesian Anal. 13(2), 485–506 (2018)

    MathSciNet  MATH  Google Scholar 

  • Zhang, Z., Ci, M., Hou, T.Y.: A multiscale data-driven stochastic method for elliptic PDEs with random coefficients. SIAM Multiscale Model. Simul. 13, 173–204 (2015)

    MathSciNet  Google Scholar 

Download references

Acknowledgements

The research of S. Li is partially supported by the Doris Chen Postgraduate Scholarship. The research of C. Zhang is partially supported by National Natural Science Foundation of China (projects 12201014 and 12292983), the Key Laboratory of Mathematics and Its Applications (LMAM) and the Key Laboratory of Mathematical Economics and Quantitative Finance (LMEQF) of Peking University. The research of Z. Zhang is supported by the Hong Kong RGC General Research Fund projects 17300318 and 17307921, National Natural Science Foundation of China (project 12171406), an R &D Funding Scheme from the HKU-SCF FinTech Academy, Seed Funding Programme for Basic Research (HKU), and the outstanding young researcher award of HKU (2020–21). The research of H. Zhao is partially supported by NSF grants DMS-2048877 and DMS-2012860. The computations were performed using research computing facilities offered by Information Technology Services, the University of Hong Kong.

Author information

Authors and Affiliations

  1. Department of Mathematics, The University of Hong Kong, Pokfulam Road, Hong Kong SAR, China

    Sijing Li & Zhiwen Zhang

  2. School of Mathematical Sciences and Center for Statistical Science, Peking University, Beijing, China

    Cheng Zhang

  3. Department of Mathematics, Duke University, Durham, NC, 27708, USA

    Hongkai Zhao

Authors
  1. Sijing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cheng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhiwen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hongkai Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Cheng Zhang or Zhiwen Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, S., Zhang, C., Zhang, Z. et al. A data-driven and model-based accelerated Hamiltonian Monte Carlo method for Bayesian elliptic inverse problems. Stat Comput 33, 90 (2023). https://doi.org/10.1007/s11222-023-10262-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11222-023-10262-y

Keywords

Mathematics subject classification

Search

Navigation