Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Expert Recommendation
  • Published:

How to verify the precision of density-functional-theory implementations via reproducible and universal workflows

Nature Reviews Physics volume 6, pages 45–58 (2024)Cite this article

Subjects

Abstract

Density-functional theory methods and codes adopting periodic boundary conditions are extensively used in condensed matter physics and materials science research. In 2016, their precision (how well properties computed with different codes agree among each other) was systematically assessed on elemental crystals: a first crucial step to evaluate the reliability of such computations. In this Expert Recommendation, we discuss recommendations for verification studies aiming at further testing precision and transferability of density-functional-theory computational approaches and codes. We illustrate such recommendations using a greatly expanded protocol covering the whole periodic table from Z = 1 to 96 and characterizing 10 prototypical cubic compounds for each element: four unaries and six oxides, spanning a wide range of coordination numbers and oxidation states. The primary outcome is a reference dataset of 960 equations of state cross-checked between two all-electron codes, then used to verify and improve nine pseudopotential-based approaches. Finally, we discuss the extent to which the current results for total energies can be reused for different goals.

Key points

  • Verification efforts are critical to assess the reliability of density-functional theory (DFT) simulations and provide results with properly quantified uncertainties.

  • Developing standard computation protocols to perform verification studies and publishing curated and FAIR reference datasets can greatly aid their use to improve codes and computational approaches.

  • The use of fully automated workflows with common interfaces between codes can guarantee uniformity, transferability and reproducibility of results.

  • A careful description of the numerical and methodological details needed to compare with the reference datasets is essential; we discuss and illustrate this point with a dataset of 960 all-electron equations of state.

  • Reference datasets should always include an explanation of the target property for which they were generated, and a discussion of their limits of applicability.

  • Further extensions of DFT verification efforts are needed to cover more functionals, more computational approaches and the treatment of magnetic and relativistic (spin–orbit) effects. They should also aim at concurrently delivering optimized protocols that not only target ultimate precision, but also optimize the computational cost for a target accuracy.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Differences in results for codes FLEUR and WIEN2k.
Fig. 2: Discrepancy between the all-electron results obtained with WIEN2k and FLEUR in our reference dataset.
Fig. 3: Effect of different choices of smearing on the equation of state of the artificial diamond structure of erbium computed with the Quantum ESPRESSO code.
Fig. 4: Comparing the results of different computational approaches.

Similar content being viewed by others

Code availability

The source code of the common workflows is released under the MIT open-source licence and is made available on GitHub (https://github.com/aiidateam/aiida-common-workflows). It is also distributed as an installable package through the Python Package Index (https://pypi.org/project/aiida-common-workflows). The source code of the scripts to generate the plots is released under the MIT open-source licence and is made available on GitHub (https://github.com/aiidateam/acwf-verification-scripts). All codes to generate the figures of this paper are available in the data entry of ref. 62.

Data availability

The data and the scripts used to create all the images in this work are available on the Materials Cloud Archive62. Moreover, the data are accessible via an interactive website, https://acwf-verification.materialscloud.org, that offers various analysis and visualization possibilities. Note that the data include the entire AiiDA provenance graph of each workflow execution presented in the main text (therefore including all input files and output files of all simulations, as well as their logical relationship, in AiiDA format), as well as the curated data extracted from that database to produce the images.

References

  1. Alberi, K. et al. The 2019 materials by design roadmap. J. Phys. D 52, 013001 (2018).

    ADS  Google Scholar 

  2. Marzari, N., Ferretti, A. & Wolverton, C. Electronic-structure methods for materials design. Nat. Mater. 20, 736–749 (2021).

    ADS  Google Scholar 

  3. Pizzi, G., Cepellotti, A., Sabatini, R., Marzari, N. & Kozinsky, B. AiiDA: automated interactive infrastructure and database for computational science. Comput. Mater. Sci. 111, 218–230 (2016).

    Google Scholar 

  4. Huber, S. P. et al. AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance. Sci. Data 7, 300 (2020).

    Google Scholar 

  5. Ong, S. P. et al. Python Materials Genomics (pymatgen): a robust, open-source Python library for materials analysis. Comput. Mater. Sci. 68, 314–319 (2013).

    Google Scholar 

  6. Jain, A. et al. Fireworks: a dynamic workflow system designed for high-throughput applications. Concurr. Comput. Pract. Exp. 27, 5037–5059 (2015).

    Google Scholar 

  7. Mathew, K. et al. Atomate: a high-level interface to generate, execute, and analyze computational materials science workflows. Comput. Mater. Sci. 139, 140–152 (2017).

    ADS  Google Scholar 

  8. Bahn, S. R. & Jacobsen, K. W. An object-oriented scripting interface to a legacy electronic structure code. Comput. Sci. Eng. 4, 56–66 (2002).

    Google Scholar 

  9. Hjorth Larsen, A. et al. The atomic simulation environment — a Python library for working with atoms. J. Phys. Condens. Matter 29, 273002 (2017).

    Google Scholar 

  10. Curtarolo, S. et al. Aflow: an automatic framework for high-throughput materials discovery. Comput. Mater. Sci. 58, 218–226 (2012).

    Google Scholar 

  11. Janssen, J. et al. pyiron: an integrated development environment for computational materials science. Comput. Mater. Sci. 163, 24 – 36 (2019).

    Google Scholar 

  12. Armiento, R. Database-driven high-throughput calculations and machine learning models for materials design. In Machine Learning Meets Quantum Physics (eds Schütt, K. et al.) Ch. 17, 377–395 (Springer, 2020).

  13. Gonze, X. et al. The ABINIT project: impact, environment and recent developments. Comput. Phys. Commun. 248, 107042 (2020).

    Google Scholar 

  14. Jain, A. et al. Commentary: The Materials Project: a materials genome approach to accelerating materials innovation. APL Mater. 1, 011002 (2013).

    ADS  Google Scholar 

  15. Landis, D. D. et al. The Computational Materials Repository. Comput. Sci. Eng. 14, 51–57 (2012).

    Google Scholar 

  16. Kirklin, S. et al. The Open Quantum Materials Database (OQMD): assessing the accuracy of DFT formation energies. NPJ Comput. Mater. 1, 15010 (2015).

    ADS  Google Scholar 

  17. Merkys, A. et al. A posteriori metadata from automated provenance tracking: integration of AiiDA and TCOD. J. Cheminform. 9, 56 (2017).

    Google Scholar 

  18. Stevanović, V., Lany, S., Zhang, X. & Zunger, A. Correcting density functional theory for accurate predictions of compound enthalpies of formation: fitted elemental-phase reference energies. Phys. Rev. B 85, 115104 (2012).

    ADS  Google Scholar 

  19. Talirz, L. et al. Materials Cloud, a platform for open computational science. Sci. Data 7, 299 (2020).

    Google Scholar 

  20. Draxl, C. & Scheffler, M. NOMAD: the FAIR concept for big data-driven materials science. MRS Bull. 43, 676–682 (2018).

    Google Scholar 

  21. Curtarolo, S. et al. AFLOWLIB.ORG: a distributed materials properties repository from high-throughput ab initio calculations. Comput. Mater. Sci. 58, 227–235 (2012).

    Google Scholar 

  22. Wilkinson, M. D. et al. The FAIR guiding principles for scientific data management and stewardship. Sci. Data 3, 1–9 (2016).

    Google Scholar 

  23. Andersen, C. W. et al. OPTIMADE, an API for exchanging materials data. Sci. Data 8, 217 (2021).

    Google Scholar 

  24. IEEE Standard for System, Software, and Hardware Verification and Validation. IEEE Std 1012-2016 (Revision of IEEE Std 1012-2012/ Incorporates IEEE Std 1012-2016/Cor1-2017) 1–260, https://doi.org/10.1109/IEEESTD.2017.8055462 (2017).

  25. Wang, A. et al. A framework for quantifying uncertainty in DFT energy corrections. Sci. Rep. 11, 15496 (2021).

    Google Scholar 

  26. Carbogno, C. et al. Numerical quality control for DFT-based materials databases. NPJ Comput. Mater. 8, 69 (2022).

    ADS  Google Scholar 

  27. Poncé, S. et al. Verification of first-principles codes: comparison of total energies, phonon frequencies, electron–phonon coupling and zero-point motion correction to the gap between ABINIT and QE/Yambo. Comput. Mater. Sci. 83, 341–348 (2014).

    Google Scholar 

  28. Pople, J. Nobel Lecture: Quantum Chemical Models. https://www.nobelprize.org/prizes/chemistry/1998/pople/lecture/ (1998).

  29. Lejaeghere, K. et al. Reproducibility in density functional theory calculations of solids. Science 351, aad3000 (2016).

    Google Scholar 

  30. Cottenier, S., Delta project — archive of old website. Materials Cloud Archive 2023.133, https://doi.org/10.24435/materialscloud:5e-mv (2023).

  31. The FLEUR project. https://www.flapw.de/.

  32. Wortmann, D. et al. Fleur. https://doi.org/10.5281/zenodo.7576163 (2023).

  33. Blaha, P. et al. WIEN2k: an augmented plane wave plus local orbitals program for calculating crystal properties. http://www.wien2k.at/.

  34. Blaha, P. et al. WIEN2k: an apw+lo program for calculating the properties of solids. J. Chem. Phys. 152, 074101 (2020).

    ADS  Google Scholar 

  35. Uhrin, M., Huber, S. P., Yu, J., Marzari, N. & Pizzi, G. Workflows in AiiDA: engineering a high-throughput, event-based engine for robust and modular computational workflows. Comput. Mater. Sci. 187, 110086 (2021).

    Google Scholar 

  36. Huber, S. P. et al. Common workflows for computing material properties using different quantum engines. NPJ Comput. Mater. 7, 136 (2021).

    ADS  Google Scholar 

  37. Huber, S. P. et al. AiiDA common workflows (ACWF) package, versions 1.0.1 and above (2023); https://github.com/aiidateam/aiida-common-workflows.

  38. Lejaeghere, K., Speybroeck, V. V., Oost, G. V. & Cottenier, S. Error estimates for solid-state density-functional theory predictions: an overview by means of the ground-state elemental crystals. Crit. Rev. Solid State Mater. Sci. 39, 1–24 (2013).

    ADS  Google Scholar 

  39. Grosso, G. & Pastori Parravicini, G. Solid State Physics 2nd edn (Academic, 2013).

  40. Martin, R. M.Electronic Structure: Basic Theory and Practical Methods 2nd edn (Cambridge Univ. Press, 2013).

  41. Cohen, M. L. & Louie, S. G. Fundamentals of Condensed Matter Physics (Cambridge Univ. Press, 2016).

  42. Gonze, X. et al. Recent developments in the ABINIT software package. Comput. Phys. Commun. 205, 106–131 (2016).

    ADS  Google Scholar 

  43. Romero, A. H. et al. ABINIT: overview and focus on selected capabilities. J. Chem. Phys. 152, 124102 (2020).

    ADS  Google Scholar 

  44. Ratcliff, L. E. et al. Flexibilities of wavelets as a computational basis set for large-scale electronic structure calculations. J. Chem. Phys. 152, 194110 (2020).

    ADS  Google Scholar 

  45. Clark, S. J. et al. First principles methods using CASTEP. Z. Kristallogr. Cryst. Mater. 220, 567–570 (2005).

    Google Scholar 

  46. The CP2K simulation package. https://www.cp2k.org.

  47. Kühne, T. D. et al. CP2K: an electronic structure and molecular dynamics software package — Quickstep: efficient and accurate electronic structure calculations. J. Chem. Phys. 152, 194103 (2020).

    ADS  Google Scholar 

  48. Mortensen, J. J., Hansen, L. B. & Jacobsen, K. W. Real-space grid implementation of the projector augmented wave method. Phys. Rev. B 71, 035109 (2005).

    ADS  Google Scholar 

  49. Enkovaara, J. et al. Electronic structure calculations with GPAW: a real-space implementation of the projector augmented-wave method. J. Phys. Condens. Matter 22, 253202 (2010).

    ADS  Google Scholar 

  50. Giannozzi, P. et al. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. J. Phys. Condens. Matter 21, 395502 (2009).

    Google Scholar 

  51. Giannozzi, P. et al. Advanced capabilities for materials modelling with QUANTUM ESPRESSO. J. Phys. Condens. Matter 29, 465901 (2017).

    Google Scholar 

  52. Soler, J. M. et al. The SIESTA method for ab initio order-N materials simulation. J. Phys. Condens. Matter 14, 2745–2779 (2002).

    ADS  Google Scholar 

  53. García, A. et al. SIESTA: recent developments and applications. J. Chem. Phys. 152, 204108 (2020).

    ADS  Google Scholar 

  54. The SIRIUS domain-specific library for electronic-structure calculations. https://github.com/electronic-structure/SIRIUS.

  55. Kresse, G. & Furthmüller, J. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set. Phys. Rev. B 54, 11169–11186 (1996).

    ADS  Google Scholar 

  56. Kresse, G. & Joubert, D. From ultrasoft pseudopotentials to the projector augmented-wave method. Phys. Rev. B 59, 1758–1775 (1999).

    ADS  Google Scholar 

  57. Perdew, J. P., Burke, K. & Ernzerhof, M. Generalized gradient approximation made simple. Phys. Rev. Lett. 77, 3865–3868 (1996).

    ADS  Google Scholar 

  58. Vanpoucke, D. E. P., Bultinck, P. & Van Driessche, I. Extending Hirshfeld-I to bulk and periodic materials. J. Comput. Chem. 34, 405–417 (2013).

    Google Scholar 

  59. Vanpoucke, D. E. P., Van Driessche, I. & Bultinck, P. Reply to “Comment on ‘Extending Hirshfeld-I to bulk and periodic materials’”. J. Comput. Chem. 34, 422–427 (2013).

    Google Scholar 

  60. Bultinck, P., Van Alsenoy, C., Ayers, P. W. & Carbó-Dorca, R. Critical analysis and extension of the Hirshfeld atoms in molecules. J. Chem. Phys. 126, 144111 (2007).

    ADS  Google Scholar 

  61. Birch, F. Finite elastic strain of cubic crystals. Phys. Rev. 71, 809–824 (1947).

    ADS  Google Scholar 

  62. Bosoni, E. et al. How to verify the precision of density-functional-theory implementations via reproducible and universal workflows. Materials Cloud Archive 2023.81, https://doi.org/10.24435/materialscloud:s4-3h (2023).

  63. Jollet, F., Torrent, M. & Holzwarth, N. Generation of projector augmented-wave atomic data: a 71 element validated table in the XML format. Comput. Phys. Commun. 185, 1246–1254 (2014).

    ADS  Google Scholar 

  64. Glantz, S. A., Slinker, B. K. & Neilands, T. B. in Primer of Applied Regression and Analysis of Variance 3rd edition, 261–298 (McGraw-Hill Education, 2017).

  65. van Setten, M. et al. The PseudoDojo: training and grading a 85 element optimized norm-conserving pseudopotential table. Comput. Phys. Commun. 226, 39–54 (2018).

    ADS  Google Scholar 

  66. The PseudoDojo website. http://www.pseudo-dojo.org/.

  67. Hartwigsen, C., Goedecker, S. & Hutter, J. Relativistic separable dual-space Gaussian pseudopotentials from H to Rn. Phys. Rev. B 58, 3641–3662 (1998).

    ADS  Google Scholar 

  68. Goedecker, S., Teter, M. & Hutter, J. Separable dual-space Gaussian pseudopotentials. Phys. Rev. B 54, 1703 (1996).

    ADS  Google Scholar 

  69. Krack, M. Pseudopotentials for H to Kr optimized for gradient-corrected exchange-correlation functionals. Theor. Chem. Acc. 114, 145–152 (2005).

    Google Scholar 

  70. VandeVondele, J. & Hutter, J. Gaussian basis sets for accurate calculations on molecular systems in gas and condensed phases. J. Chem. Phys. 127, 114105 (2007).

    ADS  Google Scholar 

  71. GPAW atomic PAW setups. https://wiki.fysik.dtu.dk/gpaw/setups/setups.html#atomic-paw-setups.

  72. Prandini, G., Marrazzo, A., Castelli, I. E., Mounet, N. & Marzari, N. Precision and efficiency in solid-state pseudopotential calculations. NPJ Comput. Mater. 4, 72 (2018).

    ADS  Google Scholar 

  73. Prandini, G. et al. A standard solid state pseudopotentials (SSSP) library optimized for precision and efficiency. Materials Cloud Archive 2023.65 Version v11, https://doi.org/10.24435/materialscloud:f3-ym (2023).

  74. García, A., Verstraete, M. J., Pouillon, Y. N. & Junquera, J. The PSML format and library for norm-conserving pseudopotential data curation and interoperability. Comput. Phys. Commun. 227, 51–71 (2018).

    ADS  Google Scholar 

  75. Topsakal, M. & Wentzcovitch, R. Accurate projected augmented wave (PAW) datasets for rare-earth elements (Re = La–Lu). Comput. Mater. Sci. 95, 263–270 (2014).

    Google Scholar 

  76. Prandini, G. et al. A standard solid state pseudopotentials (SSSP) library optimized for precision and efficiency. Materials Cloud Archive 2021.76 Version v7, https://doi.org/10.24435/materialscloud:rz-77 (2021).

  77. Prandini, G. et al. A standard solid state pseudopotentials (SSSP) library optimized for precision and efficiency. Materials Cloud Archive 2022.159 Version v8, https://doi.org/10.24435/materialscloud:3v-xt (2022).

  78. Sachs, M. et al. DFT-guided crystal structure redetermination and lattice dynamics of the intermetallic actinoid compound UIr. Inorg. Chem. 60, 16686–16699 (2021).

    Google Scholar 

  79. Dal Corso, A. Pseudopotentials periodic table: from H to Pu. Comput. Mater. Sci. 95, 337–350 (2014).

    Google Scholar 

  80. Thörnig, P. JURECA: data centric and booster modules implementing the modular supercomputing architecture at Jülich supercomputing centre. JLSRF 7, 182 (2021).

    Google Scholar 

  81. Marzari, N., Vanderbilt, D., De Vita, A. & Payne, M. C. Thermal contraction and disordering of the Al(110) surface. Phys. Rev. Lett. 82, 3296–3299 (1999).

    ADS  Google Scholar 

  82. dos Santos, F. J. & Marzari, N. Fermi energy determination for advanced smearing techniques. Phys. Rev. B 107, 195122 (2023).

    ADS  Google Scholar 

  83. Gillan, M. J. Calculation of the vacancy formation energy in aluminium. J. Phys. Condens. Matter 1, 689 (1989).

    ADS  Google Scholar 

Download references

Acknowledgements

This work was inspired and is supported in part by the European Union (EU)’s Horizon 2020 research and innovation programme under grant agreements no. 676598 and no. 824143 (European MaX Centre of Excellence ‘Materials Design at the Exascale’) and by NCCR MARVEL, a National Centre of Competence in Research, funded by the Swiss National Science Foundation (SNSF, grant no. 205602). For the purpose of Open Access, a CC BY public copyright licence is applied to any Author Accepted Manuscript (AAM) version arising from this submission. We thank F. J. dos Santos for discussions on the analysis of the smearing types and k-point convergence, and X. Gonze, M. Torrent and F. Jollet for discussions on PAW pseudopotentials. M.F. and N.M. acknowledge the contribution of S. Shankar in early discussions about the use of prototype oxides as general platform to explore the transferability of pseudopotentials. Work at ICMAB (E.B., A.G., V.D.) is supported by the Severo Ochoa Centers of Excellence Program (MCIN CEX2019-000917-S), by grant PGC2018-096955-B-C44 of MCIN/AEI/10.13039/501100011033, ‘ERDF A way of making Europe’, and by GenCat 2017SGR1506. We also thank the Barcelona Supercomputer Center (BSC) for computational resources. V.D. acknowledges support from DOC-FAM, EU Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement no. 754397. O.R. acknowledges travel support from WIEN2k (Technical University of Vienna). The Jülich team (S.B., J.B., H.J., G.M., D.W.) acknowledge support by the Joint Lab Virtual Materials Design of the Forschungszentrum Jülich, the Helmholtz Platform for Research Software Engineering — Preparatory Study, the Joint Virtual Laboratory AI, Data Analytics and Scalable Simulation of the Forschungszentrum Jülich and the French Alternative Energies and Atomic Energy Commission, and the computing time granted through JARA on the supercomputers JURECA80 at Forschungszentrum Jülich and CLAIX at RWTH Aachen University. H.M. and T.D.K. (Univ. Paderborn) acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this project by providing computing time on the GCS supercomputer JUWELS at Jülich Supercomputing Centre. S.P. and G.-M.R. (Univ. catholique de Louvain) acknowledge support from the Fonds de la Recherche Scientifique de Belgique (F.R.S.-FNRS). Computational resources have been provided by the PRACE-21 resources MareNostrum at the BSC-CNS and by the Consortium des Équipements de Calcul Intensif, funded by the F.R.S.-FNRS under grant no. 2.5020.11 and by the Walloon Region as well as computational resources awarded on the Belgian share of the EuroHPC LUMI supercomputer. G.Ka. and S.V. received funding from the VILLUM Centre for the Science of Sustainable Fuels and Chemicals (9455) from VILLUM FONDEN. Computational resources were provided by the Niflheim supercomputing cluster at the Technical University of Denmark. They also thank J. J. Mortensen and A. H. Larsen for discussions on optimizing the workflow for the GPAW code. S.C. acknowledges financial support from OCAS NV by an OCAS-endowed chair at Ghent University. The computational resources and services used at Ghent University were provided by the Vienna Scientific Cluster (VSC; Flemish Supercomputer Center), funded by the Research Foundation Flanders and the Flemish Government department EWI. M.W. acknowledges computational resources provided by the VSC. This research was funded in part by the Austrian Science Fund (FWF) [P 32711]. E.F.L. acknowledges resources provided by Sigma2 — the National Infrastructure for High Performance Computing and Data Storage in Norway, and support from the Norwegian Research Infrastructure Services. B.Z. is grateful to the UK Materials and Molecular Modelling Hub for computational resources, which is partially funded by EPSRC (EP/P020194/1 and EP/T022213/1), and acknowledges the use of the UCL Myriad and Kathleen High Performance Computing Facility (Myriad@UCL, Kathleen@UCL), and associated support services, in the completion of this work. N.M., G.Pi. and A.G. acknowledge support from the EU Horizon 2020 research and innovation programme under grant agreement no. 957189 (BIG-MAP), also part of the BATTERY 2030+ initiative under grant agreement no. 957213. G.P., J.Y. and G.-M.R. acknowledge support by the SNSF and by the F.R.S.-FNRS through the ‘FISH4DIET’ Project (SNSF grant 200021E_206190 and F.R.S.-FNRS grant T.0179.22). G.P. acknowledges support by the Open Research Data Program of the ETH Board, under the Establish project ‘PREMISE’. J.Y. acknowledges support from the EU Horizon 2020 research and innovation programme under grant agreement no. 760173 (MARKETPLACE).

Author information

Authors and Affiliations

  1. Institut de Ciència de Materials de Barcelona, ICMAB-CSIC, Bellaterra, Spain

    Emanuele Bosoni, Vladimir Dikan & Alberto Garcia

  2. Université Grenoble-Alpes, CEA, IRIG-MEM-L_Sim, Grenoble, France

    Louis Beal, Augustin Degomme & Luigi Genovese

  3. Theory and Simulation of Materials (THEOS) and National Centre for Computational Design and Discovery of Novel Materials (MARVEL), École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland

    Marnik Bercx, Kristjan Eimre, Sebastiaan P. Huber, Nicola Marzari, Jusong Yu, Austin Zadoks & Giovanni Pizzi

  4. Institute for Materials Chemistry, Technical University of Vienna, Vienna, Austria

    Peter Blaha, Georg K. H. Madsen & Thomas Ruh

  5. Peter Grünberg Institut and Institute for Advanced Simulation, Forschungszentrum Jülich and JARA, Jülich, Germany

    Stefan Blügel, Jens Bröder, Henning Janssen, Gregor Michalicek & Daniel Wortmann

  6. Institute for Advanced Simulation, Materials Data Science and Informatics (IAS-9), Forschungszentrum Jülich, Jülich, Germany

    Jens Bröder

  7. Department of Electromechanical, Systems and Metal Engineering, Ghent University, Ghent, Belgium

    Martin Callsen, Stefaan Cottenier, Michael Sluydts & Danny E. P. Vanpoucke

  8. Center for Molecular Modeling (CMM), Ghent University, Ghent, Belgium

    Martin Callsen, Stefaan Cottenier, Kurt Lejaeghere, Thomas Ruh & Michael Sluydts

  9. Institute of Atomic and Molecular Sciences, Academia Sinica, Taipei, Taiwan

    Martin Callsen

  10. Norwegian EuroHPC Competence Center, Sigma2 AS, Trondheim, Norway

    Espen Flage-Larsen

  11. SINTEF Industry, Materials Physics, Oslo, Norway

    Espen Flage-Larsen

  12. Department of Physics and Science of Advanced Materials Program, Central Michigan University, Mount Pleasant, MI, USA

    Marco Fornari

  13. Institut de la Matière Condensée et des Nanosciences (IMCN), Université catholique de Louvain, Louvain-la-Neuve, Belgium

    Matteo Giantomassi, Guido Petretto, Samuel Poncé & Gian-Marco Rignanese

  14. National Centre of Competence in Research (NCCR) Catalysis, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland

    Sebastiaan P. Huber

  15. Center for Catalysis Theory (Cattheory), Department of Physics, Technical University of Denmark (DTU), Kongens Lyngby, Denmark

    Georg Kastlunger & Sudarshan Vijay

  16. Laboratory for Materials Simulations (LMS), Paul Scherrer Institut (PSI), Villigen PSI, Switzerland

    Matthias Krack, Nicola Marzari, Jusong Yu & Giovanni Pizzi

  17. University of Vienna, Faculty of Physics and Center for Computational Materials Science, Vienna, Austria

    Georg Kresse, Martijn Marsman & Michael Wolloch

  18. VASP Software GmbH, Vienna, Austria

    Georg Kresse, Martijn Marsman & Michael Wolloch

  19. Center for Advanced Systems Understanding (CASUS) and Helmholtz-Zentrum Dresden-Rossendorf, Görlitz, Germany

    Thomas D. Kühne

  20. Paderborn Center for Parallel Computing (PC2) and Center for Sustainable Systems Design, University of Paderborn, Paderborn, Germany

    Thomas D. Kühne

  21. OCAS NV/ArcelorMittal Global R&D Gent, Zelzate, Belgium

    Kurt Lejaeghere

  22. Dynamics of Condensed Matter, Chair of Theoretical Chemistry, University of Paderborn, Paderborn, Germany

    Hossein Mirhosseini

  23. HPE HPC EMEA Research Lab, Basel, Switzerland

    Tiziano M. A. Müller

  24. Department of Materials Science and Metallurgy, University of Cambridge, Cambridge, United Kingdom

    Chris J. Pickard

  25. Advanced Institute for Materials Research, Tohoku University, Sendai, Japan

    Chris J. Pickard

  26. Department of Materials Science and Engineering, McMaster University, Hamilton, Ontario, Canada

    Oleg Rubel

  27. Chair of Physical Chemistry, Montanuniversitaet Leoben, Leoben, Austria

    Thomas Ruh

  28. ePotentia, Deurne, Antwerp, Belgium

    Michael Sluydts

  29. Institute for Materials Research (IMO-IMOMEC), UHasselt — Hasselt University, Hasselt, Belgium

    Danny E. P. Vanpoucke

  30. Swiss Federal Laboratories for Materials Science and Technology (Empa), nanotech@surfaces laboratory, Dübendorf, Switzerland

    Aliaksandr V. Yakutovich

  31. Department of Chemistry, University College London, London, UK

    Bonan Zhu

  32. The Faraday Institution, Didcot, UK

    Bonan Zhu

Authors
  1. Emanuele Bosoni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Louis Beal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marnik Bercx
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Peter Blaha
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stefan Blügel
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jens Bröder
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Martin Callsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Stefaan Cottenier
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Augustin Degomme
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Vladimir Dikan
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Kristjan Eimre
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Espen Flage-Larsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Marco Fornari
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Alberto Garcia
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Luigi Genovese
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Matteo Giantomassi
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Sebastiaan P. Huber
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Henning Janssen
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Georg Kastlunger
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Matthias Krack
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Georg Kresse
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Thomas D. Kühne
    View author publications

    You can also search for this author in PubMed Google Scholar

  23. Kurt Lejaeghere
    View author publications

    You can also search for this author in PubMed Google Scholar

  24. Georg K. H. Madsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  25. Martijn Marsman
    View author publications

    You can also search for this author in PubMed Google Scholar

  26. Nicola Marzari
    View author publications

    You can also search for this author in PubMed Google Scholar

  27. Gregor Michalicek
    View author publications

    You can also search for this author in PubMed Google Scholar

  28. Hossein Mirhosseini
    View author publications

    You can also search for this author in PubMed Google Scholar

  29. Tiziano M. A. Müller
    View author publications

    You can also search for this author in PubMed Google Scholar

  30. Guido Petretto
    View author publications

    You can also search for this author in PubMed Google Scholar

  31. Chris J. Pickard
    View author publications

    You can also search for this author in PubMed Google Scholar

  32. Samuel Poncé
    View author publications

    You can also search for this author in PubMed Google Scholar

  33. Gian-Marco Rignanese
    View author publications

    You can also search for this author in PubMed Google Scholar

  34. Oleg Rubel
    View author publications

    You can also search for this author in PubMed Google Scholar

  35. Thomas Ruh
    View author publications

    You can also search for this author in PubMed Google Scholar

  36. Michael Sluydts
    View author publications

    You can also search for this author in PubMed Google Scholar

  37. Danny E. P. Vanpoucke
    View author publications

    You can also search for this author in PubMed Google Scholar

  38. Sudarshan Vijay
    View author publications

    You can also search for this author in PubMed Google Scholar

  39. Michael Wolloch
    View author publications

    You can also search for this author in PubMed Google Scholar

  40. Daniel Wortmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  41. Aliaksandr V. Yakutovich
    View author publications

    You can also search for this author in PubMed Google Scholar

  42. Jusong Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  43. Austin Zadoks
    View author publications

    You can also search for this author in PubMed Google Scholar

  44. Bonan Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  45. Giovanni Pizzi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.F. and N.M. contributed the idea of using prototype oxides to test pseudopotentials across different coordinations and chemistries. S.C. and K.L. performed an initial assessment of this idea and analysed, together with M.C., the first exploratory datasets. M.S. was responsible for job, queue and data management in the first exploratory phase. E.B. and G.Pi. contributed the idea of using the AiiDA and aiida-common-workflows infrastructure to carry on the thousands of DFT simulations required by the project. E.B. and G.Pi. coordinated the whole project. P.B., G.M. and O.R. performed the iterative refinement of the input parameters for the AE calculations that ultimately resulted in the generation of the central volumes of our dataset. O.R. proposed the ε metric. N.M. proposed the ν metric. S.P. raised the issue of the smearing selection that ultimately led to the decision of a fixed k-point integration mesh and smearing broadening. K.E. contributed to the data analysis and the conversion of the data into a dynamic website. D.E.P.V. contributed the Hirshfeld-I calculated charges and their analysis. E.B., M.W. and G.Pi. performed the analysis of the error propagation in the fit and the estimation of the parameters of the ν metric. E.B., S.C., O.R. and G.Pi. analysed in detail the dependency and sensitivity of the metrics Δ, ε and ν. A.Z. and S.P. developed the ABINIT implementation of the common workflow which relies on the aiida-abinit plugin developed and maintained by A.Z., G.Pe. and S.P. M.G. created new pseudopotentials used for ABINIT and improved the parameter profile. A.Z. and S.P. performed the ABINIT calculations, including verification tests. The work on ABINIT was supervised by G.-M.R. and S.P. L.B., A.D. and L.G. contributed to the BigDFT-related parts of the work. A.D. developed the BigDFT implementation of the common workflows. L.B. and A.D. generated the BigDFT results under the supervision of L.G. B.Z. developed the CASTEP implementation of the common workflow, which relies on the aiida-castep plugin also maintained by B.Z., and performed all CASTEP simulations. C.J.P. created new on-the-fly generated pseudopotentials using the verification tests performed by B.Z. M.K., T.D.K., H.M., T.M.A.M. and A.V.Y. contributed to all CP2K-related parts of this work. A.V.Y. implemented the workflows and performed preliminary calculations. The workflows rely on the aiida-cp2k plugin developed by A.V.Y., T.M.A.M. and others. M.K. performed preliminary calculations, created new pseudopotentials and contributed to the design of the protocol, T.D.K. contributed to the CP2K setup, discussed the results and supervised the calculations, H.M. conducted all AiiDA calculations and analysed the results, T.M.A.M. provided implementations of CP2K input and output parsers. S.B., J.B., H.J., G.M. and D.W. contributed the FLEUR-related parts of this work. G.M. developed the parameter profile and performed the calculations, and is the contact person for the FLEUR contributions. J.B. and H.J. adapted and extended the AiiDA-FLEUR plugin and the related parts of the AiiDA common-workflows package. S.B. and D.W. contributed to the analysis and discussion of the FLEUR results. G.Ka. and S.V. contributed to the GPAW-related parts of the work. S.V. developed the GPAW implementation of the common workflows, which relies on the aiida-ase plugin, and ran the calculations. G.Ka. and S.V. analysed the GPAW calculations. M.B., S.P.H., N.M., J.Y. and G.Pi. contributed to all Quantum ESPRESSO-related parts of this work. M.B. and S.P.H. developed the Quantum ESPRESSO implementation of the common workflow, which relies on the aiida-quantumespresso plugin developed and maintained by M.B., S.P.H., G.Pi. and others. J.Y. generated and tested new pseudopotentials used for Quantum ESPRESSO. The work on Quantum ESPRESSO was supervised by G.Pi. and N.M. H.M. and T.D.K. contributed to all SIRIUS/CP2K-related parts of this work. The workflows rely on the aiida-cp2k plugin developed by A.V.Y., T.M.A.M., and others. H.M. conducted all AiiDA calculations and analysed the results. T.D.K. contributed to the SIRIUS/CP2K setup, discussed the results and supervised the calculations. E.B., V.D. and A.G. contributed to the SIESTA-related parts of the work. E.B. developed the SIESTA implementation of the common workflows, which relies on the aiida-siesta plugin developed by E.B., A.G., V.D. and others. E.B. generated the SIESTA results in collaboration with A.G. M.W., M.M. and E.F.-L. performed the execution and analysis of the VASP-related workflows used to generate the data for this work. G.Kr. generated updated potentials for the lanthanides. E.F.-L. maintains the VASP implementation of the common-workflows project and the aiida-vasp plugin (developed by a community of contributors; see full contributor list in the plugin documentation) which is used to execute the VASP calculations. P.B., G.K.H.M., O.R. and T.R. contributed the WIEN2k-related parts of this work. T.R. performed preliminary calculations, P.B. created the setup of the WIEN2k calculations and supervised and analysed the results, G.K.H.M. contributed the conversion of AiiDA structures to a WIEN2k struct file, and O.R. developed the aiida-wien2k plugin and performed all AiiDA-WIEN2k calculations. E.B., M.F. and G.Pi. wrote the first version of the manuscript, and all authors contributed to the editing and revision of the manuscript.

Corresponding author

Correspondence to Giovanni Pizzi.

Ethics declarations

Competing interests

G.Pe. and G.-M.R. are shareholders and directors of Matgenix SRL. G.Kr. is a shareholder of the VASP Software GmbH, and M.W. and M.M. are part-time employees of the VASP Software GmbH. C.J.P. is an author of the CASTEP code and receives income from its commercial sales.

Peer review

Peer review information

Nature Reviews Physics thanks Vikram Gavini, Ann Mattson and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bosoni, E., Beal, L., Bercx, M. et al. How to verify the precision of density-functional-theory implementations via reproducible and universal workflows. Nat Rev Phys 6, 45–58 (2024). https://doi.org/10.1038/s42254-023-00655-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42254-023-00655-3

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing