There is a newer version of the record available.

Published May 4, 2023 | Version v1
Dataset Open

Single-model uncertainty quantification in neural network potentials does not consistently outperform model ensembles

  • 1. Department of Materials Science and Engineering, Massachusetts Institute of Technology (MIT), Cambridge, Massachusetts, United States of America
  • 2. Innovative Technology Laboratories, AGC Inc., Yokohama, Japan
  • 3. Computational and Systems Biology, Massachusetts Institute of Technology (MIT), Cambridge, Massachusetts, United States of America

* Contact person

Description

Neural networks (NNs) often assign high confidence to their predictions, even for points far out-of-distribution, making uncertainty quantification (UQ) a challenge. When they are employed to model interatomic potentials in materials systems, this problem leads to unphysical structures that disrupt simulations, or to biased statistics and dynamics that do not reflect the true physics. Differentiable UQ techniques can find new informative data and drive active learning loops for robust potentials. However, a variety of UQ techniques, including newly developed ones, exist for atomistic simulations and there are no clear guidelines for which are most effective or suitable for a given case. In this work, we examine multiple UQ schemes for improving the robustness of NN interatomic potentials (NNIPs) through active learning. In particular, we compare incumbent ensemble-based methods against strategies that use single, deterministic NNs: mean-variance estimation, deep evidential regression, and Gaussian mixture models. We explore three datasets ranging from in-domain interpolative learning to more extrapolative out-of-domain generalization challenges: rMD17, ammonia inversion, and bulk silica glass. Performance is measured across multiple metrics relating model error to uncertainty. Our experiments show that none of the methods consistently outperformed each other across the various metrics. Ensembling remained better at generalization and for NNIP robustness; MVE only proved effective for in-domain interpolation, while GMM was better out-of-domain; and evidential regression, despite its promise, was not the preferable alternative in any of the cases. More broadly, cost-effective, single deterministic models cannot yet consistently match or outperform ensembling for uncertainty quantification in NNIPs.

Files

File preview

files_description.md

All files

Files (140.0 MiB)

Name Size
md5:bb4e78db3b88e5420156113b4bd5c4b8
378 Bytes Preview Download
md5:7bb78eefb69c724dc7276f916c1eb70c
95.7 KiB Download
md5:4e24ab2ab0d26246a5fff157182d76e1
37.6 KiB Download
md5:599dd5689f3b96ac856db3120bf33dae
2.1 KiB Preview Download
md5:3996912931795a2a3245e2cbb4e4cade
28.2 MiB Download
md5:d1e53f5238e084691cec2f23d485f154
111.6 MiB Download

References

Preprint
A. R. Tan, S. Urata, S. Goldman, J. C. B. Dietschreit, R. Gomez-Bombarelli, arXiv:2305.01754 (2023), doi: 10.48550/arXiv.2305.01754