Brief: Researchers have developed a distribution-free, computationally efficient, and practically reliable prediction interval method to quantify machine learning (ML) model prediction uncertainty.
Accomplishment: Researchers have developed a prediction interval method (PI3NN) to quantify ML model prediction uncertainty and theoretically proven that it precisely captures the uncertainty for a given confidence level and completely avoids the crossing issues suffered by the state-of-the-art methods for a regression problem đŠ = đ(đ„) + đ. PI3NN is particularly suitable for scientific ML (SciML). The produced uncertainty bound can assess the model predictionâs credibility and trustworthiness; it can identify data/domain shift and understand when the learned model could fail and why it fails; and it also can guide data collection to automate the experimental design and inform decision making. We developed PI3NN to accurately quantify model prediction uncertainty by training three neural networks (NNs), one NN to approximate the prediction value, the other two NNs to estimate the lower and upper bound of the prediction interval. The method directly communicates uncertainty using intervals which provides understandable information for decision making. PI3NN does not introduce any unusual hyperparameters resulting in stable performance and it requires no distributional assumption on data which makes it widely applicable for credible predictions in natural sciences and safe operation and control in complex engineered systems.
![Top panels illustrate the four steps of our PI3NN algorithm. Bottom panels illustrate the effectiveness of the Out-of-Distribution (OOD) identification feature. CCSD AI Initiative](/sites/default/files/styles/large/public/2022-04/prediction_interval_method.jpg?itok=lrR5oa-j)
Acknowledgement: This research was funded by the AI Initiative, as part of the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory, managed by UT-Battelle, LLC, for the U.S. Department of Energy (DOE).
Publications resulting from this work:
- Zhang, P., Liu, S., Lu, D., and Zhang G., A prediction interval method for uncertainty quantification of regression models. Published at ICLR 2021 workshop on simulation and deep learning. https://simdl.github.io/papers.
- Zhang, P., Liu, S., Lu, D., Sankaran, R., and Zhang, G., An out-of-distribution-aware autoencoder model for reduced chemical kinetics. Published in American Institute of Mathematical Sciences Journal. Doi: 10.3934/dcdss.2021138.
- Liu, S., Zhang, P., Lu, D., and Zhang G., PI3NN: Out-of-distribution-aware prediction intervals from three neural networks. Submitted to ICLR 2021. https://arxiv.org/abs/2108.02327.
Presentations resulting from this work:
- [Invited talk] Lu, D., âUncertainty Quantification for Machine Learning Predictionâ Invited speaker at the Department of Mathematics, Federal University of Parana (UFPR), Brazil, June 25, 2021, Virtual.
- [Invited talk] Lu, D., âA prediction interval method for uncertainty quantification of regression modelsâ Invited speaker at the Society for Industrial and Applied Mathematics (SIAM) Annual meeting, July 22, 2021, Virtual.
- [Invited talk] Lu, D., âMachine Learning Methods for Improving Terrestrial Ecosystem Predictionsâ Invited speaker at the Ecological Society of America (ESA) Annual Meeting, August 2021, Virtual.
Contact: Dan Lu (lud1@ornl.gov)
Team: Dan Lu, Siyan Liu, Pei Zhang, Guannan Zhang
![CCSD Communications Manager Scott Jones](/sites/default/files/styles/staff_profile_image_style/public/2023-06/badge%20photo.jpg?h=fdc857c4&itok=Df0f0kRg)