Uncertainty Quantification and Calibration in Machine Learning Models Using Bayesian Frameworks
Keywords:
UQ, Bayesian neural networks, Monte Carlo dropout, Gaussian processes, calibration, probabilistic machine learningAbstract
Uncertainty quantification (UQ) is a critical requirement for trustworthy machine learning, particularly in safety-critical applications such as healthcare, finance, and autonomous systems. Modern neural networks frequently produce overconfident predictions due to poor probabilistic calibration, limiting their reliability under distributional shift and data scarcity. While Bayesian methods provide a principled framework for modeling both epistemic and aleatoric uncertainty, a unified comparative evaluation under consistent experimental conditions has been lacking. This study systematically compares four model classes under controlled regression and classification tasks: deterministic neural networks, Bayesian neural networks, Monte Carlo dropout, and Gaussian processes. Evaluation is conducted using strictly proper scoring rules; negative log-likelihood and the Brier score, alongside the expected calibration error and predictive entropy. Results demonstrate systematic differences in uncertainty reliability across model classes: Gaussian processes achieve superior calibration in low-data regimes, while Monte Carlo dropout provides a scalable approximation with competitive performance. Deterministic networks consistently exhibit overconfidence as measured by expected calibration error. These findings motivate further development of scalable, theoretically grounded UQ methods for modern machine learning systems.
References
1. Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., et al. (2021). A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76, 243–297. https://doi.org/10.1016/j.inffus.2021.05.008
2. Abe, T., Buchanan, E. K., Pleiss, G., Zemel, R., & Cunningham, J. P. (2022). Deep ensembles work, but are they necessary? Advances in Neural Information Processing Systems, 35, 33646–33660. https://doi.org/10.52202/068431-2438
3. Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight uncertainty in neural networks. Proceedings of the 32nd International Conference on Machine Learning, 1613–1622.
https://doi.org/10.48550/arXiv.1505.05424
4. Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the 33rd International Conference on Machine Learning, 1050–1059. https://doi.org/10.48550/arXiv.1506.02142
5. Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., et al. (2023). A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl. 1), 1513–1589.
https://doi.org/10.48550/arXiv.2011.06225
6. Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. https://doi.org/10.1007/s10462-023-10562-9
7. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, 1321–1330.
https://doi.org/10.48550/arXiv.1706.04599
8. He, W., Yuan, L., Liang, Y., & Gao, J. (2024). A survey on uncertainty quantification methods for deep learning. ACM Computing Surveys, 57(4), Article 111.
https://dl.acm.org/doi/full/10.1145/3786319
9. Hüllermeier, E., & Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3), 457–506.
https://doi.org/10.1007/s10994-021-05946-3
10. Jospin, L. V., Laga, H., Boussaid, F., Buntine, W., & Bennamoun, M. (2022). Hands-on Bayesian neural networks: A tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2), 29–48. https://doi.org/10.1109/MCI.2022.3155327
11. Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30, 5574–5584.
https://doi.org/10.48550/arXiv.1703.04977
12. Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 6402–6413.
https://doi.org/10.48550/arXiv.1612.01474
13. Lambert, B., Forbes, F., Doyle, S., Dehaene, H., & Dojat, M. (2024). Trustworthy clinical AI solutions: A unified review of uncertainty quantification in deep learning models for medical image analysis. Artificial Intelligence in Medicine, 150, 102830. https://doi.org/10.1016/j.artmed.2024.102830
14. Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., et al. (2021). Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34, 15682–15694. https://doi.org/10.48550/arXiv.2106.07998
15. Neal, R. M. (2012). Bayesian Learning for Neural Networks. Springer. https://doi.org/10.1007/978-1-4612-0745-0
16. Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press.
https://doi.org/10.7551/mitpress/3206.001.0001
17. Seoni, S., Jahmunah, V., Salvi, M., Barua, P. D., Molinari, F., & Acharya, U. R. (2023). Application of uncertainty quantification to artificial intelligence in healthcare: A review of last decade (2013–2023). Computers in Biology and Medicine, 165, 107441. https://doi.org/10.1016/j.compbiomed.2023.107441
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Author(s)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Articles published in the Journal of Advanced Multidisciplinary Studies (JAMS) are licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0), unless otherwise stated. Authors retain copyright of their work and grant JAMS the right of first publication.
