Recommended by

Indexed by

Expectation consistency for calibration of neural networks (code)

Lucas Clarte1*, Bruno Loureiro2, Florent Krzakala3, Lenka Zdeborova1

1 École Polytechnique Fédérale de Lausanne (EPFL), Statistical Physics of Computation lab., CH-1015 Lausanne, Switzerland

2 Département d’Informatique, École Normale Supérieure - PSL & CNRS, Paris, France

3 École Polytechnique Fédérale de Lausanne (EPFL), Information, Learning and Physics lab., CH-1015 Lausanne, Switzerland

* Corresponding authors emails: lucas.clarte@epfl.ch
DOI10.24435/materialscloud:ws-p3 [version v1]

Publication date: Sep 19, 2023

How to cite this record

Lucas Clarte, Bruno Loureiro, Florent Krzakala, Lenka Zdeborova, Expectation consistency for calibration of neural networks (code), Materials Cloud Archive 2023.144 (2023), https://doi.org/10.24435/materialscloud:ws-p3


Despite their incredible performance, it is well reported that deep neural networks tend to be overoptimistic about their prediction confidence. Finding effective and efficient calibration methods for neural networks is therefore an important endeavour towards better uncertainty quantification in deep learning. In this manuscript, we introduce a novel calibration technique named expectation consistency (EC), consisting of a post-training rescaling of the last layer weights by enforcing that the average validation confidence coincides with the average proportion of correct labels. First, we show that the EC method achieves similar calibration performance to temperature scaling (TS) across different neural network architectures and data sets, all while requiring similar validation samples and computational resources. However, we argue that EC provides a principled method grounded on a Bayesian optimality principle known as the Nishimori identity. Next, we provide an asymptotic characterization of both TS and EC in a synthetic setting and show that their performance crucially depends on the target function. In particular, we discuss examples where EC significantly outperforms TS. This record provides the code for the paper "Expectation consistency for calibration of neural networks".

Materials Cloud sections using this data

No Explore or Discover sections associated with this archive record.


File name Size Description
29.8 MiB Compressed files of the Github repository https://github.com/SPOC-group/expectation-consistency


Files and data are licensed under the terms of the following license: Creative Commons Attribution 4.0 International.
Metadata, except for email addresses, are licensed under the Creative Commons Attribution Share-Alike 4.0 International license.


MARVEL/P2 neural networks uncertainty quantification temperature scaling

Version history:

2023.144 (version v1) [This version] Sep 19, 2023 DOI10.24435/materialscloud:ws-p3