metadata

title: ECE
datasets:
  - 'null'
tags:
  - evaluate
  - metric
description: Expected calibration error (ECE)
sdk: gradio
sdk_version: 3.19.1
app_file: app.py
pinned: false

Metric Card for ECE

Metric Description

This metrics computes the expected calibration error (ECE). It directly calls the torchmetrics package: https://torchmetrics.readthedocs.io/en/stable/classification/calibration_error.html

How to Use

Inputs

List all input arguments in the format below

predictions (float32): predictions (after softmax). They must have a shape (N,C,...) if multiclass, or (N,...) if binary.
references (int64): reference for each prediction, with a shape (N,...).

Output Values

ECE as float.

Examples

ce = evaluate.load("Natooz/ece")
results = ece.compute(
    references=np.array([[0.25, 0.20, 0.55],
                         [0.55, 0.05, 0.40],
                         [0.10, 0.30, 0.60],
                         [0.90, 0.05, 0.05]]),
    predictions=np.array(),
    num_classes=3,
    n_bins=3,
    norm="l1",
)
print(results)

Citation

@inproceedings{NEURIPS2019_f8c0c968,
     author = {Kumar, Ananya and Liang, Percy S and Ma, Tengyu},
     booktitle = {Advances in Neural Information Processing Systems},
     editor = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett},
     publisher = {Curran Associates, Inc.},
     title = {Verified Uncertainty Calibration},
     url = {https://papers.nips.cc/paper_files/paper/2019/hash/f8c0c968632845cd133308b1a494967f-Abstract.html},
     volume = {32},
     year = {2019}
}