Expected calibration error, ECE, is a summary of how far a model's stated probabilities are from the frequencies actually observed, averaged across bins of its predictions. A lower value means stated likelihoods are closer to what happened.
To compute it, predictions are grouped into bins by stated probability, the observed rate of the event in each bin is compared to the bin's stated probability and the gaps are averaged, weighted by how many predictions fall in each bin.
It condenses a reliability curve into one number, which is convenient, and loses detail, which is why the curve is shown beside it. A model can have a low overall error and still be off in a range that matters.
ECE is not accuracy. A model can rank matters well and be poorly calibrated, or be well calibrated and rank poorly.
Working a file that turns on this term?
Working through a diligence process?
Institutional partners evaluating a position against this platform's outcome and duration models are welcome to reach out directly.