Leakage

Leakage is a modeling defect in which information that would not actually be available at prediction time enters a model's training process, inflating its apparent accuracy without producing a genuinely predictive model. Leakage findings are disclosed and the affected models retired or restricted rather than left in production quietly.

Leakage occurs when a model's training process is given access, directly or indirectly, to information that would not have been available at the actual moment a prediction needs to be made, producing a model that appears highly accurate in testing while carrying no genuine predictive value once deployed against a real, unresolved matter. A common form of leakage in legal-outcome modeling involves a target variable that is itself derived, even indirectly, from data correlated with the very outcome being predicted, in a way that would not exist at the point a real prediction is needed. Leakage is dangerous precisely because it does not announce itself: a leaking model can pass conventional accuracy checks and even appear well calibrated in a naive backtest, because the leakage inflates performance on both the training and, if the leak is structural rather than a one-time data error, the testing population as well. Catching leakage requires deliberately asking, for every feature a model uses, whether that feature would genuinely have been knowable at the moment a real prediction needs to be made, not only whether it improves the model's measured performance. This platform's governance discipline treats a confirmed leakage finding as requiring disclosure and remediation, retraining or restricting the affected model, rather than a quiet internal correction with no external accounting.