Data leakage
Here, data leakage means unintended information availability that compromises training or evaluation validity. A model may appear accurate because it has seen test examples or because its inputs contain information only available after the predicted event. This is distinct from the security meaning of leakage as unauthorized disclosure, although a project can face both problems.
How it works
Leakage can arise from duplicate examples across splits, related records divided improperly, preprocessing fitted on the whole dataset, or features that reveal future outcomes. Preventing it requires defining when a prediction occurs and what information is available then.
Split data at the level appropriate to the task, such as case, organization, time period, or source document. Audit feature creation and model selection as well as raw overlap. Deduplication helps with some forms of contamination but does not cover every route.
Why it matters for licensing
A licensed evaluation dataset is useful only if it supports credible assessment. Documentation should explain collection timing, related records, preprocessing, and prior exposure. A recipient should not infer independence merely because files have different names.
Example
Fictional example: A model predicts whether a service request will need escalation, but a joined input field contains the final escalation status added later. Removing that field and rebuilding the time-appropriate dataset changes the evaluation substantially.
Limitations and misconceptions
Not every strong predictor is leakage; it may be legitimately available at decision time. Conversely, different text does not establish independence when examples describe the same underlying event. The assessment must match the intended deployment setting.
Questions to ask
- What information would actually be available at prediction time?
- Can related cases, duplicates, or preprocessing cross the train/test boundary?
- Has repeated tuning reduced the independence of the evaluation set?