Deduplication

Deduplication is the specific process of identifying records that describe the same underlying real-world event, a single docket entry, a single enforcement action, a single lien, and merging them into one accurately counted record rather than allowing duplicate representations to inflate a corpus's reported scale. Duplication enters a corpus through several distinct mechanisms: source overlap, when different harvesting processes independently capture the same event from different points in an ecosystem; temporal re-harvesting, when a periodic refresh process re-pulls records that have not actually changed since a prior harvest; and unverified merged datasets, when data acquired from a prior initiative or an external source carries its own undocumented internal duplication into the combined corpus. Proper deduplication requires a defined notion of unique record identity that goes beyond exact-match text comparison, since the same event frequently appears across sources with minor formatting differences that an exact-match process would treat as separate. Deduplication has to run as an ongoing, standing process as new data enters a growing corpus, not as a one-time exercise assumed to hold indefinitely once completed, since each new source or refresh cycle reopens the same duplication risk the original cleanup was meant to close, regardless of how thorough that original cleanup actually was.