Data lineage
Data lineage is the path from source data through transformations to a resulting record, table, or dataset. It can describe movement between systems, joins, filters, calculations, and versioned outputs. Provenance also covers origins and responsible parties; lineage commonly focuses on the processing path itself.
How it works
Record inputs, transformation rules or code versions, execution context, and outputs. The level of detail can range from dataset-level relationships to field-level derivations. A useful lineage record lets a reviewer determine how an output was produced and which upstream changes might affect it.
Keep the history consistent with actual processing. Undocumented spreadsheet edits or manual corrections can interrupt an otherwise automated lineage chain.
Why it matters for licensing
Lineage helps explain preparation work, investigate quality problems, and identify derived artifacts affected by a source restriction or correction. It supports licensing operations but does not itself establish permission to use the inputs.
Example
Fictional example: A dataset’s “resolution_days” field combines timestamps from a service platform and a billing system. Lineage records the join key, time-zone conversion, and calculation version, allowing a reviewer to identify why an incorrect value appeared.
Limitations and misconceptions
A recorded pipeline can reproduce a mistake as consistently as a correct result. Lineage may also be incomplete across external systems. It should be paired with validation, provenance, and rights documentation.
Questions to ask
- Can important outputs be traced to inputs and transformation rules?
- Are manual changes and code versions captured?
- Which downstream datasets would need correction if a source changed?
Sources
- W3C — PROV Overview · Accessed
- Gebru et al. — Datasheets for Datasets · Accessed
Explore whether your business data could be a fit.
Start with a description of your systems—not a data upload.