Direct and indirect column lineage

Last updated: 6 Oct 20265 min read

tutorial

A column can shape an output by supplying its value or by deciding which rows survive, and an impact graph should distinguish those two roles.

Trace values and selection separately

For a revenue total, payment amount is a direct value input. Payment status may only decide which rows enter the sum; it is still an indirect dependency. A dataset-level edge between payments and revenue misses this distinction. Model each output field with source-field edges labelled as value, filter, join, grouping or ordering effects. Dataset lineage remains the outer graph, while field edges answer what a column change can affect.

Include join and grouping keys

A region label in a report may come from a dimension table, yet payment region ID decides which dimension row joins. A change to either can move an amount between groups without changing the amount itself. Record the join key as an indirect dependency of the grouped total and the display label as a direct dependency of the output label. Do not infer that every column in every joined table affects every output field.

Describe transformations honestly

A pass-through customer ID, a rounded currency conversion and a count of customers have different propagation properties. Record transformation kind and whether a masking step occurred, but do not mark hashing as irreversible protection without considering the input domain. A computed constant has no value-source edge, though a WHERE clause can still change whether its row exists. If SQL parsing cannot resolve a stored procedure or user function, mark the field relation unknown rather than fabricating confidence.

Bind edges to a release

A view definition can change while an earlier table snapshot remains in use. Attach lineage edges to the transform revision and committed output generation; failed attempts do not become parents of the current answer. Run evidence lets an operator move from a published result to the actual code and input versions instead of to only a planned DAG.

Test impact queries

Ask which outputs are affected if the payment status domain gains a new value. The graph should include the revenue total even though status contributes no numeric bytes. Ask which outputs are affected by renaming an unused ingestion field; they should not appear unless the field is propagated. Use hand-reviewed SQL fixtures to measure missed and spurious edges before relying on the graph for change approval.

Implementation

field_edges = {
    "mart.revenue_cents": {
        ("payments.amount_cents", "value"),
        ("payments.status", "filter"),
        ("payments.region_id", "join"),
    },
    "mart.region_name": {
        ("regions.name", "value"),
        ("payments.region_id", "join"),
    },
}

def impacted_outputs(source_field, edges):
    return {output for output, dependencies in edges.items()
            if any(field == source_field for field, _ in dependencies)}

assert impacted_outputs("payments.status", field_edges) == {"mart.revenue_cents"}
assert impacted_outputs("payments.region_id", field_edges) == {
    "mart.revenue_cents", "mart.region_name"}

Performance and operating cost

For E stored field edges, a linear impact query is O(E) time and O(O) result space for O affected outputs. An inverse index improves repeated lookups but adds update and storage cost. Capturing full field relations across many transforms can grow much faster than table-level lineage; retain only verified edges and an explicit unknown state where extraction is incomplete.

Common Mistakes

Do not omit filter or join fields just because they do not supply output values.
Do not infer every source column affects every destination column.
Do not treat an unparseable transform as proof of no dependency.