Column-lineage coverage and sensitive-change gates

Last updated: 6 Oct 20265 min read

tutorial

A field-impact graph is useful only when its coverage is visible and a sensitive source-field change cannot bypass unresolved paths.

State what was observed

Report the number of published output fields, fields with verified edges, fields with inferred edges and fields whose transform remains opaque. Coverage is measured against actual published schema, not merely the SQL files the parser could read. A graph that omits a notebook, stored procedure or generated query can look tidy while missing the most important path. Field lineage needs a confidence label on each edge.

Walk both directions

For a proposed input change, traverse forward to outputs and consumers. For a suspicious output, traverse backward to candidate inputs and run generations. Keep direct and indirect paths visible; a consent flag used only in a filter can still alter a published customer count. Join against contract rules to see which tests protect affected fields and which outputs have no explicit assertion.

Gate sensitive changes

If a source field is classified as restricted, a new downstream edge must trigger access and masking review before release. Do not assume that a hashed value is safe for every consumer or that an aggregate cannot disclose a small group. If the path reaches an unknown transform, pause automatic approval for that branch and route it to an owner. Metadata minimization also limits who may inspect detailed paths and partition values.

Compare graph to execution

A static parser may claim a column is used, but a runtime branch can be disabled for one release. Record the transform revision and actual successful run alongside static edges. Sample selected output values and candidate input versions where policy allows, without placing raw records in the broad catalog. A failed candidate can inform debugging but must not replace lineage for the currently published snapshot.

Exercise a change request

Rename a harmless unused field, alter a payment amount column and add a consent condition. The impact query should return different sets and reasons. Intentionally hide one transform from the extractor: coverage must fall and the sensitive-change gate must fail closed for the affected path. Store reviewer decision, unknown edges, test results and final committed generation for later audit.

Implementation

published_fields = {
    "mart.revenue_cents", "mart.customer_count", "mart.region_name"
}
verified_edges = {"mart.revenue_cents", "mart.customer_count"}
unknown_transform_fields = {"mart.region_name"}

def lineage_gate(outputs, verified, unknown, sensitive_change):
    uncovered = outputs - verified
    if sensitive_change and (uncovered or unknown):
        return {"approved": False, "uncovered": uncovered | unknown}
    return {"approved": True, "uncovered": uncovered}

result = lineage_gate(published_fields, verified_edges,
                      unknown_transform_fields, sensitive_change=True)
assert result["approved"] is False
assert result["uncovered"] == {"mart.region_name"}

Performance and operating cost

Coverage comparison is O(F) expected time and O(U) output state for F published fields and U uncovered fields. Forward and backward graph traversals cost O(V + E) in the inspected subgraph. Collecting verified field edges, running review gates and retaining run-specific evidence add metadata work, but prevent a high-confidence label from hiding an unobserved transform.

Common Mistakes

Do not report coverage only over fields a parser happened to recognize.
Do not treat unknown lineage as an empty dependency set.
Do not grant a sensitive change solely because the displayed output field is aggregated.