Pipeline lineage and impact analysis

Last updated: 6 Oct 20265 min read

tutorial

Lineage records which input datasets and transformations produced an output, allowing a changed source or failed run to be traced to affected consumers.

Record concrete dataset identities

A name such as orders is ambiguous across environments, regions and snapshots. Use a namespace, dataset name, version or snapshot ID, and run ID. Record input and output versions at completion, even when a job declares its expected dependencies in code. The run manifest links observed lineage to the exact published artifact.

Avoid false edges

A job may read customers and orders but independently write a customer count and an order total. Treating every input as a source of every output invents dependencies that do not exist. Capture edges at the transformation or output-field level when impact decisions require that precision. A coarse DAG remains useful for initial discovery, but it should not claim exact column provenance.

Use lineage to plan a change

Before changing amount_cents to a new currency unit, traverse descendants to find reports, features, exports and caches. For each consumer, identify owner, compatibility requirement, test fixture and rollout order. Schema rollout needs this dependency list. The existence of a lineage edge alone does not prove the consumer can decode or interpret the change.

Use lineage during an incident

If an incorrect exchange-rate table was published at 06:10, find runs that consumed that version and outputs they published. Mark those outputs suspect, pause their downstream publication, repair the source, and replay affected intervals. A broad table-level edge may overstate the blast radius; that's safer than missing a real descendant, but costs more recovery work.

Keep manual data flows visible

Not every export, notebook or one-off upload emits metadata events. Maintain a register for critical manual transfers and periodically compare read logs to the catalog. A lineage graph can be technically correct for observed jobs yet incomplete for the business. Deletion work cannot rely only on instrumented edges.

Implementation

from collections import deque

downstream = {
    "raw.orders": {"curated.orders", "audit.order_events"},
    "curated.orders": {"finance.daily_revenue", "ml.order_features"},
    "audit.order_events": set(),
    "finance.daily_revenue": set(),
    "ml.order_features": set(),
}

def affected_datasets(graph, changed):
    seen, pending = set(), deque([changed])
    while pending:
        dataset = pending.popleft()
        for child in graph.get(dataset, ()):
            if child not in seen:
                seen.add(child)
                pending.append(child)
    return seen

assert "finance.daily_revenue" in affected_datasets(downstream, "raw.orders")

Performance and operating cost

A breadth-first traversal is O(V + E) over V datasets and E dependency edges, with O(V) memory. Producing and retaining accurate run-level lineage costs instrumentation and catalog storage. Missing edges are a bigger risk than traversal speed.

Common Mistakes

Do not infer every input feeds every output in a multi-output job.
Do not use a dataset name without environment and version identity.
Do not assume an instrumented graph includes manual exports.