Project: audit a payment column change across a mart

Model the pipeline

Create a raw payments feed, cleaned payments table, region dimension and revenue mart. Include output fields for regional revenue, customer count and region name. Give amount a direct aggregation edge, status a filter edge, region ID a join and group edge, and name a direct label edge. Pin each published dataset generation and transform revision. The field relation model should distinguish changing values from changing row membership.

Build impact queries

Implement forward lookup from source field to every reachable output and consumer, then backward lookup from one suspicious output to input fields and committed runs. Change payment status from a two-value domain to a three-value domain and show why it affects a revenue total even though status is not summed. Rename an unused landing field and show that it has no verified path into the mart.

Expose the blind spot

Route the customer-count calculation through a stored procedure or opaque transform that the extractor cannot parse. Mark the output unknown, decrease coverage and require owner review before a sensitive field change. Do not invent all-to-all edges just to hide uncertainty. The coverage gate should identify the precise unresolved output and its owner.

Test the release

Add a restricted customer identifier to the cleaned feed and attempt to propagate a derived identifier into the mart. Check field classification, access policy, masking review and small-group release rule. Simulate a failed candidate run, then a successful one; only the successful output generation becomes current lineage. Record a query result before and after the status change to validate the declared indirect edge.

Deliver an audit trail

Submit the graph, edge types, coverage denominator, unknown-transform report, forward and backward query outputs, test results, access decision, failed and successful run IDs and final generation. Include one deliberately missing filter edge and show the impact query failing its reviewed fixture. A polished graph with an invisible blind spot does not pass this project.

Implementation

edges = {
    "clean.amount_cents": {"raw.amount_cents"},
    "clean.status": {"raw.status"},
    "mart.revenue_cents": {"clean.amount_cents", "clean.status"},
    "mart.customer_count": {"UNKNOWN:customer_counter"},
}

def downstream_fields(source_field, dependencies):
    reached = {source_field}
    frontier = [source_field]
    while frontier:
        current = frontier.pop()
        for output_field, inputs in dependencies.items():
            if current in inputs and output_field not in reached:
                reached.add(output_field)
                frontier.append(output_field)
    return reached - {source_field}

assert downstream_fields("raw.status", edges) == {
    "clean.status", "mart.revenue_cents"}
assert "mart.customer_count" not in downstream_fields("raw.status", edges)

Performance and operating cost

The simple traversal scans E edges for each visited field and can cost O(V × E) without an inverse index; building that index reduces traversal to O(V + E) time and space for the affected graph. Field-level evidence and run versions add storage. Use them to focus review and tests, while keeping sensitive field paths restricted to authorized operators.

Common Mistakes

Do not treat a filter-only status column as irrelevant to revenue.
Do not publish a sensitive derived identifier before access review.
Do not convert an opaque transform into an asserted no-impact result.