Contract gate severity and release evidence

Last updated: 6 Oct 20265 min read

tutorial

A candidate dataset should not become the current answer until its required checks pass and any permitted exceptions have named owners and expiry.

Separate reject from observe

A missing business key or mismatched money total can block publication. A mild distribution shift may warn while the candidate is still usable; an intermediate rule may require a narrow, approved exception. Give each contract rule an owner, severity, scope and threshold before execution. Do not downgrade a failing rule merely to push a late batch through. Correction-aware alerts help distinguish a legitimate backfill from a broken source.

Bind evidence to the candidate

A green test from yesterday is not proof for today’s files. Store candidate generation, contract version, input generations, test runner revision, counts and failure samples in a restricted report. Advance the public pointer only if the report belongs to that exact candidate and every blocking rule passes. A replay manifest should make the same result reproducible later.

Handle exceptions as data

Sometimes a source cannot repair a known issue before an agreed deadline. Record a narrowly scoped exception with a rule ID, affected interval, owner, reason and expiration. The gate may apply it only to those rows and that candidate generation. An expired exception fails closed. Keep the original failure count visible so an exception does not make the data appear clean.

Test consumers, not only tables

A table may pass its schema checks while a dependent dashboard misinterprets a new status value. Include selected consumer queries and metric controls in the release gate for high-impact fields. Verify row policies and masks after a contract change; a new column can escape a broad projection. Access tests and output tests inspect different failure surfaces.

Prove rollback behavior

Plant a duplicate payment ID and verify that the candidate stays unpublished while the previous generation serves requests. Repair the duplicate, rerun all applicable checks and then move the pointer atomically. Publish the failure and success evidence under separate attempt IDs. A retry must not reuse a passing result produced for another candidate, even when the scheduled interval is the same.

Implementation

from datetime import date

checks = [
    {"rule": "payment_key_unique", "severity": "block", "passed": True},
    {"rule": "country_mix_shift", "severity": "waivable", "passed": False},
]
exception = {"rule": "country_mix_shift", "owner": "payments-ops",
             "expires": date(2026, 10, 7), "candidate": "payments-47"}

def publishable(results, candidate, exceptions, today):
    for result in results:
        if result["passed"] or result["severity"] == "warn":
            continue
        if result["severity"] == "block":
            return False
        if not (result["severity"] == "waivable"
                and exceptions["rule"] == result["rule"]
                and exceptions["candidate"] == candidate
                and exceptions["owner"]
                and today <= exceptions["expires"]):
            return False
    return True

assert publishable(checks, "payments-47", exception, date(2026, 10, 6))
assert not publishable(checks, "payments-47", exception, date(2026, 10, 8))
assert not publishable([{**checks[0], "passed": False}], "payments-47", exception, date(2026, 10, 6))

Performance and operating cost

Evaluating R rule results is O(R) time and O(B) state for B blocking failures. The expensive part is producing each result against the candidate data, especially full-key and cross-table reconciliations. Persisting immutable evidence costs storage but prevents a stale pass report from authorizing a different release. Gate time belongs in the pipeline freshness budget.

Common Mistakes

Do not reuse a passing report for a different candidate generation.
Do not let an exception survive beyond its approved interval and owner.
Do not move the serving pointer while a blocking rule still fails.