Synthetic-data releases: seeds, manifests and rejection gates

Last updated: 5 Oct 20265 min read

tutorial

A generated dataset should be traceable to a versioned method and released only after declared checks pass.

Record the full recipe

Store generator code revision, configuration, seed, schema version, training-source cutoff, constraint rules and any postprocessing. A seed helps reproduce a deterministic test fixture, but a seeded pseudorandom generator is not a privacy mechanism. If a learned model has nondeterministic training, record the environment and artifact hash as well. Publication boundaries prevent partial versions from leaking into consumers.

Gate distinct risks

Run schema and relational checks, scenario coverage, downstream utility against an untouched holdout, disclosure review and access-policy checks. A failure in any required gate blocks release; a high utility score cannot cancel a privacy failure. Capture counts of rejected rows and reasons, not only a pass flag. Manual approval should reference the exact dataset checksum being approved.

Plan withdrawal

Publish a manifest that names the dataset version, intended consumers, expiry or review date and rollback location. If a later disclosure test finds a problematic near copy, withdraw that version and invalidate downstream caches or training runs as policy requires. Keep an incident trace linking the generated row to the generator and source cohort without exposing sensitive source records in public reports.

Rebuild and compare

Generate 47 receipt rows twice with the same recipe and verify checksums match in the deterministic fixture case. Change the tax constraint and confirm a new version and checksum appear. Force one orphan refund and one source-copy match; both gates should block publication while preserving a diagnostic report. The release decision must name the exact failed checks.

Implementation

def release_allowed(gate_results):
    required = {"schema", "relationships", "coverage", "utility", "disclosure"}
    return required <= gate_results.keys() and all(gate_results[name] for name in required)

Performance and operating cost

Checking a fixed set of gates is O(1). Computing a checksum and scanning N rows is O(N) time; model training, subgroup evaluation and near-copy analysis usually dominate the actual release cost.

Common Mistakes

Do not mistake a seed for a privacy guarantee.
Do not approve one checksum and publish another.
Do not leave downstream consumers on a withdrawn dataset version.