Hermetic CI tests: declare every service and input the check consumes

Operational decision

A receipt integration suite passes on one runner because an old PostgreSQL daemon and an inherited configuration file happen to exist. Start from a clean runner image, provision an explicitly versioned disposable database, apply migrations, seed synthetic fixtures, and fail if an unexpected outbound connection occurs. The policy fragment records the expected inputs. Run the check twice with a cleared workspace and cache, then change runner host and timezone to see whether output remains valid. Pin the database image by immutable digest after it is reviewed; a mutable tag alone cannot prove the same server bytes were used. Give every test a writable temporary directory and never write into the source checkout. Keep one small, separately controlled external contract test if a real payment provider interaction is essential, and label failures of that path as dependency failures rather than silently blaming code.

Receipt integration input contract
Source: checked revision and lockfile digest
Runtime: declared toolchain image digest
Database: disposable instance at approved image digest
Schema: migrations from checked revision
Fixtures: synthetic and versioned
Network: deny unexpected external endpoints
Workspace: clean checkout and unique temporary directory

Cost and verification

Provisioning fresh services and downloading pinned images can slow a cold run, while caches can recover much of that cost if they remain untrusted inputs. The larger payoff is fewer environment-only failures and stronger culprit identification. Measure cold and warm duration, unexpected network attempts, cache-hit dependence, and failures that reproduce on a clean runner. A hermetic integration check does not replace a separate production-path smoke test; each answers a different question.

Common Mistakes

• Do not assume a container automatically makes a test hermetic.
• Do not let a mutable service image tag define the test database.
• Do not use a hidden runner daemon as an undeclared dependency.

Connected lessons

• DevOps: delivery, infrastructure, and reliable operations
• CI dependency caches: speed without hidden build inputs
• CI runner isolation: treat repository code as untrusted
• CI test data: give each run an isolated ownership scope
• Container builds: small runtime, explicit privilege

Practice and check

• Project: prove a receipt release passed the checks that matter
• DevOps CI verification decisions