Reproducible builds: investigate why equal inputs produce different bytes

Last updated: 5 Oct 20266 min read

tutorial

A reproducible build emits the same artifact bytes from the same declared inputs and build process. Differences may come from timestamps, filesystem order, environment locale, toolchain version, random identifiers, mutable dependencies, or undeclared files. A mismatch is evidence of uncontrolled input; it is not by itself proof of malicious modification. The investigation must preserve both outputs and the input manifest so the first differing layer can be located.

Operational decision

Build the receipt package twice in separate clean runners using the same source revision, lockfiles, toolchain image digest, and dependency repository snapshot. The fragment fixes the source timestamp input for tools that honor it and records a package digest; it is a diagnostic scaffold, not a claim that the package will now be bit-for-bit identical. Compare archive member names, ordering, modes, timestamps, generated source, and dependency bytes before comparing the whole package again. If a generated manifest embeds the wall clock, change it to a source-derived timestamp or move display time to runtime. If compression adds unstable metadata, adjust packaging rather than normalizing a compromised output after the fact. Record each unexplained difference and block promotion when the release policy requires reproducibility. After repair, repeat on a different runner and compare the actual registry artifact, not only local files.

set -euo pipefail
export SOURCE_DATE_EPOCH="$(git show -s --format=%ct HEAD)"
./ci/build-receipt-package.sh
sha256sum dist/receipt-api.tar

Cost and verification

Two clean builds roughly double build compute and artifact storage for the check. Making a build reproducible can require pinning toolchains and dependency repositories, which adds maintenance work but improves incident analysis. Measure the fraction of builds that match, the first differing file or layer, and time to identify an undeclared input. A digest comparison alone gives a yes-or-no signal; a useful diff report explains what changed and whether it can affect the deployed behavior.

Common Mistakes

Do not assume setting one timestamp variable makes every tool reproducible.
Do not normalize away a binary mismatch before inspecting its cause.
Do not compare builds whose dependency or toolchain inputs differ.

Connected lessons

Practice and check

devops

operations