Object integrity: verify bytes without trusting an ETag shortcut

Last updated: 5 Oct 20266 min read

tutorial

Multipart uploads and some encryption or copy paths produce ETags that cannot be treated as an MD5 digest of the entire payload. A transfer success response proves the storage API accepted an operation, not that the application selected the correct source version or can parse the resulting document. Store a digest or supported checksum with an explicit algorithm and checksum type, then compare like with like across upload, copy, and restore paths.

Operational decision

A receipt archive copies nightly bundles into a recovery account. The producer computes a SHA-256 digest over the final uncompressed payload and writes it into a signed manifest with key, version identifier, byte length, record count, and schema generation. Upload with provider-supported integrity checks and keep the producer digest as the application-level authority. After copy, inspect destination version, size, encryption accessibility, and provider checksum type. If a multipart source becomes a single-part copy, a provider checksum or ETag can change even when bytes do not; download a controlled sample and recompute the application digest. For bulk estates, use a provider-supported at-rest checksum job where available, but keep a sampled application parse check because byte integrity does not prove semantic validity. Compare the manifest against both source and destination inventories, quarantine mismatches, and never overwrite the good source to make a failed comparison disappear.

Receipt bundle manifest
Key: receipts/2026-09-27/batch-47.bin
Source version: recorded at upload completion
Payload SHA-256: computed over final uncompressed bytes
Length and record count: recorded by producer
Destination version: recorded after copy
Acceptance: digest, length, decrypt, and parser result agree

Cost and verification

Hashing B bytes costs O(B) CPU and O(1) streaming memory, while a full readback also incurs B bytes of retrieval and network traffic. Inventory reconciliation over N entries costs O(N) comparisons with memory bounded by the chosen join strategy. Measure checksum mismatch rate, unverified-copy age, and time to retrieve and parse a sample. A small verification sample is cheaper but leaves residual risk; size it from the recovery objective and observed defect rate.

Common Mistakes

Do not compare an ETag with a whole-file MD5 across multipart or encryption paths.
Do not compare checksums with different algorithms or composite/full-object types.
Do not call a copy recoverable until a reader can decrypt and parse it.

Connected lessons

Practice and check

devops

object-storage