Skip to content
AITroveRead. Build. Understand.
Reading list
Make this comfortable

Multipart uploads: bound abandoned parts and retry cost

Last updated: 1 Oct 20266 min read
tutorial
AdvancedBy AITrove Editorial

A multipart upload has an initiation identifier, uploaded parts, and an explicit completion or abort. A failure after some parts arrive can leave those parts stored without creating the final readable object. Listing normal objects will not reveal that cost. An abort policy controls abandoned uploads, but it must leave enough time for legitimate slow transfers to finish and account for clients retrying parts after a network break.

Operational decision

A media-ingest worker sends large evidence bundles from remote offices. Define a maximum upload duration from observed slow-link transfers, then choose an abort-after age beyond that duration with an explicit margin. List incomplete uploads and their ages in a disposable bucket before applying the rule. Kill a worker after several parts are accepted, let its retry lease expire, and verify the client either resumes with the correct upload identifier or starts a new upload and aborts the old one. Complete a different upload just before the deadline to ensure the cleanup policy does not interrupt legitimate work. Keep the authoritative ingest record separate from the object key; mark it complete only after the storage service confirms the completed object and its checksum. When a retry creates a second upload identifier for the same key, choose one winner and clean up the loser. Track incomplete-part bytes and oldest age, not only final object count. Avoid placing secrets in object keys or upload metadata because listings and logs can expose them.

Output
Evidence upload control
Maximum legitimate duration: measured on slowest supported link
Abort threshold: duration plus retry and scheduling margin
Completion gate: object exists and checksum matches ingest record
Failed worker: old upload ID recorded for cleanup
Alert: incomplete-part bytes or oldest age breaches budget

Cost and verification

Listing U open uploads is O(U) and may need pagination; part inspection adds work proportional to the selected upload's parts. Storage cost grows with incomplete bytes and time until abort, while an aggressive threshold increases re-upload bandwidth and CPU. Measure abort count, successful completion age, duplicate upload identifiers, and bytes reclaimed. The right threshold is a service-specific tradeoff, not a copied default.

Common Mistakes

  • Do not infer zero abandoned storage from a normal object listing.
  • Do not acknowledge ingest before completing and verifying the object.
  • Do not set cleanup age shorter than supported slow-link transfers.

Connected lessons

Practice and check

devops
object-storage
Storage details