ACME issuance: test challenge reachability without burning production orders

Last updated: 1 Oct 20266 min read

tutorial

A renewal controller may be healthy while HTTP routing, DNS propagation, delegated challenge permissions, or issuer rate limits block issuance. Repeating a failed production request faster does not repair a misrouted challenge and may consume an order or validation budget. Separate certificate-expiry time, first retry time, propagation allowance, issuer backoff, and rollback margin. Use a staging or disposable issuer path for negative tests where the provider supports one.

Operational decision

A regional storefront uses DNS-based validation for a wildcard certificate. Before renewal season, the team checks that the delegated challenge zone is writable only by its issuance identity, that TXT records become visible from independent resolvers, and that cleanup does not remove another order's record. In an isolated drill, deny the DNS write permission and verify the Certificate request shows a clear failure without creating a loop of new private keys. Then restore permission and verify one order reaches Ready. A second drill changes the challenge delegation and measures propagation delay. The team tracks issuer response codes and retry guidance rather than hard-coding a guessed rate limit, which may change by provider or account. If renewal is blocked near expiry, use a preapproved fallback certificate or endpoint plan; do not turn off TLS verification to keep traffic moving. Roll new certificates to a canary listener and check the client matrix before replacing every edge. Monitor the shortest remaining lifetime across leaf and intermediates, not only the controller's successful scheduling of a renewal attempt.

Storefront issuance budget
Expiry: leaf and chain dates recorded
Challenge: delegated DNS write and public visibility tested
Retry: issuer response and backoff respected
Margin: propagation, deployment, and rollback time reserved
Canary: client handshake matrix passes
Fallback: approved certificate or endpoint, never no-verify

Cost and verification

For D domains and R validation vantage points, challenge verification requires at least O(D × R) lookups; large fleets also consume DNS updates and issuer requests. Retrying every second can increase load without reducing propagation time. Measure time from renewal eligibility to issued certificate, failed challenge causes, issuer throttling, and time from issue to serving. Keep separate alerts for issuance failure and deployed certificate expiry because their remediation paths differ.

Common Mistakes

Do not hammer the production issuer after a challenge routing failure.
Do not count a Ready certificate as deployed until a canary handshake proves it.
Do not disable peer validation as an emergency renewal workaround.

Connected lessons

Practice and check

devops

tls