A Lease object records a leader candidate's ownership and renewal time. It helps controllers coordinate active work, but a paused process can resume after its lease has expired and another candidate has taken over. The old process may still have network access to an external database or provider. The lease alone cannot make that external side effect safe; the effect path needs an epoch or another ownership check the old leader cannot pass.
Make this comfortable
Leader leases: fence side effects after ownership changes
Operational decision
Two reconciliation workers manage one regional payment route. Inspect the Lease holder and renewal state with the read-only command, then pause the active worker in a disposable test long enough for a standby to take leadership. On resumption, the old worker must recheck its ownership before scheduling more work and must not commit an already prepared route change under an obsolete epoch. A durable route record can require a monotonically increasing generation on every write; the downstream store rejects a lower generation even if the old process is alive. Record lease transitions, accepted and rejected generations, and the actual provider route state. Test a second interruption after the external call succeeds but before the controller records success, using the same operation key to reconcile rather than issuing a new conflicting change. A single current Lease holder is necessary evidence for coordination but insufficient evidence that no stale worker committed an effect.
kubectl get lease -n payment-system payment-route-leader -o yaml
kubectl get pods -n payment-system -l app=payment-route-controller -o wideCost and verification
A shorter lease speeds failover but increases sensitivity to API latency and clock or scheduling pauses. A longer lease slows recovery after a real failure. A durable fencing write adds storage cost and latency, yet prevents a stale owner from corrupting external state. Test the lease duration against observed control-plane delays and stop work when renewal cannot be confirmed. Audit the effect destination, not only the Lease object's holder field.
Common Mistakes
- Do not infer a prior leader stopped executing because its Lease expired.
- Do not reuse an old epoch after leadership changes.
- Do not retry an ambiguous external mutation under a new operation key.
Connected lessons
- DevOps: delivery, infrastructure, and reliable operations
- Consumer rebalances: preserve ordering and effect ownership
- Failover fencing: prevent two writable database leaders
- Ambiguous cloud creates: reconcile before repeating a timed-out mutation
- Clock skew: verify time before debugging credentials and leases
Practice and check
devops
operations