Clock skew: verify time before debugging credentials and leases

Last updated: 5 Oct 20266 min read

tutorial

Clock skew is the difference between a machine's clock and a trusted time reference. A clock that is too far ahead or behind can make a valid certificate appear expired or not yet valid, can invalidate time-bounded credentials, and can confuse event ordering. Monotonic timers are appropriate for elapsed deadlines; wall-clock timestamps are needed for external validity windows but depend on synchronization.

Operational decision

A claims worker begins rejecting short-lived service tokens after a host replacement. Before rotating every credential, read UTC time and the time service's tracking state on both the issuing and consuming hosts. The shell fragment uses chrony on a Linux host; use the installed time service's equivalent elsewhere. Compare measured offset and synchronization status with the token's issued-at and expiry fields without logging the token itself. Check whether the VM resumed from a paused state or lost its time source. Restore time synchronization through the approved host procedure, then issue a new test token and verify one authorized request. If lease ownership changed during the skew, inspect the old holder and durable effects before resuming work; simply correcting the clock does not undo operations performed under a disputed lease. Alert on offset beyond the system's tested tolerance, not a guessed universal number.

date -u
chronyc tracking
chronyc sources -v

Cost and verification

A highly available time service uses network and operational effort, but avoids failures that otherwise look like unrelated TLS, identity, or scheduling faults. Aggressive clock stepping can disturb applications that use wall time for intervals; test the host procedure. Keep token lifetimes short enough for security needs yet longer than expected network and clock uncertainty. Record offset, last synchronization, and failed authorization rate together. Do not copy credential contents into the incident record while investigating time.

Common Mistakes

Do not rotate every secret before checking host time when many tokens fail together.
Do not use wall-clock subtraction for local elapsed-time deadlines.
Do not assume clock correction reverses effects of a disputed lease.

Connected lessons

Practice and check

Linux host change follow-up

SSH user certificates: bind host login to an expiry and a narrow principal

devops

operations