GitOps verification: separate source sync, resource health, and user success

Last updated: 5 Oct 20266 min read

tutorial

Source freshness says which revision the controller fetched. Sync compares rendered desired state with tracked live fields. Resource health interprets rollout conditions. None alone proves authentication, database access, queue processing, or edge routing. A release gate should join those observations to a synthetic transaction and to the immutable artifact actually running, while understanding that ignored differences may hide fields the application still uses.

Operational decision

A claims-upload release reaches the cluster and its Deployment becomes available. The application is still failing uploads because an external secret refresher has not delivered the new signing key. The release record captures source revision, rendered digest, controller sync result, Deployment observed generation and available replicas, secret generation, running image ID, and a synthetic upload that reads the uploaded claim back. The gate fails despite a green controller status. In another drill, a custom resource appears unknown or healthy because its health check is too permissive; add a check for the controller's observed generation and a workload-level probe. If an autoscaler intentionally changes replica count, narrow the ignored-difference rule to that field and confirm that image, ServiceAccount, and security fields remain compared. Test a missing Secret and a failing edge route; the release should give a specific failure state instead of an undifferentiated 'sync failed.' A user-path probe must use disposable data and should report its own error separately from controller health so incident responders know whether the problem is rendering, rollout, dependency delivery, or routing.

Claims release acceptance
Source: approved revision fetched
Render: accepted digest observed
Sync: no unexpected field differences
Health: observed generation and replicas ready
Runtime: image ID matches tested digest
Dependency: expected secret generation delivered
User path: upload accepted and queryable

Cost and verification

Collecting M status signals per release is O(M); a synthetic transaction adds external calls and cleanup work. Probes that are too frequent or high-cardinality can create load and telemetry cost, so use a bounded cadence and stable labels. Measure false-green releases, time from sync to user success, stale source age, ignored field scope, and failure classification. A green sync is useful evidence, but it is a configuration statement rather than a service-level acceptance test.

Common Mistakes

Do not declare a release successful at the first Synced status.
Do not ignore an entire resource to suppress one expected field difference.
Do not use a local readiness response as the only customer-path test.

Connected lessons

Practice and check

devops

gitops