Project: test GitOps render, ownership, prune, and rollback boundaries

Last updated: 5 Oct 202610 min read

project

Use disposable clusters and repositories with a small invoice API. Put its workload in a Kustomize base, create two environment overlays, and reconcile each through a GitOps controller. The exercise is accepted only when source revisions and rendered objects match the approved candidate, one controller owns each resource, unexpected deletion is blocked, a customer-path probe passes, and a rollback returns both desired state and controller authority to a known-good release.

Prove the rendered candidate

Pin the image by digest and every external chart, base, or values input by immutable revision. Render each overlay with a recorded tool version, compute a resource inventory, and compare it with the last accepted release. Change a shared base without editing production's overlay and verify production is still checked. Move an upstream values reference while holding the application commit fixed; the candidate must be detected as a different render. Add a patch whose target no longer matches and require a visible failure or review warning. Compare the controller's observed artifact with the approved input and inspect the running image after reconciliation.

Test ownership and pruning

Create a shared ConfigMap that both Applications try to manage. Confirm collision detection or a deliberate ownership inventory catches it before two controllers alternately write the object. Transfer the ConfigMap to one owner with the old owner's deletion path disabled and verify its data and object identity after the old Application is removed. Remove a Deployment from the new render and inspect the proposed prune set. Then point the source at an empty directory and show the release guard blocks mass deletion. Keep one PVC and a namespace under a reviewed retention rule; verify the actual objects remain after the test, not just the controller status.

Invoice GitOps acceptance
All source, renderer, values, and image inputs pinned
Rendered resource identities match approved inventory
No two Applications claim the same managed resource
Unexpected empty or large prune set blocked
Sync, health, runtime image, and invoice transaction pass
Source outage reports stale artifact without restarting service
Rollback changes declared intent and restores normal sync
Fleet test holds later clusters after one failed cohort

Separate health from customer success

Delay delivery of the invoice-signing Secret while leaving the Deployment healthy. Require the synthetic invoice issue-and-read journey to fail the release gate. Then make the Git source unavailable after a successful revision and submit a new intent change; observe whether the service can continue on last-good while the pending change is clearly marked unapplied. Restore source access and confirm that the intended revision, not an unreviewed intermediate head, is applied. Keep timestamps for source fetch, sync, resource health, and user success so a false-green interval is visible.

Rollback and bound fleet impact

Deploy a bad configuration and attempt a direct live patch without changing Git. Observe whether reconciliation replaces it. Perform the reviewed rollback through desired-state change, verify image and data compatibility, and resume ordinary sync. Repeat across several disposable clusters in two or more cohorts; force the first cohort to fail and prove later targets do not advance. Record unselected clusters, source age, resource diffs, prune candidates, rollback time, and customer-path failures. Mark provider- or controller-version-specific behavior unverified if the disposable environment does not reproduce it.

Common Mistakes

Do not approve a patch without seeing its full rendered output.
Do not delete an Application before inspecting its resource-retention behavior.
Do not count Synced as proof an invoice can be issued.

Connected lessons

devops

project