Linux host change drill: reboot, access, firewall, and disk recovery

Last updated: 5 Oct 202613 min read

project

Use disposable Linux virtual machines with console access, a simulated bastion, a synthetic API, and a cloned encrypted volume. Version-control the host configuration and keep an external ledger of expected and observed states. Use dedicated test identities and test data. The exercise is complete only when a failed change can be reversed without relying on the access path that it broke.

Stage the reboot boundary

Record the running kernel, installed candidate, selected boot entry, API acceptance, and spare capacity. Drain one canary, reboot into the approved kernel, and check a synthetic write before returning it to service. Inject a boot failure on a clone and select the retained prior entry from the console. Repeat in a five-host cohort while preserving workload quorum. A package-install receipt without a new running kernel fails this gate.

Challenge every access path

Rotate the bastion's SSH host key with an overlap and a fingerprint distributed through trusted inventory. Prove strict-check connections work for managed and ordinary clients; an unexpected fingerprint must stop the change. Issue one 27-minute user certificate for a narrow principal and reject an expired certificate and a wrong-account login. Allow a responder to run one root-owned maintenance helper through sudo, then reject a substituted service name, an added argument, and an operator-writable helper.

Acceptance ledger
Kernel: approved running release after reboot; prior entry bootable
SSH server: fingerprint verified; strict clients connect; old key removed after overlap
SSH user: db-operator principal accepted only for maintenance account
Sudo: one helper accepted; shell, extra arguments, and writable replacement denied
Firewall: independent probes pass; bad rule triggers timed restore
LUKS: cloned volume unlocks with recovery path and matching header backup

Recover the network and disk

Load a checked nftables candidate on a canary only after arming a timed restore from outside the SSH session. Inject a rule that blocks the bastion subnet and observe automatic recovery. Confirm the persistent ruleset survives reboot and that IPv6 was tested. On the cloned LUKS volume, rotate an unlock slot, prove cold boot recovery, then simulate header damage and restore a protected matching backup. Verify file hashes from an independent data backup; unlocking alone is not the completion signal.

Common Mistakes

Do not run the recovery drill against the only copy of production data.
Do not let the same SSH session be the sole path to reversing its own firewall rule.
Do not count a configuration check as a successful login, reboot, or restore.

Connected lessons

devops

project