Inode exhaustion: diagnose a full filesystem with free bytes

Operational decision

An invoice exporter writes one temporary file per item and leaves files behind after failed uploads. On the affected host or mounted volume, compare byte and inode usage with the read-only commands. Count files in the exporter-owned spool only after identifying its mount; a recursive count can itself be expensive on a severely loaded filesystem. Capture a sample of file age and ownership before cleanup. Stop new export intake, move recoverable work to the documented queue, and remove only files that the application has marked complete. Then verify a synthetic export can create, upload, and delete its temporary file. A restart may move the failure to another node while leaving the original disk full. Set a retention and cleanup policy with an alert on inode use and oldest incomplete export age; byte alerts alone will miss the next incident.

df -h /var/lib/invoice-exporter
df -i /var/lib/invoice-exporter
find /var/lib/invoice-exporter/spool -maxdepth 1 -type f | wc -l

Cost and verification

Removing completed files returns inodes but can erase evidence if the completion state is uncertain. Larger filesystems or fewer files per batch may increase storage cost and operational complexity. Recursive scans consume I/O and may worsen an incident; prefer application counters or bounded directory inspection during routine monitoring. Track both inode and byte utilization, plus work-item age, so the alert points to business impact rather than a single storage number.

Common Mistakes

• Do not conclude a filesystem has room because df reports free bytes.
• Do not delete spool files before determining whether their work completed.
• Do not run repeated full-tree scans on a failing volume.

Connected lessons

• DevOps: delivery, infrastructure, and reliable operations
• Linux service diagnostics: process, socket, and journal
• Node pressure eviction: trace lost Pods to exhausted local resources
• Log pipelines: preserve incident evidence without ingesting secrets
• Queue consumers: acknowledgement, idempotency, and backlog

Practice and check

• Project: recover a host and protocol failure chain
• DevOps host and protocol failure decisions

Linux storage follow-up

• Linux disk pressure: explain missing space before deleting application data