Local ephemeral storage: account for logs, writable layers, and emptyDir

Last updated: 5 Oct 20266 min read

tutorial

Kubernetes local ephemeral-storage requests influence scheduling, while limits can trigger eviction when measured usage exceeds the allowed amount. Non-memory emptyDir contents, container writable layers, and container logs can contribute to local storage use. A tmpfs emptyDir instead consumes memory. Files deleted while still open can be missed by directory-scan accounting, so measured use and actual node pressure may diverge.

Operational decision

A document-import Pod expands compressed files before indexing. Estimate peak expanded bytes, application logs during a failed import, writable-layer changes, and the temporary overlap of a rolling update. Set an explicit ephemeral-storage request and limit, plus a smaller emptyDir sizeLimit for import scratch. The fragment is a container-and-volume portion of a Pod template. Fill scratch in a disposable node pool, then separately generate logs and writable-layer files; observe which limit and eviction signal appears in each case. Test the same workload with memory-backed emptyDir and confirm the memory budget, not disk budget, becomes the constraint. Inspect node filesystem layout because a kubelet that cannot measure local ephemeral storage as configured may not enforce a nominal per-Pod limit even while node disk pressure still evicts Pods. Add alerting for node free space and the importer's scratch high-water mark. Avoid retrying the same oversized archive indefinitely after eviction.

spec:
  containers:
    - name: document-importer
      resources:
        requests:
          ephemeral-storage: 880Mi
        limits:
          ephemeral-storage: 2Gi
      volumeMounts:
        - name: import-scratch
          mountPath: /var/tmp/import
  volumes:
    - name: import-scratch
      emptyDir:
        sizeLimit: 1400Mi

Cost and verification

A larger request lowers the chance of disk contention but reduces how many Pods fit on a node. A lower limit contains one workload's growth but may evict useful work during a legitimate large import. Measure bytes per document, expansion ratio, log rate, evictions, and node free space at surge peak. The chosen numbers are synthetic; derive a real budget from observed high-percentile imports and recovery headroom, and reject input above the safe bound before downloading it.

Common Mistakes

Do not equate emptyDir sizeLimit with the Pod's whole local-storage limit.
Do not classify a tmpfs emptyDir as disk usage; it consumes memory.
Do not assume a deleted-but-open file frees node space immediately.

Connected lessons

Practice and check

Memory failure follow-up

Memory-backed emptyDir: budget file bytes as container memory

Linux storage follow-up

Linux disk pressure: explain missing space before deleting application data

devops

operations