JVM container memory: leave room beyond the Java heap

Operational decision

A payment-ledger worker has a 1536 MiB container limit and peaks at 910 MiB committed heap under an import. Start with a measured nonheap allowance, then choose a heap ceiling that still admits peak direct buffers and thread stacks. In a disposable environment, enable JVM native memory tracking at summary level, record a baseline, and compare it during a representative import. Native tracking cannot explain every allocation made outside the JVM, so compare its figures with process RSS and the container charge. If the application throws Java OutOfMemoryError, preserve its message and heap evidence; if the kernel kills the process, inspect the cgroup limit and all native consumers. Store heap dumps in a bounded, access-controlled location because they may contain customer data.

jcmd 1 VM.native_memory baseline
jcmd 1 VM.native_memory summary.diff
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max

Cost and verification

Native memory tracking adds CPU and memory overhead; measure the diagnostic run before enabling it broadly. A larger heap may reduce garbage collection pressure but leaves less native headroom and can increase restart cost. Track peak RSS, heap occupancy after collection, direct-buffer use, thread count, and OOM events through the same import window. On systems without cgroup v2, use the equivalent runtime and OS counters rather than assuming these paths exist.

Common Mistakes

• Do not set the heap maximum equal to the container limit.
• Do not assume a kernel OOM produces a Java heap dump.
• Do not treat native tracking as a complete accounting of external library allocations.

Connected lessons

• DevOps: delivery, infrastructure, and reliable operations
• Container OOM attribution: separate a limit kill from node eviction
• Kubernetes requests and limits: schedule for real load
• Telemetry redaction: remove sensitive fields before an exporter or sampler sees them

Practice and check

• Project: diagnose a memory failure without moving it to another Pod
• DevOps memory failure decisions