Java WatchService OVERFLOW: reconcile against directory state

Last updated: 5 Oct 20265 min read

tutorial

WatchService OVERFLOW means changes may have been discarded. A consumer must compare current directory state with its last accepted snapshot rather than trying to reconstruct missing events.

Operational contract

The event stream does not identify which changes were lost. Treat it as a signal to rescan the registered directory and rebuild the candidate set under the same eligibility rule. A directory listing can itself race with writers or fail; its result is a point-in-time observation, not an atomic filesystem transaction. Keep a scan generation and record failures. The code below demonstrates a bounded reconciliation, refusing a directory with more than 47 visible entries instead of silently truncating it.

Failure case

A depot initially knows 42 names. A burst of writes produces OVERFLOW. The service cannot say whether the remaining five expected manifests were created, renamed, or removed. A fresh listing obtains the current names and replaces the remembered set only if it completes inside the cap. The caller then validates candidate contents using a separate publication rule before counting them as accepted manifests.

Java code

import java.io.IOException;
import java.nio.file.DirectoryStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.HashSet;
import java.util.Set;

public class DirectoryReconciliation {
    public static Set<Path> snapshot(Path staging) throws IOException {
        Set<Path> visible = new HashSet<>();
        try (DirectoryStream<Path> entries = Files.newDirectoryStream(staging)) {
            for (Path candidate : entries) {
                if (visible.size() >= 47) throw new IOException("Listing cap exceeded");
                visible.add(candidate.getFileName());
            }
        }
        return visible;
    }
}

Performance and ownership cost

Rebuilding a set from N directory entries is O(N) expected time and O(N) memory. Comparing it with the old set is also O(N+K) expected for old-set size K. A cap protects memory but deliberately fails completeness when exceeded; increase capacity or page the intake design, never report the capped set as complete.

Common Mistakes

Do not treat OVERFLOW as one ordinary file event.
Do not retain a partial listing as a complete snapshot.
Do not assume a rescan is atomic with ongoing writers.

Connected lessons

java

filesystem observation

watch-service-overflow-reconciliation