Notes on Asynchronous File Watching

In an ETL modernization project I led, one of the pieces we built was a readiness-criteria system: a pipeline stage wouldn't trigger until its input files were confirmed present and stable. The implementation used file watching. It looked simple. It was not.

These are notes from that work — specific things that surprised me, and the patterns that handled them.

The kernel APIs

On Linux, file watching is built on inotify. You open a watch descriptor on a path and receive events: IN_CREATE, IN_MODIFY, IN_DELETE, IN_MOVED_FROM, IN_MOVED_TO. On macOS the equivalent is FSEvents, which operates at the directory level and batches events. On Windows, ReadDirectoryChangesW provides similar functionality with a slightly different event model.

Most library abstractions — Python's watchdog, Node's chokidar — sit on top of these and smooth over the platform differences. They also inherit the limitations, which the abstractions don't always surface clearly.

The rename problem

The most common pattern for atomic file writes is: write to a temp file, rename to the target path. This is safe against partial reads because the rename is atomic on POSIX systems. But it means the file appears via a MOVED_TO event, not a CREATE event. Libraries that only watch for CREATE will miss it.

The reliable pattern is to watch for both CREATE and MOVED_TO (or their library equivalents) and treat both as "file appeared." More importantly: don't assume a CREATE event means the file is complete. It means the file exists. A process writing a large file will generate a CREATE event at the start and a series of MODIFY events as it writes. If you try to process the file on CREATE, you'll process a partial file.

Debouncing is not optional

Text editors, build tools, and ETL producers all generate bursts of events during a write. A single save in an editor can generate MODIFY, DELETE, CREATE, and MODIFY events in rapid succession depending on how the editor handles atomic writes. Processing each event would trigger your handler multiple times for a single logical operation.

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
import threading

class DebouncedHandler(FileSystemEventHandler):
    def __init__(self, callback, delay=0.5):
        self.callback = callback
        self.delay = delay
        self._timers = {}

    def on_any_event(self, event):
        if event.is_directory:
            return
        path = event.src_path
        if path in self._timers:
            self._timers[path].cancel()
        t = threading.Timer(self.delay, self.callback, args=[path])
        self._timers[path] = t
        t.start()

Network filesystems lie

inotify does not work reliably on network filesystems — NFS, SMB, or most cloud storage mounts. Events may be delayed, duplicated, or not delivered at all, depending on the mount configuration and the server implementation. If your files land on a network share, polling is more reliable than event-based watching, even though it's less efficient.

The signal to look for: if your file watcher works perfectly in local development and has mysterious misses in the environment where files come from an upstream system via a mount, the mount is probably the issue. Add logging at the event level before assuming your handler logic is wrong.

Large directory watches

inotify has a system-level limit on the number of watches (/proc/sys/fs/inotify/max_user_watches, default 8192 on most systems). Watching a large directory tree recursively creates one watch per directory. In our ETL setup, the input directory tree had several thousand subdirectories, which hit this limit quickly.

The fix is either to increase the limit (sysctl fs.inotify.max_user_watches=524288) or to restructure the watch to cover only the directories where files actually land, rather than the whole tree. We did both — raised the limit and flattened the input directory structure so we were watching fewer paths.

What I'd do differently

For trigger-based pipelines where reliability matters more than latency, I'd consider a different architecture: have the producer write a manifest file or send a message to a queue when a file is ready, rather than having the consumer watch for the file directly. File watching is best suited for developer tooling (build systems, hot reload) where missed events are annoying but not catastrophic. For production ETL, the queue model is more auditable, more reliable across network filesystems, and easier to reason about under failure conditions.