Appends to an existing file get reported as Change.added

[lanzz]

Description

When I'm watching a directory for changes, and a new file gets created in it, subsequent appends to that file generate more Change.added events instead of Change.modified for a while, then eventually switch to generating Change.modified events. If the file is already existing before I start watching, it starts emitting Change.modified events straight away.

I'm running the example code provided below in one terminal, while I'm repeatedly running echo foo >> test.log in another, inside the watched directory. I can confirm that the appends happen as intended, with new lines getting added to the file.

This is a problem for my use case, because on Change.modified my logic will just read one line from the already-opened file, while on Change.added it will reopen the file and read all lines from the start (assuming the reason for the Change.added event was because the file was deleted and recreated by whatever process writes to that file). This means that for a while after the file gets created, every line added will result in all lines being reprocessed, resulting in the same lines being processed repeatedly.

Example Code

def debug_awatch(watch_path):
    import asyncio
    import logging
    from watchfiles import awatch

    async def main() -> None:
        async for changeset in awatch(watch_path, debug=True):
            for change, path in changeset:
                print(f'{change!r}: {path}')

    logging.basicConfig(level=logging.DEBUG)
    asyncio.run(main())

Watchfiles Output

In [4]: debug_awatch('./sample_journals')
DEBUG:asyncio:Using selector: KqueueSelector
watcher: FsEventWatcher { paths: 0x6000026b4990, since_when: 18446744073709551615, latency: 0.0, flags: 18, event_handler: 0x600003db4a50, runloop: Some((0x600001db43c0, JoinHandle { .. })), recursive_info: {"/Users/lanzz/Projects/edva/sample_journals": true} }
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Create(File), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=1
DEBUG:watchfiles.main:1 change detected: {(<Change.added: 1>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.added: 1>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Modify(Data(Content)), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=2
DEBUG:watchfiles.main:1 change detected: {(<Change.modified: 2>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.modified: 2>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Modify(Data(Content)), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=2
DEBUG:watchfiles.main:1 change detected: {(<Change.modified: 2>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.modified: 2>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Modify(Data(Content)), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=2
DEBUG:watchfiles.main:1 change detected: {(<Change.modified: 2>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.modified: 2>: /Users/lanzz/Projects/edva/sample_journals/test.log
raw-event=Event { kind: Modify(Data(Content)), paths: ["/Users/lanzz/Projects/edva/sample_journals/test.log"], attr:tracker: None, attr:flag: None, attr:info: None, attr:source: None } change=2
DEBUG:watchfiles.main:1 change detected: {(<Change.modified: 2>, '/Users/lanzz/Projects/edva/sample_journals/test.log')}
<Change.modified: 2>: /Users/lanzz/Projects/edva/sample_journals/test.log
DEBUG:watchfiles.main:rust notify timeout, continuing
DEBUG:watchfiles.main:rust notify timeout, continuing
DEBUG:watchfiles.main:rust notify timeout, continuing
DEBUG:watchfiles.main:rust notify timeout, continuing

Operating System & Architecture

macOS-14.4.1-arm64-arm-64bit
Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000

Environment

No response

Python & Watchfiles Version

python: 3.11.9 (main, May 8 2024, 08:06:17) [Clang 15.0.0 (clang-1500.3.9.4)], watchfiles: 0.21.0

Rust & Cargo Version

No response