python - win32file.ReadDirectoryChangesW doesn't find all moved files

Question

Good morning,

I've come across a peculiar problem with a program I'm creating in Python. It appears that when I drag and drop files from one location to another, not all of the files are registered as events by the modules.

I've been working with win32file and win32con to try an get all events related to moving files from one location to another for processing.

Here is a snip bit of my detection code:

import win32file
import win32con
def main():
    path_to_watch = 'D:\'
    _file_list_dir = 1
    # Create a watcher handle
    _h_dir = win32file.CreateFile(
        path_to_watch,
        _file_list_dir,
        win32con.FILE_SHARE_READ |
        win32con.FILE_SHARE_WRITE |
        win32con.FILE_SHARE_DELETE,
        None,
        win32con.OPEN_EXISTING,
        win32con.FILE_FLAG_BACKUP_SEMANTICS,
        None
    )
    while 1:
        results = win32file.ReadDirectoryChangesW(
            _h_dir,
            1024,
            True,
            win32con.FILE_NOTIFY_CHANGE_FILE_NAME |
            win32con.FILE_NOTIFY_CHANGE_DIR_NAME |
            win32con.FILE_NOTIFY_CHANGE_ATTRIBUTES |
            win32con.FILE_NOTIFY_CHANGE_SIZE |
            win32con.FILE_NOTIFY_CHANGE_LAST_WRITE |
            win32con.FILE_NOTIFY_CHANGE_SECURITY,
            None,
            None
        )
        for _action, _file in results:
            if _action == 1:
                print 'found!'
            if _action == 2:
                print 'deleted!'

I dragged and dropped 7 files and it only found 4.

# found!
# found!
# found!
# found!

What can I do to detect all dropped files?

Answer 1

[ActiveState.Docs]: win32file.ReadDirectoryChangesW (this is the best documentation that I could find for [GitHub]: mhammond/pywin32 - Python for Windows (pywin32) Extensions) is a wrapper over [MS.Docs]: ReadDirectoryChangesW function. Here's what it states (about the buffer):

1. General

When you first call ReadDirectoryChangesW, the system allocates a buffer to store change information. This buffer is associated with the directory handle until it is closed and its size does not change during its lifetime. Directory changes that occur between calls to this function are added to the buffer and then returned with the next call. If the buffer overflows, the entire contents of the buffer are discarded, the lpBytesReturned parameter contains zero, and the ReadDirectoryChangesW function fails with the error code ERROR_NOTIFY_ENUM_DIR.

• My understanding is that this is a different buffer than the one passed as an argument (lpBuffer):

  • The former is passed to every call of ReadDirectoryChangesW (could be different buffers (with different sizes) passed for each call)

  • The latter is allocated by the system, when the former clearly is allocated (by the user) before the function call
    and that is the one that stores data (probably in some raw format) between function calls, and when the function is called, the buffer contents is copied (and formatted) to lpBuffer (if not overflew (and discarded) in the meantime)

2. Synchronous

Upon successful synchronous completion, the lpBuffer parameter is a formatted buffer and the number of bytes written to the buffer is available in lpBytesReturned. If the number of bytes transferred is zero, the buffer was either too large for the system to allocate or too small to provide detailed information on all the changes that occurred in the directory or subtree. In this case, you should compute the changes by enumerating the directory or subtree.

• This somewhat confirms my previous assumption

  • "the buffer was either too large for the system to allocate" - maybe when the buffer from previous point is allocated, it takes into account nBufferLength?

Anyway, I took your code and changed it "a bit".

code00.py:

import sys
import msvcrt
import pywintypes
import win32file
import win32con
import win32api
import win32event


FILE_LIST_DIRECTORY = 0x0001
FILE_ACTION_ADDED = 0x00000001
FILE_ACTION_REMOVED = 0x00000002

ASYNC_TIMEOUT = 5000

BUF_SIZE = 65536


def get_dir_handle(dir_name, asynch):
    flags_and_attributes = win32con.FILE_FLAG_BACKUP_SEMANTICS
    if asynch:
        flags_and_attributes |= win32con.FILE_FLAG_OVERLAPPED
    dir_handle = win32file.CreateFile(
        dir_name,
        FILE_LIST_DIRECTORY,
        (win32con.FILE_SHARE_READ |
         win32con.FILE_SHARE_WRITE |
         win32con.FILE_SHARE_DELETE),
        None,
        win32con.OPEN_EXISTING,
        flags_and_attributes,
        None
    )
    return dir_handle


def read_dir_changes(dir_handle, size_or_buf, overlapped):
    return win32file.ReadDirectoryChangesW(
        dir_handle,
        size_or_buf,
        True,
        (win32con.FILE_NOTIFY_CHANGE_FILE_NAME |
         win32con.FILE_NOTIFY_CHANGE_DIR_NAME |
         win32con.FILE_NOTIFY_CHANGE_ATTRIBUTES |
         win32con.FILE_NOTIFY_CHANGE_SIZE |
         win32con.FILE_NOTIFY_CHANGE_LAST_WRITE |
         win32con.FILE_NOTIFY_CHANGE_SECURITY),
        overlapped,
        None
    )


def handle_results(results):
    for item in results:
        print("    {} {:d}".format(item, len(item[1])))
        _action, _ = item
        if _action == FILE_ACTION_ADDED:
            print("    found!")
        if _action == FILE_ACTION_REMOVED:
            print("    deleted!")


def esc_pressed():
    return msvcrt.kbhit() and ord(msvcrt.getch()) == 27


def monitor_dir_sync(dir_handle):
    idx = 0
    while True:
        print("Index: {:d}".format(idx))
        idx += 1
        results = read_dir_changes(dir_handle, BUF_SIZE, None)
        handle_results(results)
        if esc_pressed():
            break


def monitor_dir_async(dir_handle):
    idx = 0
    buffer = win32file.AllocateReadBuffer(BUF_SIZE)
    overlapped = pywintypes.OVERLAPPED()
    overlapped.hEvent = win32event.CreateEvent(None, False, 0, None)
    while True:
        print("Index: {:d}".format(idx))
        idx += 1
        read_dir_changes(dir_handle, buffer, overlapped)
        rc = win32event.WaitForSingleObject(overlapped.hEvent, ASYNC_TIMEOUT)
        if rc == win32event.WAIT_OBJECT_0:
            bufer_size = win32file.GetOverlappedResult(dir_handle, overlapped, True)
            results = win32file.FILE_NOTIFY_INFORMATION(buffer, bufer_size)
            handle_results(results)
        elif rc == win32event.WAIT_TIMEOUT:
            #print("    timeout...")
            pass
        else:
            print("Received {:d}. Exiting".format(rc))
            break
        if esc_pressed():
            break
    win32api.CloseHandle(overlapped.hEvent)


def monitor_dir(dir_name, asynch=False):
    dir_handle = get_dir_handle(dir_name, asynch)
    if asynch:
        monitor_dir_async(dir_handle)
    else:
        monitor_dir_sync(dir_handle)
    win32api.CloseHandle(dir_handle)


def main():
    print("Python {:s} on {:s}
".format(sys.version, sys.platform))
    asynch = True
    print("Attempting {}ynchronous mode using a buffer {:d} bytes long...".format("As" if async else "S", BUF_SIZE))
    monitor_dir(".\test", asynch=asynch)


if __name__ == "__main__":
    main()

Notes:

• Used constants wherever possible
• Split your code into functions so it's modular (and also to avoid duplicating it)
• Added print statements to increase output
• Added the asynchronous functionality (so the script doesn't hang forever if no activity in the dir)
• Added a way to exit when user presses ESC (of course in synchronous mode an event in the dir must also occur)
• Played with different values for different results

Output:

e:WorkDevStackOverflowq049799109>dir /b test
0123456789.txt
01234567890123456789.txt
012345678901234567890123456789.txt
0123456789012345678901234567890123456789.txt
01234567890123456789012345678901234567890123456789.txt
012345678901234567890123456789012345678901234567890123456789.txt
0123456789012345678901234567890123456789012345678901234567890123456789.txt
01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt
0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt

e:WorkDevStackOverflowq049799109>
e:WorkDevStackOverflowq049799109>"C:Installx64HPEOPSWpython2.7.10__00python.exe" code00.py
Python 2.7.10 (default, Mar  8 2016, 15:02:46) [MSC v.1600 64 bit (AMD64)] on win32

Attempting Synchronous mode using a buffer 512 bytes long...
Index: 0
    (2, u'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt') 104
    deleted!
Index: 1
    (2, u'012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt') 94
    deleted!
Index: 2
    (2, u'01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt') 84
    deleted!
Index: 3
    (2, u'0123456789012345678901234567890123456789012345678901234567890123456789.txt') 74
    deleted!
    (2, u'012345678901234567890123456789012345678901234567890123456789.txt') 64
    deleted!
Index: 4
    (2, u'01234567890123456789012345678901234567890123456789.txt') 54
    deleted!
Index: 5
    (2, u'0123456789012345678901234567890123456789.txt') 44
    deleted!
    (2, u'012345678901234567890123456789.txt') 34
    deleted!
Index: 6
    (2, u'01234567890123456789.txt') 24
    deleted!
    (2, u'0123456789.txt') 14
    deleted!
Index: 7
    (1, u'0123456789.txt') 14
    found!
Index: 8
    (3, u'0123456789.txt') 14
Index: 9
    (1, u'01234567890123456789.txt') 24
    found!
Index: 10
    (3, u'01234567890123456789.txt') 24
    (1, u'012345678901234567890123456789.txt') 34
    found!
    (3, u'012345678901234567890123456789.txt') 34
    (1, u'0123456789012345678901234567890123456789.txt') 44
    found!
Index: 11
    (3, u'0123456789012345678901234567890123456789.txt') 44
    (1, u'01234567890123456789012345678901234567890123456789.txt') 54
    found!
    (3, u'01234567890123456789012345678901234567890123456789.txt') 54
Index: 12
Index: 13
    (1, u'01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt') 84
    found!
Index: 14
Index: 15
    (1, u'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt') 104
    found!
Index: 16
    (3, u'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt') 104
Index: 17
    (1, u'a') 1
    found!
Index: 18
    (3, u'a') 1

e:WorkDevStackOverflowq049799109>
e:WorkDevStackOverflowq049799109>"C:Installx64HPEOPSWpython2.7.10__00python.exe" code00.py
Python 2.7.10 (default, Mar  8 2016, 15:02:46) [MSC v.1600 64 bit (AMD64)] on win32

Attempting Synchronous mode using a buffer 65536 bytes long...
Index: 0
    (2, u'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt') 104
    deleted!
Index: 1
    (2, u'012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt') 94
    deleted!
Index: 2
    (2, u'01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt') 84
    deleted!
Index: 3
    (2, u'0123456789012345678901234567890123456789012345678901234567890123456789.txt') 74
    deleted!
Index: 4
    (2, u'012345678901234567890123456789012345678901234567890123456789.txt') 64
    deleted!
Index: 5
    (2, u'01234567890123456789012345678901234567890123456789.txt') 54
    deleted!
Index: 6
    (2, u'0123456789012345678901234567890123456789.txt') 44
    deleted!
Index: 7
    (2, u'012345678901234567890123456789.txt') 34
    deleted!
    (2, u'01234567890123456789.txt') 24
    deleted!
    (2, u'0123456789.txt') 14
    deleted!
Index: 8
    (1, u'0123456789.txt') 14
    found!
Index: 9
    (3, u'0123456789.txt') 14
Index: 10
    (1, u'01234567890123456789.txt') 24
    found!
Index: 11
    (3, u'01234567890123456789.txt') 24
Index: 12
    (1, u'012345678901234567890123456789.txt') 34
    found!
Index: 13
    (3, u'012345678901234567890123456789.txt') 34
Index: 14
    (1, u'01234567890123456789012345678