Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filesystem backend: Add optional file lock for multiprocess usage #603

Open
JWCook opened this issue Apr 20, 2022 · 3 comments
Open

Filesystem backend: Add optional file lock for multiprocess usage #603

JWCook opened this issue Apr 20, 2022 · 3 comments
Labels
backends Features or changes related to specific backends enhancement performance

Comments

@JWCook
Copy link
Member

JWCook commented Apr 20, 2022

Currently, the filesystem backend uses a simple threading lock for thread safety. To be safe for multiple processes, it will need a file-level locking mechanism, like py-filelock. That could be added as an optional dependency, and enabled with a keyword argument.

Please comment if this is something you have a use case for!

@JWCook JWCook added this to the v1.0 milestone Apr 20, 2022
@JWCook JWCook added performance backends Features or changes related to specific backends labels Apr 20, 2022
@JWCook JWCook removed this from the v1.0 milestone Apr 20, 2022
@JWCook JWCook added this to the Long-term backlog milestone Jun 20, 2022
@rgammans
Copy link

I've used it with (a monkey patch to use) multiprocessing.RLock() as my use case would normally be ok with threading but if I cache I'm CPU bound so I need multiprocessing so I can avoid GIL throttling

@JWCook
Copy link
Member Author

JWCook commented Oct 20, 2022

Note that multiprocessing.RLock will work only if you have child processes spawned from a common parent process; for concurrent access from multiple unrelated processes, a file-level lock is required.

@rgammans
Copy link

Note that multiprocessing.RLock will work only if you have child processes spawned from a common parent process; for concurrent access from multiple unrelated processes, a file-level lock is required.

True, but that was what I was doing,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backends Features or changes related to specific backends enhancement performance
Projects
None yet
Development

No branches or pull requests

2 participants