response is ERROR when using 'multiprocessing' #3265
Comments
|
We don't currently support multiprocessing. You may be able to sort of work around this by using two separate instances of w3, however. |
The problem is that even though I use two seperate instances of w3 in different processes, this ERROR still exists just like the above example. Though it doesn't support multiprocessing, it shouldn't occur any errors when using seperate instances in different processes. The key BUG is about when to initialize |
|
Got it. We'll triage and put it in our queue. Thanks for reporting! |
|
@kimroniny this is not reproducible, though I agree that a session cache should exist per provider instance even. If you have another reproducible example, it would help quite a bit. I did start a PR towards strapping a session cache manager to each instance of an http provider (or async http provider) which I think is the most reasonable approach and I hope that solves whatever problem you are seeing. However, on the current with the following code: from web3 import Web3, HTTPProvider
import multiprocessing
from web3._utils.request import (
_session_cache
)
def conn():
w3 = Web3(HTTPProvider(endpoint_uri=HTTP_LOCAL, request_kwargs={"timeout": 1000}))
w3.eth.block_number
print(f"Process: {multiprocessing.current_process().name}")
print(f"cache: {_session_cache._data.items()}")
if __name__ == "__main__":
conn()
proc1 = multiprocessing.Process(target=conn)
proc2 = multiprocessing.Process(target=conn)
proc1.start()
proc2.start()
proc1.join()
proc2.joinSimilar results on |
pip freezeoutputWhat was wrong?
Please include any of the following that are applicable:
How can it be fixed?
explain the output
I print session id in the format
"{current_process().name}, session: {id(session)}".The exception happens when querying
eth.accounts. The response shoule betype: List[Account], while it istype: boolwhich should be the response type ofunlock_account.The detailed exception line is the code. It has obtained the response but faces the error type.
reason
When sending requests, web3 uses the cached session with
SimpleCache(). 'SimpleCache()' is instantiated upon importing the web3 and won't be re-instantiated whenever creating a newWeb3().In the main process, after executing
conn()the first session is cached. When forking the children process, all objects including the cached session in the main process are also forked. However, the underlying connections of this session will be shared across all processes, which could cause the responses mixed up in different processes.It is an absolutely dangerous BUG !!!
solution
The easiest solution is to change the cache key. Now the key is just tied to the thread identifier which is identical in the above codes. Just add the process identifier into the key.
Solution 1 is not the best because all process are still using the same one
SimpleCache(). I think, it should check whether it is in a new forked process or not. If it is, then a new SimpleCache()` should be instantiated.The text was updated successfully, but these errors were encountered: