ValueError: {'code': -32700, 'message': 'parse error'} and Can't pickle local object 'construct_web3_formatting_middleware'

[fridary]

What happened?

Erigon: 2.59.3-088fd8ef (last March 2024)

I have a problem to parallel querying local Erigon node. By the end of execution there is an error ValueError: {'code': -32700, 'message': 'parse error'}.
Error is always returned on last request, whether I query 4 or 50 transactions in Pool.
Although, if I test code on public node (https://eth-pokt.nodies.app in example) instead of 'http://127.0.0.1:8545', problem is gone.
Yes, perhaps problem is in Erigon. But if I do requests.post(), there is no error.
If I do another query like w3.eth.get_block(block), error is the same.
Any ideas what's wrong?
Update: it seems problem is in declaring w3 outside pool process together with inside (calculate_transaction()) for local nodes.

Erigon:
/home/fridary/erigon/build/bin/erigon --internalcl --datadir=/disk_sde/erigon --http.api=eth,erigon,engine,web3,net,debug,trace,txpool --authrpc.jwtsecret=/home/fridary/erigon/jwtsecret --metrics --prune.h.before=13916166 --prune.r.before=13916166 --prune.t.before=13916166 --prune.c.before=13916166 --torrent.download.rate=128mb

Code that produced the error

import requests
from web3 import Web3
from multiprocessing import Pool, cpu_count


# testnet = 'https://eth-pokt.nodies.app'
testnet = 'http://127.0.0.1:8545' # here is my local Erigon node


# def calculate_transaction(w3, i, hash_): # this gives error Can't pickle local object 'construct_web3_formatting_middleware'
def calculate_transaction(i, hash_):
    global w3

    # gives error
    transaction = w3.eth.get_transaction(hash_)

    # no error
    # transaction = requests.post(testnet, json={"method":"eth_getTransactionByHash","params":[hash_.hex()],"id":i,"jsonrpc":"2.0"}, headers={"Content-Type": "application/json"}).json()['result']

    print(i, transaction['hash'].hex())



w3 = Web3(Web3.HTTPProvider(testnet))
if not w3.is_connected():
    exit("w3 not connected")

block = w3.eth.get_block('latest')
transactions = block['transactions'][:min(cpu_count(), 4)]
    
params = []
for i, hash_ in enumerate(transactions):
    params.append((i, hash_))

pool = Pool(min(cpu_count(), 4))
pool.starmap_async(calculate_transaction, params).get()
pool.close()

Full error output

2 0x05aee6ea3d6a19b76e810e481736b16487a252525efc1640289b97e29eaafe08
3 0xbfc4114da15e7944038571362c191859f9a75655ed7a265fa9623102823851df
0 0x9a2b77146b8ab62fe9b1d50ca353f5faf841d744029ea913f76a702c79ec648c
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/python/ethereum/parallel_3.py", line 16, in calculate_transaction
    transaction = w3.eth.get_transaction(hash_)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/site-packages/web3/eth/eth.py", line 325, in get_transaction
    return self._get_transaction(transaction_hash)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/site-packages/web3/module.py", line 75, in caller
    result = w3.manager.request_blocking(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/site-packages/web3/manager.py", line 329, in request_blocking
    return self.formatted_response(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/site-packages/web3/manager.py", line 292, in formatted_response
    raise ValueError(error)
ValueError: {'code': -32700, 'message': 'parse error'}
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fridary/python/ethereum/parallel_3.py", line 33, in <module>
    pool.starmap_async(calculate_transaction, params).get()
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
ValueError: {'code': -32700, 'message': 'parse error'}

### Fill this section in if you know how this could or should be fixed

_No response_

### web3 Version

6.16.0

### Python Version

3.11.8

### Operating System

Ubuntu 20.04.6 LTS

### Output from `pip freeze`

```shell
aiohttp==3.9.3
aiosignal==1.3.1
alchemy-sdk-py==0.2.0
attributedict==0.3.0
attrs==23.2.0
bitarray==2.9.2
blessings==1.7
cachetools==5.3.3
certifi==2024.2.2
chardet==5.2.0
charset-normalizer==3.3.2
codecov==2.1.13
colorama==0.4.6
coloredlogs==15.0.1
colour-runner==0.1.1
coverage==7.4.4
cytoolz==0.12.3
deepdiff==6.7.1
distlib==0.3.8
eth-abi==4.2.1
eth-account==0.11.0
eth-hash==0.7.0
eth-keyfile==0.8.0
eth-keys==0.5.0
eth-rlp==1.0.1
eth-typing==4.0.0
eth-utils==4.0.0
filelock==3.13.3
frozendict==2.3.10
frozenlist==1.4.1
hexbytes==0.3.1
humanfriendly==10.0
idna==3.6
inspecta==0.1.3
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
lru-dict==1.2.0
markdown-it-py==3.0.0
mdurl==0.1.2
moralis==0.1.45
multidict==6.0.5
numpy==1.26.4
ordered-set==4.1.0
packaging==24.0
pandas==2.2.1
parsimonious==0.9.0
platformdirs==4.2.0
pluggy==1.4.0
protobuf==5.26.1
pycryptodome==3.20.0
Pygments==2.17.2
pyproject-api==1.6.1
python-dateutil==2.8.2
python-dotenv==1.0.1
pytz==2024.1
pyunormalize==15.1.0
referencing==0.34.0
regex==2023.12.25
requests==2.31.0
rich==13.7.1
rlp==4.0.0
rootpath==0.1.1
rpds-py==0.18.0
six==1.16.0
tabulate==0.9.0
termcolor==2.4.0
toolz==0.12.1
tox==4.14.2
typing_extensions==4.3.0
tzdata==2024.1
urllib3==1.26.18
virtualenv==20.25.1
web3==6.16.0
web3-input-decoder==0.1.11
websockets==12.0
yarl==1.9.4

[fridary]

Update: I figured out that if I don't declare w3 outside the calculate_transaction() function, I mean if I remove this code:

w3 = Web3(Web3.HTTPProvider(testnet))
if not w3.is_connected():
    exit("w3 not connected")

and if I declare w3 each time/process in calculate_transaction(), error will gone. So the problem is in defining w3 in/outside pool function. But how to fix I don't know, because in real program I have to define w3 also outside the pool processing.
The only reason could be to define w3 once at start and send w3 in func params like def calculate_transaction(w3, i, hash_), but then I will get this error:

Traceback (most recent call last):
  File "/home/fridary/python/ethereum/parallel_4.py", line 53, in <module>
    pool.starmap_async(calculate_transaction, params).get()
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'construct_web3_formatting_middleware.<locals>.formatter_middleware'

Maybe there is any way to close connection w3?

[fridary]

Update: this solution works, but it's ugly option. Creating new w3 every step on another address. Any other ideas?

testnet = 'http://127.0.0.1:8545/'
def calculate_transaction(i):
    w3 = Web3(Web3.HTTPProvider(testnet + f'?{i}'))

[kclowes]

On first glance, it looks like something to do with how we generate the cache key and multiprocessing. I'm glad you found a workaround though. We'll put this in our queue to look into. Thanks for the issue!

[fselmo]

@fridary when I run your example, I do error out but with the following message:

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

        To fix this issue, refer to the "Safe importing of main module"
        section in https://docs.python.org/3/library/multiprocessing.html   

When I do run with the if __name__ == '__main__': pattern I do not get any errors. Try this quick and dirty refactor of your example to see if that resolves it:

from web3 import Web3
from multiprocessing import Pool, cpu_count

w3 = Web3(Web3.HTTPProvider(HTTP_LOCAL))


def calculate_transaction(i, hash_):
    global w3

    # gives error
    transaction = w3.eth.get_transaction(hash_)

    print(i, transaction['hash'].hex())


def main():
    global w3

    if not w3.is_connected():
        exit("w3 not connected")

    block = w3.eth.get_block('latest')
    transactions = block['transactions'][:min(cpu_count(), 50)]

    params = []
    for i, hash_ in enumerate(transactions):
        params.append((i, hash_))

    pool = Pool(min(cpu_count(), 50))
    pool.starmap_async(calculate_transaction, params).get()
    pool.close()


if __name__ == "__main__":
    main()

I'm going to close this as I can't reproduce it. If that does not resolve the issue and you see that something else is going on, please reach back out to re-open.

If it helps, my HTTP_LOCAL is a Nethermind client.