Rinkeby failure with web3.py -- Could not format value '0x...' as field 'extraData'

Question

I am trying to connect to Rinkeby using web3.py. But when I call a method, like w3.version.node, it throws an exception ending in:

...
  File "/home/hacker/venv/lib/python3.5/site-packages/web3/utils/formatters.py", line 69, in apply_formatters_to_dict
    raise type(exc)("Could not format value %r as field %r" % (item, key)) from exc
ValueError: Could not format value '0x{LONG_HEX}' as field 'extraData'

(Where LONG_HEX is a lot of hex characters, like 194)

How do I fix my connection to get block data?

Versions:

• geth 1.8.2
• web3.py v4.0.0-beta.13

Note: In addition to the Rinkeby network, I get this same exception after connecting to a node started with geth --dev

Answer 1

Solution

Insert a special middleware in web3.py v5+ to handle geth-style proof-of-authority, like this:

py> from web3 import Web3, IPCProvider

# connect to the default geth --dev IPC location
py> w3 = Web3(IPCProvider('/tmp/geth.ipc'))

py> from web3.middleware import geth_poa_middleware

# inject the poa compatibility middleware to the innermost layer
py> w3.middleware_onion.inject(geth_poa_middleware, layer=0)

# confirm that the connection succeeded
py> w3.version.node
'Geth/v1.8.2-stable-b8b9f7f4/linux-amd64/go1.9.4'

Why is geth_poa_middleware necessary?

A slightly modified version of the web3.py docs:

There is no strong community consensus on a single Proof-of-Authority (PoA) standard yet. Some nodes have successful experiments running, though. One is go-ethereum (geth), which uses a prototype PoA for it’s development mode and the Rinkeby test network.

Unfortunately, the prototype does deviate from the yellow paper specification, which constrains the extraData field in each block to a maximum of 32-bytes. Geth’s PoA uses more than 32 bytes, which causes a validation failure in web3.py. This middleware modifies the block data to shorten extraData a bit before returning it.

Answer 2

Addendum to previous answer: In web3py version 5.x.x you have to change it this way:

    # inject the poa compatibility middleware to the innermost layer
    w3.middleware_stack.inject(geth_poa_middleware, layer=0)

to this:

    # inject the poa compatibility middleware to the innermost layer
    w3.middleware_onion.inject(geth_poa_middleware, layer=0)

I mean, middleware_stack was renamed to: middleware_onion.