3

I've upgraded from python 2 to python 3 and some code that used to work is now broken. I can't seem to fix it. Any suggestions?

Worked in python 2

# Convert hex string to byte array
hex_string = "0x00012345...."
data_bytearray = bytearray.fromhex(hex_string.replace("0x", ""))

# Then I pass data_bytearray into transactions.Transaction and it works.  
try:
    return {'error': False, 'sign': rlp.encode(transactions.Transaction(nonce, gasPrice_int, gas_int, to, value, data_bytearray).sign(privkey)).encode('hex')}
except Exception as msg:
    return {'error': True, 'message': msg}

Same code doesn't work in python 3 and note the byte array data type has changed to

<class 'bytearray'>   # python 3 produces this from bytearray.fromhex
<type 'bytearray'>    # python 2 produces this from bytearray.fromhex

I get the following error:

{'error': True, 'message': ObjectSerializationError('Serialization failed because of field data ("Object is not a serializable (<class \'bytearray\'>)")',)}

I tried serializing the output using pickle, but got another error:

import pickle
data_bytearray = pickle.dumps(data_bytearray)

{'error': True, 'message': AttributeError("'bytes' object has no attribute 'encode'",)}

What am I doing wrong here?

Solution:

try:
    data_hex_with0xRemoved = data_hex.replace("0x", "")
    # If I use bytearray.fromhex, it results in {'error': True, 'message': ObjectSerializationError('Serialization failed because of field data ("Object is not a serializable (<class \'bytearray\'>)")',)}
    # data = bytearray.fromhex('deadbeef')
    # So instead I use bytes.fromhex
    data = bytes.fromhex(data_hex_with0xRemoved)
    unsigned_transaction = transactions.Transaction(nonce, gasPrice_int, gas_int, to, value, data)
    raw_transaction_bytes = rlp.encode(unsigned_transaction.sign(privkey))
    raw_transaction_hex = Web3.toHex(raw_transaction_bytes)
    raw_transaction_hex_0xRemoved = raw_transaction_hex.replace("0x", "")
    return {'error': False, 'sign': raw_transaction_hex_0xRemoved}
except Exception as msg:
    return {'error': True, 'message': msg}
3
  • Have you tried bytes instead of bytearray? Also, you might like: data_bytes = Web3.toBytes(hexstr="0x00012345") from web3.py
    – carver
    Feb 26, 2018 at 18:12
  • Yes both bytes and Web3.toBytes gave me a similar error to pickle.dumps: {'error': True, 'message': AttributeError("'bytes' object has no attribute 'encode'",)}. Feb 26, 2018 at 19:14
  • This is not really a solution, but you could call a py2 script that performs this operation and returns the value to py3 script. I ran into this same problem and had too many more important battles to pick so I took the quick workaround that allowed me to move on to more important things. I'm curious about the right way to do this as well.
    – LampShade
    Feb 26, 2018 at 19:21

1 Answer 1

1

Solution

try:
    unsigned_transaction = transactions.Transaction(nonce, gasPrice_int, gas_int, to, value, data_bytearray)
    raw_transaction = rlp.encode(unsigned_transaction.sign(privkey))
    return {'error': False, 'sign': Web3.toHex(raw_transaction)}
except Exception as msg:
    return {'error': True, 'message': msg}

The only notable change is from raw_transaction.encode('hex') to Web3.toHex(raw_transaction).

The other differences from the original line were simply to reduce the amount of logic per line.


Why?

both bytes and Web3.toBytes gave me a similar error to pickle.dumps: {'error': True, 'message': AttributeError("'bytes' object has no attribute 'encode'",)}

This ^ comment was the most helpful, although it doesn't explain why you don't get that same error with bytearray (I think you should). See:

In [1]: b'\xaf'.encode('hex')
AttributeError: 'bytes' object has no attribute 'encode'

In [2]: bytearray(b'\xaf').encode('hex')
AttributeError: 'bytearray' object has no attribute 'encode'

This doesn't really have anything to do with Ethereum, only with learning how encoding and pickling works in Python 3. For example, I would expect the following to behave exactly as the first code sample from the question does:

try:
    return {'error': False, 'sign': bytearray(b'\xaf').encode('hex')}
except Exception as msg:
    return {'error': True, 'message': msg}

Encoding in Python 2 had a sloppy API, and permitted people to do nonsensical things, like "encode" binary data (which is already encoded, being binary). So, encode() was removed from the Python 3 bytes type altogether. If you want to convert bytes to hex in Python 3, there are a few ways to do it:

My personal favorite option

> Web3.toHex(b'\xaf')
'0xaf'

Alternatively, in Python 3.5+, you can use:

> b'\xaf'.hex()
'af'
# note that you don't get the "0x" prefix

Another built-in option is binascii

> import binascii
> hex_in_bytes = binascii.hexlify(b'\xaf')
b'af'

# binascii made the nonsensical choice to return the hex value in a `bytes` type
# which you can fix up by decoding to a string

> hex_in_bytes.decode('utf8')
'af'

(All of these except Web3 work with bytearray as well).

2
  • You were right, the problems i ran into were all related to my misunderstandings around python 3 and not ethereum. Thanks for breaking this down for me. I posted the final code I went with above. I ended up having to use bytes.fromhex instead of bytearray.fromhex for the reason listed above. Mar 2, 2018 at 21:30
  • Got it, you may also like data = Web3.toBytes(hexstr=data_hex).
    – carver
    Mar 2, 2018 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.