Web3.py: ValueError: Could not format value when trying to deploy smart contract bytecode from Truffle

Question

I have a working solidity contract which compiles and gets deployed successfully by both truffle and web3.js + metamask. Importing the very same contract into python and deploying it using web3.py (through infura, but it should not matter) does not work. The bytecode looks invalid.

The python code loading the output of the truffle compile is the following:

w3 = importlib.import_module("web3.auto.infura.kovan").w3
smart_contract_string = open('./../resources/IndexRate.json', "r").read()
smart_contract_json = json.loads(smart_contract_string)
abi = smart_contract_json["abi"]
bytecode = smart_contract_json["bytecode"]
contract = w3.eth.contract(abi=abi, bytecode=bytecode)

and it outputs the following:

Traceback (most recent call last):
  File "/Users/g/Documents/projects/my-project/venv/lib/python3.7/site-packages/eth_utils/applicators.py", line 84, in apply_formatters_to_dict
    yield key, formatters[key](item)
  File "/Users/g/Documents/projects/my-project/venv/lib/python3.7/site-packages/web3/_utils/normalizers.py", line 258, in normalize_bytecode
    bytecode = HexBytes(bytecode)
  File "/Users/g/Documents/projects/my-project/venv/lib/python3.7/site-packages/hexbytes/main.py", line 24, in new
    bytesval = hexstr_if_str(to_bytes, val)
  File "/Users/g/Documents/projects/my-project/venv/lib/python3.7/site-packages/eth_utils/conversions.py", line 160, in hexstr_if_str
    hexstr_or_primitive
ValueError: when sending a str, it must be a hex string. Got: <whole bytecode goes here>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/g/Documents/projects/my-project/src/index_rate_manager.py", line 125, in <module>
    contract = w3.eth.contract(abi=abi, bytecode=bytecode)
  File "/Users/g/Documents/projects/my-project/venv/lib/python3.7/site-packages/web3/eth.py", line 526, in contract
    ContractFactory = ContractFactoryClass.factory(self.web3, **kwargs)
  File "/Users/g/Documents/projects/my-project/venv/lib/python3.7/site-packages/web3/contract.py", line 364, in factory
    normalizers=normalizers,
  File "/Users/g/Documents/projects/my-project/venv/lib/python3.7/site-packages/web3/_utils/datatypes.py", line 55, in new
    namespace,
  File "/Users/g/Documents/projects/my-project/venv/lib/python3.7/site-packages/eth_utils/functional.py", line 45, in inner
    return callback(fn(*args, **kwargs))
  File "/Users/g/Documents/projects/my-project/venv/lib/python3.7/site-packages/eth_utils/applicators.py", line 88, in apply_formatters_to_dict
    ) from exc
ValueError: Could not format value

The smart contract bytecode is available here

the ABI file is available here

I noticed that the bytecode does contain non hexadecimal values. In particular, it contains __DateTime______________________________ which is the name of one library I am using, but it is obviously not binary

I am using:

1. solc 0.4.25
2. truffle 5.1.26
3. nodejs 10.16.3
4. npm 6.9.0
5. python 3.7.4
6. web3.py 5.6.0

Answer 1

Your hunch is correct.

__DateTime______________________________ is a marker for linking in Solidity bytecode. This marker will be replaced by DateTime library smart contract address when the smart contract bytecode is linked (should be 32 bytes if I recall correctly).

The full deployment process looks like this

1. Deploy libraries, get their smart contract addresses (note that this requires to do an Ethereum transaction, so it will take up to 5-10 minutes on the mainnet) - Truffle will do this for you
2. Preprocess bytecode of any smart contract using this library with the actual deployment address
3. Deploy the remaining contracts

You need to pass actual linked bytecode to web3.py, but I am not exactly sure where in Solidity output it is or do you need to pass any extra switches to Truffle to generate it.