How to parse blocks with Python?

Question

I'm building a basic analytics framework for the Ethereum blockchain. As a first step I need to extract the data and I'd like to do this in Python. I've been using Geth as a client on OSX and so, as far as I understand, the blockchain is stored locally at ~/Library/Ethereum/chaindata in LevelDB, i.e. in binary files like 496355.ldb.

1. How do I parse the .ldb files? Is there a Python library I can use for this? I've looked at pyethereum but am not sure where to look as the documentation is pretty sparse.

2. I can use the etherchain.org API but this would involve many API calls and will probably make the good people at etherchain.org unhappy. This is why I prefer to parse the blockchain as stored locally on my machine. But ignoring this problem for a moment, does anyone know a short guide to the result of an API call for transaction data resulting in data like this:

{'accountNonce': '200642',
   'amount': 2678970350000000000,
   'blockHash': '0x47525d00eab0dcd87e9f8b0e9de2fe9f553f72576ef4f892ace49b2832e985bd',
   'block_id': 1039153,
   'gasLimit': 21000,
   'hash': '0x41bb75d8b20ae7e23fa7457d21a30a4007d0eb97fdb887caac834b759eeb3572',
   'isContractTx': None,
   'newContract': 0,
   'parentHash': '0x41bb75d8b20ae7e23fa7457d21a30a4007d0eb97fdb887caac834b759eeb3572',
   'price': 50000000000,
   'recipient': '0x30906581413d556de1a018adbe6cc63c88d58512',
   'sender': '0x2a65aca4d5fc5b5c859090a6c34d164135398226',
   'time': '2016-02-21T18:53:56.000Z',
   'txIndex': None}

I believe I understand most of this data but would really appreciate more information to fill in the gaps of my understanding.

Answer 1

I implemented a bitcoin blockchain parser after following along with this blog post. I havent tried changing it to work with the Ethereum blockchain but maybe it can give you some guidance. Here's a parser implemented in Go to hopefully give you some Ethereum specific examples to go off of.

Answer 2

Your geth node will also act as a JSON-RPC server. You can use the APIs wiki page to crawl the blockchain and extract block data. Here is an interesting git repo that you could fork. It's a bit show though and requires some code modifications as the owner isn't maintaining it any more. You'll find a pull request on that repo that runs substantially faster, however it requires more modifications to work.

Answer 3

Since you want the results in python, you could connect to your geth instance over IPC with web3.py*. It would look something like:

>>> from web3.auto import w3

>>> w3.eth.getBlock(2000000)
AttributeDict({
    'difficulty': 49824742724615,
    'extraData': '0xe4b883e5bda9e7a59ee4bb99e9b1bc',
    'gasLimit': 4712388,
    'gasUsed': 21000,
    'hash': '0xc0f4906fea23cf6f3cce98cb44e8e1449e455b28d684dfa9ff65426495584de6',
    'logsBloom': '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
    'miner': '0x61c808d82a3ac53231750dadc13c777b59310bd9',
    'nonce': '0x3b05c6d5524209f1',
    'number': 2000000,
    'parentHash': '0x57ebf07eb9ed1137d41447020a25e51d30a0c272b5896571499c82c33ecb7288',
    'receiptRoot': '0x84aea4a7aad5c5899bd5cfc7f309cc379009d30179316a2a7baa4a2ea4a438ac',
    'sha3Uncles': '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347',
    'size': 650,
    'stateRoot': '0x96dbad955b166f5119793815c36f11ffa909859bbfeb64b735cca37cbf10bef1',
    'timestamp': 1470173578,
    'totalDifficulty': 44010101827705409388,
    'transactions': ['0xc55e2b90168af6972193c1f86fa4d7d7b31a29c156665d15b9cd48618b5177ef'],
    'transactionsRoot': '0xb31f174d27b99cdae8e746bd138a01ce60d8dd7b224f7c60845914def05ecc58',
    'uncles': [],
})

---

* Try the beta version (v4), installed with:

pip install --pre web3