15

I'm building a basic analytics framework for the Ethereum blockchain. As a first step I need to extract the data and I'd like to do this in Python. I've been using Geth as a client on OSX and so, as far as I understand, the blockchain is stored locally at ~/Library/Ethereum/chaindata in LevelDB, i.e. in binary files like 496355.ldb.

  1. How do I parse the .ldb files? Is there a Python library I can use for this? I've looked at pyethereum but am not sure where to look as the documentation is pretty sparse.

  2. I can use the etherchain.org API but this would involve many API calls and will probably make the good people at etherchain.org unhappy. This is why I prefer to parse the blockchain as stored locally on my machine. But ignoring this problem for a moment, does anyone know a short guide to the result of an API call for transaction data resulting in data like this:

{'accountNonce': '200642',
   'amount': 2678970350000000000,
   'blockHash': '0x47525d00eab0dcd87e9f8b0e9de2fe9f553f72576ef4f892ace49b2832e985bd',
   'block_id': 1039153,
   'gasLimit': 21000,
   'hash': '0x41bb75d8b20ae7e23fa7457d21a30a4007d0eb97fdb887caac834b759eeb3572',
   'isContractTx': None,
   'newContract': 0,
   'parentHash': '0x41bb75d8b20ae7e23fa7457d21a30a4007d0eb97fdb887caac834b759eeb3572',
   'price': 50000000000,
   'recipient': '0x30906581413d556de1a018adbe6cc63c88d58512',
   'sender': '0x2a65aca4d5fc5b5c859090a6c34d164135398226',
   'time': '2016-02-21T18:53:56.000Z',
   'txIndex': None}

I believe I understand most of this data but would really appreciate more information to fill in the gaps of my understanding.

3 Answers 3

9

I implemented a bitcoin blockchain parser after following along with this blog post. I havent tried changing it to work with the Ethereum blockchain but maybe it can give you some guidance. Here's a parser implemented in Go to hopefully give you some Ethereum specific examples to go off of.

2
  • 2
    I had previously seen your blog posts and e-book on parsing Bitcoin blogs and learned a great deal from them! Feb 28, 2016 at 21:51
  • 1
    The link to the blog post is no longer working :-( Sep 1, 2017 at 15:38
1

Your geth node will also act as a JSON-RPC server. You can use the APIs wiki page to crawl the blockchain and extract block data. Here is an interesting git repo that you could fork. It's a bit show though and requires some code modifications as the owner isn't maintaining it any more. You'll find a pull request on that repo that runs substantially faster, however it requires more modifications to work.

0

Since you want the results in python, you could connect to your geth instance over IPC with web3.py*. It would look something like:

>>> from web3.auto import w3

>>> w3.eth.getBlock(2000000)
AttributeDict({
    'difficulty': 49824742724615,
    'extraData': '0xe4b883e5bda9e7a59ee4bb99e9b1bc',
    'gasLimit': 4712388,
    'gasUsed': 21000,
    'hash': '0xc0f4906fea23cf6f3cce98cb44e8e1449e455b28d684dfa9ff65426495584de6',
    'logsBloom': '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
    'miner': '0x61c808d82a3ac53231750dadc13c777b59310bd9',
    'nonce': '0x3b05c6d5524209f1',
    'number': 2000000,
    'parentHash': '0x57ebf07eb9ed1137d41447020a25e51d30a0c272b5896571499c82c33ecb7288',
    'receiptRoot': '0x84aea4a7aad5c5899bd5cfc7f309cc379009d30179316a2a7baa4a2ea4a438ac',
    'sha3Uncles': '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347',
    'size': 650,
    'stateRoot': '0x96dbad955b166f5119793815c36f11ffa909859bbfeb64b735cca37cbf10bef1',
    'timestamp': 1470173578,
    'totalDifficulty': 44010101827705409388,
    'transactions': ['0xc55e2b90168af6972193c1f86fa4d7d7b31a29c156665d15b9cd48618b5177ef'],
    'transactionsRoot': '0xb31f174d27b99cdae8e746bd138a01ce60d8dd7b224f7c60845914def05ecc58',
    'uncles': [],
})

* Try the beta version (v4), installed with:

pip install --pre web3

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.