Exploring and Analyzing the Ethereum blockchain

Question

I would like to explore the data inside of the Ethereum blockchain. All transactions and contracts of the Ethereum Blockchain are stored there. So it's an interesting open dataset and I'd like to get some insights out of it. But I am not entirely sure how can I access this data. Inside ~/.ethereum/chaindata (Linux) I can see the files containing the blockchain but unlike the keys they cannot be explored easily. What kind of encoding does it have? What are the best technical tools (like databases) to work with it (in a non-interactive way)?

I don't really think using a website to explore certain transactions or contract code like what you can do on https://www.etherchain.org/ is the best solution. But instead I want to explore the (whole) data first and do a classical exploratory data analysis, so that later I can focus on certain parts I want to explore.

Suggestions about how to work with this data, about its format and how to manipulate it are welcome.

Answer 1

Data structures are stored in Merkle Patricia tries (read this and this), usually inside a LevelDB store. The chaindata is exactly that.

I think the structure might slightly depend on the actual node implementation?

Here are two of the trie implementations, could be a good starting point:

• in Go (geth): https://github.com/ethereum/go-ethereum/tree/master/trie
• in Javascript: https://github.com/ethereumjs/merkle-patricia-tree

Once that is understood, you will need to look up the data structures sent over the network, especially blocks, transactions, accounts and storage. Accounts refer to both externally owned and contract accounts, the latter with contract code. Storage refers to contract storage.

Back to the point of data analysis

You might be better off building a traditional NoSQL or SQL database if you want to have quick lookups about any transactions done by an account or contract. What the chaindata is optimised for is validating transactions and maintaining a valid state, and not so for historical lookups.

Answer 2

I am also interested in analysing the Ethereum blockchain. In fact, I'm primarily looking into smart contracts. While I was researching papers and articles about this, I stumbled upon this paper where the authors analyse Ethereum, namecoin, and peercoin. They have quite some intersting findings around number of contracts, termination of contracts, and transactions.

More specifically, they provide a modified geth client to dig deep in the Ethereum network. It is provided on their GitHub page. I haven't had the time yet to try it out myself, but will do so in the coming days.

Answer 3

I am interested in the same project. Currently, it appears to me that the javascript API provides the most robust access to the blockchain data:

https://github.com/ethereum/wiki/wiki/JavaScript-API

You can use this API from Geth or a standalone Node.js script. I populated a table of transactions in PostgreSQL using this API. However I am not aware of a way to obtain contract-to-contract messages in addition to transactions via this API. These messages are available through the etherchain.org API but I agree with you that the etherchain API is probably not a good fit for analytics. See this post:

Is it possible to get contract messages from Geth?

Answer 4

That is your location: provide full transparency of the smart contract execution:

https://live.ether.camp/

Here is how smart contract is traceable

Answer 5

Here is a guide on how to export Ethereum data to csv and analyze it with Amazon Athena https://medium.com/@medvedev1088/exporting-and-analyzing-ethereum-blockchain-f5353414a94e

It uses https://github.com/medvedev1088/ethereum-etl which outputs the data into blocks.csv, transactions.csv, erc20_transfers.csv.

blocks.csv

Column                  | Type               |
------------------------|---------------------
block_number            | bigint             |
block_hash              | hex_string         |
block_parent_hash       | hex_string         |
block_nonce             | hex_string         |
block_sha3_uncles       | hex_string         |
block_logs_bloom        | hex_string         |
block_transactions_root | hex_string         |
block_state_root        | hex_string         |
block_miner             | hex_string         |
block_difficulty        | bigint             |
block_total_difficulty  | bigint             |
block_size              | bigint             |
block_extra_data        | hex_string         |
block_gas_limit         | bigint             |
block_gas_used          | bigint             |
block_timestamp         | bigint             |
block_transaction_count | bigint             |

transactions.csv

Column              |    Type     |
--------------------|--------------
tx_hash             | hex_string  |
tx_nonce            | bigint      |
tx_block_hash       | hex_string  |
tx_block_number     | bigint      |
tx_index            | bigint      |
tx_from             | hex_string  |
tx_to               | hex_string  |
tx_value            | bigint      |
tx_gas              | bigint      |
tx_gas_price        | bigint      |
tx_input            | hex_string  |

erc20_transfers.csv

Column              |    Type     |
--------------------|--------------
erc20_token         | hex_string  |
erc20_from          | hex_string  |
erc20_to            | hex_string  |
erc20_value         | bigint      |
erc20_tx_hash       | hex_string  |
erc20_block_number  | bigint      |