32

I would like to explore the data inside of the Ethereum blockchain. All transactions and contracts of the Ethereum Blockchain are stored there. So it's an interesting open dataset and I'd like to get some insights out of it. But I am not entirely sure how can I access this data. Inside ~/.ethereum/chaindata (Linux) I can see the files containing the blockchain but unlike the keys they cannot be explored easily. What kind of encoding does it have? What are the best technical tools (like databases) to work with it (in a non-interactive way)?

I don't really think using a website to explore certain transactions or contract code like what you can do on https://www.etherchain.org/ is the best solution. But instead I want to explore the (whole) data first and do a classical exploratory data analysis, so that later I can focus on certain parts I want to explore.

Suggestions about how to work with this data, about its format and how to manipulate it are welcome.

15

Data structures are stored in Merkle Patricia tries (read this and this), usually inside a LevelDB store. The chaindata is exactly that.

I think the structure might slightly depend on the actual node implementation?

Here are two of the trie implementations, could be a good starting point:

Once that is understood, you will need to look up the data structures sent over the network, especially blocks, transactions, accounts and storage. Accounts refer to both externally owned and contract accounts, the latter with contract code. Storage refers to contract storage.

Back to the point of data analysis

You might be better off building a traditional NoSQL or SQL database if you want to have quick lookups about any transactions done by an account or contract. What the chaindata is optimised for is validating transactions and maintaining a valid state, and not so for historical lookups.

  • 4
    Ethereum chain parser listens for Ethereum (block) chain events and turns them into MongoDB records. It is out of date & not currently compiling but it could be a good starting point. – JackWinters Mar 21 '16 at 14:18
10

I am also interested in analysing the Ethereum blockchain. In fact, I'm primarily looking into smart contracts. While I was researching papers and articles about this, I stumbled upon this paper where the authors analyse Ethereum, namecoin, and peercoin. They have quite some intersting findings around number of contracts, termination of contracts, and transactions.

More specifically, they provide a modified geth client to dig deep in the Ethereum network. It is provided on their GitHub page. I haven't had the time yet to try it out myself, but will do so in the coming days.

  • please let me know how well that goes and/or consider making a guide or tutorial about this tool. There are many people interested in blockchain analysis but not enough resources. – wacax Jan 26 '17 at 17:38
6

I am interested in the same project. Currently, it appears to me that the javascript API provides the most robust access to the blockchain data:

https://github.com/ethereum/wiki/wiki/JavaScript-API

You can use this API from Geth or a standalone Node.js script. I populated a table of transactions in PostgreSQL using this API. However I am not aware of a way to obtain contract-to-contract messages in addition to transactions via this API. These messages are available through the etherchain.org API but I agree with you that the etherchain API is probably not a good fit for analytics. See this post:

Is it possible to get contract messages from Geth?

  • just 2 questions: how long does it take to populate a table of transactions in PostgreSQL and have you tried the same in a NoSQL database? – wacax Mar 21 '16 at 15:37
  • 1
    It took several days for my code to populate the transaction table though its entirely possible my code is inefficient. The API calls to read the blockchain seem to be the slow part. I haven't tried a NoSQL approach. – antianticamper Mar 21 '16 at 15:57
3

That is your location: provide full transparency of the smart contract execution:

https://live.ether.camp/

Here is how smart contract is traceable

enter image description here

1

Here is a guide on how to export Ethereum data to csv and analyze it with Amazon Athena https://medium.com/@medvedev1088/exporting-and-analyzing-ethereum-blockchain-f5353414a94e

It uses https://github.com/medvedev1088/ethereum-etl which outputs the data into blocks.csv, transactions.csv, erc20_transfers.csv.

blocks.csv

Column                  | Type               |
------------------------|---------------------
block_number            | bigint             |
block_hash              | hex_string         |
block_parent_hash       | hex_string         |
block_nonce             | hex_string         |
block_sha3_uncles       | hex_string         |
block_logs_bloom        | hex_string         |
block_transactions_root | hex_string         |
block_state_root        | hex_string         |
block_miner             | hex_string         |
block_difficulty        | bigint             |
block_total_difficulty  | bigint             |
block_size              | bigint             |
block_extra_data        | hex_string         |
block_gas_limit         | bigint             |
block_gas_used          | bigint             |
block_timestamp         | bigint             |
block_transaction_count | bigint             |

transactions.csv

Column              |    Type     |
--------------------|--------------
tx_hash             | hex_string  |
tx_nonce            | bigint      |
tx_block_hash       | hex_string  |
tx_block_number     | bigint      |
tx_index            | bigint      |
tx_from             | hex_string  |
tx_to               | hex_string  |
tx_value            | bigint      |
tx_gas              | bigint      |
tx_gas_price        | bigint      |
tx_input            | hex_string  |

erc20_transfers.csv

Column              |    Type     |
--------------------|--------------
erc20_token         | hex_string  |
erc20_from          | hex_string  |
erc20_to            | hex_string  |
erc20_value         | bigint      |
erc20_tx_hash       | hex_string  |
erc20_block_number  | bigint      |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.