Exporting complete blockchain data

Question

I'm considering running statistical analysis on ethereum's blockchain data. That idea assumes that I have all the relevant data in some readable format.

So far I've ran a geth node that synced with the network and the documentation says I can do an export.

https://github.com/ethereum/go-ethereum/wiki/Backup-&-restore

I'm just curious about this binary format of an export. Is there a way of consuming that data with some 3rd party app?

If so, can you point me to some documentation? If this is a bad approach please do point me in the right direction.

The only goal that I need to achieve is to get all of the blockchain data in some shape or form that I can read with a 3rd party app that I'm writing at the moment.

Answer 1

As outlined here,

"Data structures are stored in Merkle Patricia tries"

I have encountered the same problem as you have described and opted to collect the data in a more consumable format by taking advantage of the fact that both Geth and Parity implement the full set of JSON RPC endpoints.

You can use web3.js for example to query these endpoints and pull data about blocks or transactions from the chain. You can then store this data in a format optimised for processing it.

Something as simple as var info = web3.eth.getBlock(3150, true); will return information about a block, and all the transactions contained within as outlined in the API docs.

Answer 2

Here is a guide about exporting Ethereum blockchain data into csv and analyzing it with AWS Athena https://medium.com/@medvedev1088/exporting-and-analyzing-ethereum-blockchain-f5353414a94e

I used this tool to export the data https://github.com/medvedev1088/ethereum-etl

blocks.csv

Column                  | Type               |
------------------------|---------------------
block_number            | bigint             |
block_hash              | hex_string         |
block_parent_hash       | hex_string         |
block_nonce             | hex_string         |
block_sha3_uncles       | hex_string         |
block_logs_bloom        | hex_string         |
block_transactions_root | hex_string         |
block_state_root        | hex_string         |
block_miner             | hex_string         |
block_difficulty        | bigint             |
block_total_difficulty  | bigint             |
block_size              | bigint             |
block_extra_data        | hex_string         |
block_gas_limit         | bigint             |
block_gas_used          | bigint             |
block_timestamp         | bigint             |
block_transaction_count | bigint             |

transactions.csv

Column              |    Type     |
--------------------|--------------
tx_hash             | hex_string  |
tx_nonce            | bigint      |
tx_block_hash       | hex_string  |
tx_block_number     | bigint      |
tx_index            | bigint      |
tx_from             | hex_string  |
tx_to               | hex_string  |
tx_value            | bigint      |
tx_gas              | bigint      |
tx_gas_price        | bigint      |
tx_input            | hex_string  |

erc20_transfers.csv

Column              |    Type     |
--------------------|--------------
erc20_token         | hex_string  |
erc20_from          | hex_string  |
erc20_to            | hex_string  |
erc20_value         | bigint      |
erc20_tx_hash       | hex_string  |
erc20_block_number  | bigint      |

Answer 3

Every Ethereum node (including geth, of course) provides an access to a built-in JSON API. It allows to export transactions, blocks and other data in JSON format.

Usually the API is used over HTTP (RPC), but when you run your own node, you may efficiently fetch the data from a node using geth IPC (via a unix socket).

In fact, there is no need to run your own node. Feel free to rely on one of free public nodes with the same JSON RPC API support: ethereumnodes.com.

Here is a good article on how to setup a Data Platform for processing Ethereum data: “How to Export a Full History of Ethereum Blockchain to S3” (Medium).