3

I'm considering running statistical analysis on ethereum's blockchain data. That idea assumes that I have all the relevant data in some readable format.

So far I've ran a geth node that synced with the network and the documentation says I can do an export.

https://github.com/ethereum/go-ethereum/wiki/Backup-&-restore

I'm just curious about this binary format of an export. Is there a way of consuming that data with some 3rd party app?

If so, can you point me to some documentation? If this is a bad approach please do point me in the right direction.

The only goal that I need to achieve is to get all of the blockchain data in some shape or form that I can read with a 3rd party app that I'm writing at the moment.

1

As outlined here,

"Data structures are stored in Merkle Patricia tries"

I have encountered the same problem as you have described and opted to collect the data in a more consumable format by taking advantage of the fact that both Geth and Parity implement the full set of JSON RPC endpoints.

You can use web3.js for example to query these endpoints and pull data about blocks or transactions from the chain. You can then store this data in a format optimised for processing it.

Something as simple as var info = web3.eth.getBlock(3150, true); will return information about a block, and all the transactions contained within as outlined in the API docs.

  • This approach of consuming JSON RPC actually suits me much better, great idea. – talfirevic Aug 8 '17 at 10:26
  • 1
    Check out quickblocks.io. QuickBlocks is a work in progress, but the speed of doing the above is significantly improved. – Thomas Jay Rush Aug 9 '17 at 2:24
1

Here is a guide about exporting Ethereum blockchain data into csv and analyzing it with AWS Athena https://medium.com/@medvedev1088/exporting-and-analyzing-ethereum-blockchain-f5353414a94e

I used this tool to export the data https://github.com/medvedev1088/ethereum-etl

blocks.csv

Column                  | Type               |
------------------------|---------------------
block_number            | bigint             |
block_hash              | hex_string         |
block_parent_hash       | hex_string         |
block_nonce             | hex_string         |
block_sha3_uncles       | hex_string         |
block_logs_bloom        | hex_string         |
block_transactions_root | hex_string         |
block_state_root        | hex_string         |
block_miner             | hex_string         |
block_difficulty        | bigint             |
block_total_difficulty  | bigint             |
block_size              | bigint             |
block_extra_data        | hex_string         |
block_gas_limit         | bigint             |
block_gas_used          | bigint             |
block_timestamp         | bigint             |
block_transaction_count | bigint             |

transactions.csv

Column              |    Type     |
--------------------|--------------
tx_hash             | hex_string  |
tx_nonce            | bigint      |
tx_block_hash       | hex_string  |
tx_block_number     | bigint      |
tx_index            | bigint      |
tx_from             | hex_string  |
tx_to               | hex_string  |
tx_value            | bigint      |
tx_gas              | bigint      |
tx_gas_price        | bigint      |
tx_input            | hex_string  |

erc20_transfers.csv

Column              |    Type     |
--------------------|--------------
erc20_token         | hex_string  |
erc20_from          | hex_string  |
erc20_to            | hex_string  |
erc20_value         | bigint      |
erc20_tx_hash       | hex_string  |
erc20_block_number  | bigint      |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.