How can I export Ethereum blockchain data into csv?

Question

I would like to download the Ethereum blockchain and use the data for machine learning purposes. For this, I have to get the data into a workable format like csv or a flat JSON etc.

I've read about approaches and the most common one seems to be through a monkeyDB. Do you know a source code that implements this method? Or do you know other methods that could be used to get the Ethereum blockchain data into a modeling friendly format?

Thanks!

Answer 1

Here is an article on how to export Ethereum data to csv and analyze it with Amazon Athena https://medium.com/@medvedev1088/exporting-and-analyzing-ethereum-blockchain-f5353414a94e

It uses https://github.com/medvedev1088/ethereum-etl which outputs the data into blocks.csv, transactions.csv, erc20_transfers.csv.

blocks.csv

Column                  | Type               |
------------------------|---------------------
block_number            | bigint             |
block_hash              | hex_string         |
block_parent_hash       | hex_string         |
block_nonce             | hex_string         |
block_sha3_uncles       | hex_string         |
block_logs_bloom        | hex_string         |
block_transactions_root | hex_string         |
block_state_root        | hex_string         |
block_miner             | hex_string         |
block_difficulty        | bigint             |
block_total_difficulty  | bigint             |
block_size              | bigint             |
block_extra_data        | hex_string         |
block_gas_limit         | bigint             |
block_gas_used          | bigint             |
block_timestamp         | bigint             |
block_transaction_count | bigint             |

transactions.csv

Column              |    Type     |
--------------------|--------------
tx_hash             | hex_string  |
tx_nonce            | bigint      |
tx_block_hash       | hex_string  |
tx_block_number     | bigint      |
tx_index            | bigint      |
tx_from             | hex_string  |
tx_to               | hex_string  |
tx_value            | bigint      |
tx_gas              | bigint      |
tx_gas_price        | bigint      |
tx_input            | hex_string  |

erc20_transfers.csv

Column              |    Type     |
--------------------|--------------
erc20_token         | hex_string  |
erc20_from          | hex_string  |
erc20_to            | hex_string  |
erc20_value         | bigint      |
erc20_tx_hash       | hex_string  |
erc20_block_number  | bigint      |

Answer 2

To export Ethereum data, you may rely on a number of public nodes listed at ethereumnodes.com: some of them are free of charge.

They provide access to JSON RPC API which allows you to export blocks, transactions and other blockchain data in JSON.

Here is a modern approach to export the full history of Ethereum into S3 using the public nodes: “How to Export a Full History of Ethereum Blockchain to S3” (Medium).

As an alternative, you may rely on BigQuery public datasets with Ethereum data. This will free your hands from exporting the data. Using BigQuery datasets is also covered by the article.