0

I know that the "internal transactions" are not included in the blockchain and can only be retrieved by replaying the transactions and listening to CALL, CALLCODE and other similar opcodes. Both Parity and Geth provide means to do that.

However I am having trouble performing a full sync of Geth / Parity. Geth took over a month to sync, uses ~800 GB and as of yesterday doesn't want to run anymore, failing with Fatal: Error starting protocol stack: missing block number for head header hash. I was not able to find a solution since resync is not really an option for me. On the other hand I have started syncing Parity about a week ago and the process, fast at first, slowed to a crawl at around block #2393000.

As far as I know I can't use warp / fast sync if I want to get internal transactions. Also, I can't use Infura because it doesn't offer appropriate JSON RPC interface. I could use Etherscan, but that seems like cheating - what is the point of having a decentralized system if you fetch your data from a central endpoint? I might have to use it if everything else fails though, but I would appreciate some pointers. I am sure I am not the only one attempting to do this.

TL;DR: what is the fastest way to synchronize nodes so I can fetch the internal transactions from them? Failing that, what are other options to get this data?

Improve this question
6
  • getting internal transactions is hell of a difficult. I am writing an entire blockchain explorer for this. You have to parse execute every transaction and reproduce the stack of the virtual machine, to find out the correct flow of funds. Not only CALL/CALLCODE but also SELFDESTRUCT affects balance. You could use Etherescan or you could wait for my explorer to be out. It is going to export to Postgres. Will be out in a few weeks more.
    – Nulik
    Feb 13 '18 at 22:24
  • @Nulik Thanks for the tip about SELFDESTRUCT! I am not that concerned about the explorer itself, I am using a python app and Neo4j for that. However getting the internal txs from the blockchain is... slow, at least via JSON RPC. Will your explorer be able to get internal txs faster somehow? If so, I would be really curious how...
    – johndodo
    Feb 15 '18 at 7:52
  • My explorer is being written in golang, it is a modified version of geth, so it is fast, though I didn't focus on code optimization yet. The genesis block with 8893 accounts takes a few minutes to process, mostly because every INSERT is separate, and lookups are done since I replace addresses with uint IDs, to save on space and increase performance. I have worked with Neo4j , it is fast, but there is no point in exporting data from one NoSQL format (LevelDB) to another NoSQL format like Neo4j. The key is to export it into SQL because it is much flexible than NoSQL.
    – Nulik
    Feb 15 '18 at 15:39
  • I just want to say I love your comment about, "what is the point of having a decentralized system...". I've been talking about this exact issue for months, and am not the first to bring it up. Here's one example: github.com/ethereum/go-ethereum/issues/2104 Unfortunately, it seems Ethereum devs want to stick their heads in the sand.
    – EvilJordan
    Feb 18 '18 at 20:00
  • You definitely want to look at QuickBooks. We have fully solve this issue. Including accumulating per account transaction lists internal and external plus we can do per block accounting.
    – Thomas Jay Rush
    Feb 19 '18 at 21:26
1

It turns out that Parity on SSD disk is syncing much faster.

EDIT: ...it is, but fetching internal transactions is very slow. Recently Geth 1.8.0 came out which seems to be syncing faster (though still very slow on HDD) and which supports tracers. We'll see how that goes.

Improve this answer
0

Nice qus,

As per geth help document there are 3 ways you can do sync to network:

  1. --syncmode full: Geth client will download Block header + Block data + full Validation [Is called eth full node]

  2. --syncmode fast: Geth client will download Block header + Block data + validate for last 1k transactions.

  3. --syncmode light: Geth client will download Current state + Asks nodes for as its need. [Light node [It will request missing blocks from full nodes]

You can change syncmode my specifying --syncmode along with command prompt. fast is good. But if you dont have time and space try to use light.

Improve this answer
1
  • 1
    Thank you for the answer, however I don't see how this applies to my question? AFAIK I need syncmode full to get the internal transactions and I am looking for a faster way to achieve this, as syncmode full is very slow.
    – johndodo
    Feb 5 '18 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.