0

I'm writing a nodejs program that gets a large amount of blocks (>100,000) and looks at each transaction inside to filter / analyse the data.

I'm currently using web3 (v1) getBlock function:

eth.getBlock(i, true, handleFunction)

However, even though the handleFunction is called asynchronously, my program (if not manually throttled) blasts out 100,000 requests, overheating the infura websocket I think...

I have manually made a throttle to make only 10 requests and wait 'till all of them are handled, then go to the next 10.

My question: I imagine that a lot of people face this same problem, so is there any method from web3 more appropriate to request multiple blocks? Or is there any design aspect I can improve on?

  • Would running your own light node be out of question? – Vignesh Karthikeyan Jul 6 '18 at 13:24
1

Sorry for the self-promotion, but you can use QuickBlocks to do the same thing much more effectively because it will additionally cache the blocks locally so that the next time you do the same (or similar) query, the blocks will be local and therefore much faster. We're working on a wrapper for QuickBlocks to be callable from node.js, but that's not ready yet. It works as a command line currently.

|improve this answer|||||
  • Yes I looked at that, but I'd have to wait for nodejs version... I have a setup where I save results locally myself and do things like extract certain fields to a csv file etc.etc. Basically if only etherscan.io would allow for more data to be exported / get through their API all would be much easier, but they have such strict limits... – mesqueeb Jul 7 '18 at 5:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.