I'm writing a nodejs program that gets a large amount of blocks (>100,000) and looks at each transaction inside to filter / analyse the data.
I'm currently using web3 (v1) getBlock function:
eth.getBlock(i, true, handleFunction)
However, even though the handleFunction is called asynchronously, my program (if not manually throttled) blasts out 100,000 requests, overheating the infura websocket I think...
I have manually made a throttle to make only 10 requests and wait 'till all of them are handled, then go to the next 10.
My question: I imagine that a lot of people face this same problem, so is there any method from web3 more appropriate to request multiple blocks? Or is there any design aspect I can improve on?