Web3 Event Filtering timeout

Question

This is the code I use to get all events from a certain contract.

Solidity:

event NewEvent(address indexed contractAddress, address newBOPAddress, address payer, uint commitThreshold, bool hasDefaultRelease, uint defaultTimeoutLength, string initialPayerString);

web3 code:

event = Factory.contractInstance.NewBOP({contractAddress:'0x384da3....'},{"fromBlock": 0, "toBlock": 'latest'});

event.get(function(err, res) {
  if(err){
    console.log(err.message);
  }
  else{
    console.log(res);
  }
});

If I specify a fromBlock that is relatively young, I get a fast response and everything works fine, if I use a block that is around 200,000 blocks in the past, I get this timeout error:

Gateway timeout. The request took too long to process. This can happen when querying logs over too wide a block range.

I tried this on ropsten with different nodes (metamask, myetherapi, infura) but always have the same problem. I aquired a private node with quicknode.io, which doesn't solve the problem, for users that need to use public nodes. I know that I could use a newer "fromBlock", but I would like to use the events of my factory contract as historical data, so I can display it to the user. And I found a few sources supporting, that it is technically possible, especially with indexed arguments to filter on:

https://media.consensys.net/technical-introduction-to-events-and-logs-in-ethereum-a074d65dd61e (part 3)

How to access the event log by knowing the contract address (web3)? (uses different filter options)

My exact question is: Do I need a full private node in every case for the desired performance, or are there other reasons / ways to make the event filtering faster (to stop the timeout)?

Thanks for any answer!

Answer 1

Even with a private node i don't think this will work. The Blockchain is not as queryable as a relational or non relational database. Its main goal is to keep consistency and a global state in each block, not performance when reading past events. So your option is to use pagination and iterate using a fromBlock and toBlock with a difference of let's say 1000 until you reach the latest block. But if you want to show this to a user, then it will still take some time, so my recommendation is fetch the data using pagination and store it in a database. Then in your user App just query from the database.

Answer 2

Do I need a full private node in every case for the desired performance, or are there other reasons / ways to make the event filtering faster (to stop the timeout)?

For the performance critical in the situations like this

• You maintain your own centralised database e.g. SQL

• This database offers queries for the events within any performance criteria you desire

• Database is populated from the past events

• A live daemon reads all new incoming blocks and adds events from those to the database

As described on the answers, Ethereum node is not optimised for performance criticial queries and it is better to do it yourself.

Some Python code that reads events from Ethereum node and populates SQL database with them