28

Augur uses event logs to store data that never needs to be accessed on-contract, since it is about 10x cheaper than on-contract storage. However, I've noticed that retrieving event logs (e.g., using eth_getLogs) takes significantly longer than retrieval of on-contract data. (I know filters let you listen for new event logs, but I'm specifically concerned with looking up old event logs.)

I don't really understand how event logging works under the hood. Currently, we're indexing one (or more) of the log's arguments, then using eth_getLogs to search on the indexed argument. For example, all Augur trades are logged and indexed by both market ID and account. Our UI looks up trades made by the current account as part of its initial loading. Right now this takes about 10 seconds to finish. The eth_getLogs request is made asynchronously, and interleaved with the other initial market loading tasks, so it's not overly annoying at the moment. However I'm concerned about how this will scale as both Augur and the Ethereum blockchain as a whole grow.

My question: will retrieving event logs become prohibitively slow as the blockchain becomes larger? More specifically, what is the time complexity of eth_getLogs? (Is blockchain size the relevant variable, or something else? Is eth_getLogs even the correct RPC request for what I'm trying to do? Basically any insight into this problem is appreciated!)

Improve this question

1 Answer 1

Reset to default
24

I will try to anwser your question. I worked on bloom filters in cpp-ethereum and Parity.

will retrieving event logs become prohibitively slow as the blockchain becomes larger?

Not necessarily. Everything depends on the implementation, logs density (average number of logs / block) and number of cache levels.

More specifically, what is the time complexity of eth_getLogs?

In worst case, where every block contains log matching your query it is 0(n). But it's rarely a case. Bloom filters utilize probability of false positives, so the more sophisticated your filter is (more topics it has), the faster you will get your results.

Is eth_getLogs even the correct RPC request for what I'm trying to do?

Yes

To summarise, I believe, that 10s response time is caused by sub-optimal imlementation of bloom filters in go-ethereum. Here are the results of benchmarks with parity:

Find all logs from block 0 to 986082 with address: 0x33990122638b9132ca29c723bdf037f1a891a70c (should return 1602 logs). 
time curl -X POST --data '{"id":8,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0xf0be2", "address": "0x33990122638b9132ca29c723bdf037f1a891a70c"}]}' -H "Content-Type: application/json" http://127.0.0.1:3030 >> /dev/null

geth first request:

real    0m17.003s

geth second request (I assumed, that results should be cached after the first one).

real    0m18.023s

parity first request (~24x faster then geth)

real    0m0.770s

parity second request (~30x faster then geth)

real    0m0.668s

The gap between Parity and geth closes dramatically when there are no logs to be found:

Find all logs from block 0 to 986082 with address: 0x33990122638b9132ca29c723bdf037f1a891a70d (address does not exist, 0 logs returned). 
time curl -X POST --data '{"id":8,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0xf0be2", "address": "0x33990122638b9132ca29c723bdf037f1a891a70d"}]}' -H "Content-Type: application/json" http://127.0.0.1:3030 >> /dev/null

geth first request:

real    0m0.022s

geth second request 

real    0m0.021s

parity first request (4x slower than geth)

real    0m0.080s

parity second request (1.5x slower than geth)

real    0m0.030s
Improve this answer
12
  • 1
    I believe that both Go, Cpp and Parity use a similar approach (please correct me if I'm wrong) to bloom filters and log retrieval using mip maps (blooming on several different resolutions). Looking at the second request we can say that we're equally fast and quickness depends mostly on the resolutions of the mipmapping.
    – Jeffrey W.
    Mar 21, 2016 at 17:12
  • Jeff, it might be worth checking into geth's bloom filter implementation to see if it's suboptimal (as debris suggests). I can't speak for other dapps, but Augur makes heavy use of logging simply because it's so much cheaper than on-contract storage, so log retrieval performance is pretty critical to us! Am downloading Parity now to see how well it handles Augur's wild and crazy data...
    – tinybike
    Mar 22, 2016 at 5:19
  • 1
    After some benchmarking and testing it isn't the bloom filter but the db lookups. The bloom filter performs very well but unfortunately leveldb (which Parity doesn't make use of IIRC) sucks when it comes to performance :-/ It's a known issue we're looking in to. Thanks @debris for bringing this to our attention. The actual filtering only takes 96ms. The rest of the time is db ops overhead.
    – Jeffrey W.
    Mar 22, 2016 at 9:52
  • 1
    Interesting! RocksDB, perhaps? :)
    – tinybike
    Mar 22, 2016 at 9:57
  • 1
    @tinybike A year on from your question, how are you finding event log performance these days? Is querying indexed fields giving you acceptable performance for a DApp or do you find you need some other kind of caching?
    – Edmund Edgar
    Apr 19, 2017 at 3:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.