How do Block Explorers keep their databases in sync?

Question

If we take blockchain.info; how do they know when a new block has arrived?

I understand that there is a full node running on the background, how would they have setup their infrastructure, so that their db which serves the users, is in sync with the full node db? I cannot think of a way to be alerted of when a new block arrives. It's not like the full node will tell you that we have a new block right? So do they watch the leveldb for changes?

Also, is it possible that they have full nodes that only respond to requests from blockchain.info?

Answer 1

I cannot think of a way to be alerted of when a new block arrives. It's not like the full node will tell you that we have a new block right?

I believe that it will. Check the events you can subscribe to via the websocket interface.

But you could instead just poll every 500ms (or whatever). There are only new blocks every 10-15 seconds. Being a few hundred milliseconds behind won't hurt you.

Answer 2

You can easily poll on the RPC interface for getBlockByNumber (“latest”) which will return the latest block. That’s not difficult at all because as the other answer says, you only get a block every 15 seconds or so.

The difficult part is that more often than you might think, the chain re-orgs, which means a block changes. So you have to revisit blocks to make sure there aren’t reorgs.

A lot of explorers even go a step further and scrape pending transactions which aren’t even in a block yet, but in the pending transaction queue. There’s an RPC interface for this too.

Some explorers even dig into to guts and read the database directly, as you mention.