How long did a transaction spend in the mempool?

Question

I'm trying to create some statistics on how long a transaction spends in the mempool before being included in a block, so I can reason about future gas prices. It would appear transactions in Ethereum don't receive a timestamp until they're mined, so instead, what is the best way of observing when a transaction was submitted? If I ask Parity to inform me of all new pending transactions, and recording the local timestamp at that time, quite often the block it ends up being included in, has a timestamp that is prior to when I saw it, so this seems wildly unreliable.

Wondering how other API's obtain this information reliably, and how "gas station" sites intelligently calculate this information.

Answer 1

It is done similarly as you said. An application frequently checks to parity/geth the pending queue and stores in a database the changes.

You can have this application running in several nodes in different cloud providers to have a more broad view.

Some times block explorer companies will modify the clients their run to suit their usage. Like instead of polling for changes just trigger them when a new transaction appear.