3

I know that the state of the txpool of a geth node can be queried using the txpool API. However, I was wondering if it is somehow possible to subscribe to new transactions entering a geth node's txpool, possibly on a websockets or local socket rpc connection? I'd like to write the program that subscribes to new txpool transactions in Go as well.

I discovered this subscription in go-ethereum internally, but it doesn't look like it's exposed, unfortunately.

Note that I'm not only talking about local pending transactions (answered here), but all transactions arriving to the txpool.

Background: I'd like to do research on frontrunning. The authors of the Flash Boys 2.0 paper released their geth fork on github, which they used to observe gas auctions in the txpool. However, using a fork is not as reusable and maintainable than having the possibility to extend geth's API so that external applications can subscribe to txpool udpates.

4
  • Hey sebastian did you find something to solve the problem?
    – Cristofor
    Jul 10, 2021 at 12:45
  • Not really. I wanted to use it from within Go code but go-ethereum doesn't expose the pendingTransactions subscription on the ethclient API. It seems to be available (sometimes) wiht JS web3 though, see Eugene's answer.
    – sebastian
    Jul 11, 2021 at 15:36
  • I did find asolution actually but it uses web3.py
    – Cristofor
    Jul 12, 2021 at 16:02
  • do you have github? or twitter?
    – Cristofor
    Jul 12, 2021 at 16:02

4 Answers 4

6

We have been running ethviewer.live since late 2017 with a modified geth client. It has certainly been painful to keep up with latest versions of geth. Also, we have had outages when we were unable to keep up with forks. As suggested by @Eugene I tried with web3.eth.subscribe('pendingTransactions') and it seems to be working fine. One caveat though -- Subscription API on Geth is best-effort, which means it may occasionally miss out a TX or block. If you need all data, you need to do state management and pull any missing data using Web3.eth functions. As per GitHub discussions, Geth may not fix this, while Parity seems to fix it.

As you are interested in frontrunning, be aware that a node gets large fraction of transactions after the block containing them is received. For e.g., with --maxpeers = 50 on ethviewer.live about 40% of transactions arrived after the block containing them arrived (i.e., either block timestamp or arrival time < transaction arrival time). I also checked by dumping blockchair.com transaction pool and realized the fraction is about 55% (don't know what they used as --maxpeers). These measurements were done for several days. So you may want to play with high --- maxpeers values in addition to having good bandwidth.

4
  • How did you go about testing your claim that 40% of txs arrived after the block containing them arrived?
    – samlaf
    Mar 3, 2022 at 4:06
  • We locally log the time that block and TX announcements are received by our node. For a ~2 months trace, ~40% of the TXs were received by our node after the block that contain them were received, i.e., block had an earlier local timestamp than TXs. I tried with --maxpeers = 25 and = 5. Didn't observe a significant improvement. Mar 5, 2022 at 5:33
  • @DilumBandara I have experienced the same phenomen. About 35-40% come in to late, even by more than 10 seconds! Have you experimented with nodes in different locations around the world? Is there any reason to explain this? Looking at ethereum.stackexchange.com/a/28679 it should not be an issue to receive all but a few outliers in time.
    – Plus Ultra
    Aug 8, 2022 at 11:50
  • @PlusUltra We haven't tried it across geographies. I think it comes down to a factors such as bandwidth, peers connected to, and transaction originating peer. We have seen sometimes 200+ new transactions arrive to the pool within a 1 sec. Given transaction arrival is bursty, network is massive, we are connected only to 25 nodes (out of 5,000+), and inter-block time for some blocks is 2-5 sec, there's no way a gossip protocol is going to disseminate them before the next block arrives. There's an error in my previous comment. It should be "I tried with --maxpeers = 25 and = 50" Aug 9, 2022 at 21:06
4

It's a bit counterintuitive, but while web3.eth.getPendingTransactions() returns only local pending transactions (from transaction field value matching accounts within the node), web3.eth.subscribe('pendingTransactions') will return a stream of global pending transactions.

Please find usage examples here.

3
  • 1
    Thanks, didn't know that subscribe('pendingTransactions') does cover all incoming txs in the pool. Unfortunately, this doesn't seem to be exposed in go-ethereums ethclient.
    – sebastian
    Nov 11, 2020 at 16:43
  • Eugene, I tried your examples from the link. I am connected directly on a node via icp. The problem is, I get only a stream of tx which are already mined. Not pending anymore. I have to say I do not use the geth client but opera of fantom. Do you have an idea if this is a general setting maybe somewhere which can be changed?
    – flo
    Oct 7, 2021 at 14:12
  • How does the node get access to global pending txs without them also becoming local? Is it because subscribe(pendingTransactions) queries neighboring nodes' txpools, which might have larger sizes and not fit in the current node's local mempool?
    – samlaf
    Mar 3, 2022 at 4:04
3

Supplement to Eugene's answer, I have this fiddle testing the speed of two nodes catching pending transactions. It was built with web3.subscription. Hope it helps.

Explanation of the code is also here.

1

You can create an account with Infura and subscribe to their pending pool as easy as this:

const { time } = require("console");
const Web3 = require("web3");
let fs = require('fs');


const web = new Web3();

const rpcUrl = 'INFURA NODE RPC'

const web3 = new Web3(rpcUrl)

const message = fs.createWriteStream("./pendingPool.txt");

message.write("tx-hash" + ' ' + "Arrival" + "\n");

var subscription = web3.eth.subscribe('pendingTransactions', function (error, result) {
    if (error)
        console.log(error)
})
    .on("data", function (result) {
        message.write(result + ' ' + Date.now() + "\n");
    });

This code listens to the pending pool and writes the tx and arrival time to a file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.