Subscribe to new txpool transactions on geth node

Question

I know that the state of the txpool of a geth node can be queried using the txpool API. However, I was wondering if it is somehow possible to subscribe to new transactions entering a geth node's txpool, possibly on a websockets or local socket rpc connection? I'd like to write the program that subscribes to new txpool transactions in Go as well.

I discovered this subscription in go-ethereum internally, but it doesn't look like it's exposed, unfortunately.

Note that I'm not only talking about local pending transactions (answered here), but all transactions arriving to the txpool.

Background: I'd like to do research on frontrunning. The authors of the Flash Boys 2.0 paper released their geth fork on github, which they used to observe gas auctions in the txpool. However, using a fork is not as reusable and maintainable than having the possibility to extend geth's API so that external applications can subscribe to txpool udpates.

Answer 1

We have been running ethviewer.live since late 2017 with a modified geth client. It has certainly been painful to keep up with latest versions of geth. Also, we have had outages when we were unable to keep up with forks. As suggested by @Eugene I tried with web3.eth.subscribe('pendingTransactions') and it seems to be working fine. One caveat though -- Subscription API on Geth is best-effort, which means it may occasionally miss out a TX or block. If you need all data, you need to do state management and pull any missing data using Web3.eth functions. As per GitHub discussions, Geth may not fix this, while Parity seems to fix it.

As you are interested in frontrunning, be aware that a node gets large fraction of transactions after the block containing them is received. For e.g., with --maxpeers = 50 on ethviewer.live about 40% of transactions arrived after the block containing them arrived (i.e., either block timestamp or arrival time < transaction arrival time). I also checked by dumping blockchair.com transaction pool and realized the fraction is about 55% (don't know what they used as --maxpeers). These measurements were done for several days. So you may want to play with high --- maxpeers values in addition to having good bandwidth.

Answer 2

It's a bit counterintuitive, but while web3.eth.getPendingTransactions() returns only local pending transactions (from transaction field value matching accounts within the node), web3.eth.subscribe('pendingTransactions') will return a stream of global pending transactions.

Please find usage examples here.

Answer 3

Supplement to Eugene's answer, I have this fiddle testing the speed of two nodes catching pending transactions. It was built with web3.subscription. Hope it helps.

Explanation of the code is also here.

Answer 4

You can create an account with Infura and subscribe to their pending pool as easy as this:

const { time } = require("console");
const Web3 = require("web3");
let fs = require('fs');


const web = new Web3();

const rpcUrl = 'INFURA NODE RPC'

const web3 = new Web3(rpcUrl)

const message = fs.createWriteStream("./pendingPool.txt");

message.write("tx-hash" + ' ' + "Arrival" + "\n");

var subscription = web3.eth.subscribe('pendingTransactions', function (error, result) {
    if (error)
        console.log(error)
})
    .on("data", function (result) {
        message.write(result + ' ' + Date.now() + "\n");
    });

This code listens to the pending pool and writes the tx and arrival time to a file.