Concurrency patterns for account nonce

Question

The account nonce is the only element in the transaction structure that prevents replay attack (as explained in this very good answer). It essentially forces sequentialisation of an account's tranctions.

This usually doesn't cause problems when transactions are submitted through the web3 interface to an Ethereum node (e.g. Geth, Parity), which then does the signing. The sequentialisation is done internally by the node, and thus is transparent to the end-user.

This is not the case when transactions are created and signed by a user application (i.e. not Geth or Parity), before sent over to a node (e.g. through web3.eth.sendRawTransaction).

Constructing the raw transaction requires obtaining an account's nonce, which can be done by web3.eth.getTransactionCount, as explained in this question. This works fine when no concurrency is involved.

However, consider an end-user that has to sign multiple transactions in parallel. The transaction sequentialisation becomes a responsibility of the user application.

"web3.eth.getTransactionCount" returns confirmed transaction counts only (see this question). As a result, when transactions are signed in parallel, unless we manipulate the count in a smart way, only one of them will succeed.

My question is: what are some good patterns to handle this situation?

Candidate 1: multiple keys/accounts per user

This does allow concurrent transactions. However, it introduces extra complexity in the user application. Plus, it's not too scalable: consider if tens or hundreds of concurrent transactions are needed. Furthermore, given that Hierachical Deterministic wallet is not native in Ethereum (I am aware there is the LightWallet), this doesn't sound like an easy solution.

Candidate 2: Loop and retry

This makes a loop that checks transaction sending result. If the return result indicates duplicate nonce (e.g. Geth would return 'replacement transaction underpriced'), re-try with a new nonce.

This solution is not portable: different clients return different strings for the same error. In practice, I seem to have noticed Geth sometimes does not even return the above string, but silently drops the other transactions (not confirmed as I haven't found a reliable way to reproduce). As a result, the detection doesn't work reliably.

Candidate 3: a singleton 'nonce manager'

Each transaction construction involves "applying for" a nonce from a global "nonce manager". The singleton "nonce manager" can have some smart logic to hand out ascending nonce numbers. This exploits the behaviour discussed here.

Not only this introduces a singleton (arguable an anti-pattern), but also it doesn't sound easy to get it right: if a low nonce transaction fails somehow (e.g. doesn't even get sent out to the network), the rest of the transactions will become stuck indefinitely. In general, it's a poor solution that introduces more complexity and problems than trying to solve.

Candidate 4: delegate to a local Geth/Parity node

This is not really a solution. The most obvious issue is the extra dependency doesn't seem to be justifiable, if it's merely for the nonce field.

Also, if the user application already manages private keys, this introduces extra complexity: the private keys will need to be co-managed (or copied) by the Geth or Parity node. Again, the complexity (and potential vulnerability) doesn't appear justifiable if it's only for dealing with the nonce.

---

Off topic. It appears to me the account nonce is not the best design in Ethereum. It forces user applications to either handle low-level protocol details, or to introduce dependency on a node (Geth/Parity). Either way, it adds disproportional bulkiness to user applications.

Answer 1

I haven't yet got to the point where I can test any of these things, but my gut feel is that the Singleton Nonce Manager would be the way to go, with a few enhancements:

• Make it a singleton transaction sender
• That has a maximum buffer or 'head' of pending transactions per 'from' address that matches or is less than the maximum number of transactions that the peer can have in its transaction pool queue per from address. Let us call these pending transactions 'slots.'
• Is implemented as a rolling queue, where oldest, confirmed transaction are deleted from the 'tail'.
• Each new slot gets a new nonce
• Each new slot is associated with an asynchronous call to sendRawTransaction. On failure of any kind, the slot becomes free.
• Any new transactions are added to the lowest free slot.
• In the event that the higher slots are pending for more than N milliseconds after a slot is freed on error, cancel the highest transaction by sending 0 eth to self with same nonce as highest slot nonce, and replaying that transaction in the freed slot.

The implementation details would require intimate familiarity of concurrent programming, locks, mutexs, async or whatever. Essentially the above is a guess as I have not worked directly yet with the RPC calls and cannot test it, but I think that's the bare bones of the route I would go. I see that the queue structure would offer an internal API for listing pending transactions, processed, and supporting other UI operations as a bonus.

Answer 2

getTransactionCount can also include pending transactions which is a better way to determine the nonce for a subsequent raw transaction. This solution is simply but unlikely without it's own issues. But for a firing off a handful of transactions in close succession it seems to work as expected.

nonce: web3.toHex(web3.eth.getTransactionCount(fromAddress, "pending")),

Answer 3

It all depends on your context. If you're an end user trying to issue multiple transactions yourself, you could simply store the nonce somewhere and increment it on each parallel transaction. Obviously, you want to avoid creating gaps in the nonce, but as long as you're 100% certain that you can cover the funds when the transactions do execute, a gap is fine temporarily. The worst thing that can happen is that a nonce gets played twice, in which case, you've created invalid transactions.

If you're acting on behalf of a user and trying to issue bulk transactions, you could just use an atomic database operation to store the nonce. This could work in a hypothetical batching wallet attempting to send multiple transactions at once. You'd probably want to slap a disclaimer on that a user should avoid using the key while interacting with the wallet, to prevent the internal counter from going out of sync.