I'd like to compare the transaction speed of private Ethereum(ex. PoA), Eris and HydraChain.
How should we benchmark them?
Ethereum Stack Exchange is a question and answer site for users of Ethereum, the decentralized application platform and smart contract enabled blockchain. It only takes a minute to sign up.Sign up to join this community
I don't think you can do that realistically. Ethereum doesn't have a theoretical upper limit on the transaction count. When you push in more transactions than the blocks fit in, the block sizes are gradually increased to cater for them, so in theory it's unlimited.
In practice after pushing the block size above some threshold, the network propagation time, old block processing in full nodes and new transaction processing in miners will start to saturate their computing capacity. That threshold would be your throughput limit. The problem is, that this highly depends on the network participants (i.e. if full nodes are big machines with plenty of bandwidth, this will be high; whereas small nodes with little bandwidth draw this downwards).
All in all you could measure some throughput for your own private network, but it won't be even remotely reflective of the main network (which may be higher or lower). The only way to accurately measure the main networks capacity is to essentially SPAM it full of transactions, which would cost you an enormous amount.
Before the Frontier launch (since when a ton of polishes, fixes, performance tunings, optimizations went in) we had an Olympic test network in place and gave out prizes to try and break/overload stuff. Then I think we reached a throughput of about 25 TPS, but again, that was a fraction of our current network size and resources, the influence of which we have no idea of. The record for the main net (i.e. real, paid transactions) was something along 8 TPS, but that didn't even dent the system.
At eris, we've been working on a benchmarking suite.
The basic idea we pursue is to first establish a network of set configuration suite (same data center, various node numbers, cross data centers, cross cloud providers, etc.).
Then we run that network against a set number of transactions (basically flog the mempool) where txes are randomly sent via various network nodes.
Later we add a set suite of basic contracts and flog the mempool with call tx's.
Within a few months we'll have the test framework stable enough for open sourcing and reporting on more fully.
We don't do it on our side, but assume a similar paradigm would work for both geth and hydra clients.