Measuring the efficiency/performance of a smart contract presented as an algorithm

Question

I am trying to theoretically analyze the expected performance of a smart contract presented in the form of an algorithm which consists of multiple functions.

What is the best way to measure the performance of the functions within?

Some suggest that measuring the amount of Gas is the best solution. However, measuring the gas cost is dependent on the code and would require the actual implementation of the smart contract. Some developer may implement the contract in ways better than others. Therefore, this may not be too accurate.

On the other hand, others suggest presenting the computational complexity in terms of the Big O notation. However, Big O notation is usually used for complex programs that might require a lot of time

What would be the better method and why?

Answer 1

As I see it, the only way to measure computational efficiency on the blockchain is using gas. There is not concept of "faster" or "slower" running contracts since execution of transactions happen at discrete intervals (when a block is mined).

Algorithms which are more complex, and will have a larger Big O complexity will manifest its complexity in the form of a higher gas cost to perform a function.

However, it is hard to theoretically measure the gas cost of a hypothetical smart contract. As you mentioned, people may implement contracts in different ways. There are a lot of gas saving behaviors programmers could adopt, but there are also a lot of "best practices" which encourage safety to the user. Those safety practices may add additional functions and checks which cause the contract's gas cost to go up.

Additionally, even given a specific contract with access to the full source code, it is hard to determine the gas needed for any particular transaction because many contracts can behave different based on the state of the contract. For example, when interacting with a contract, users 1-99 all have the same gas cost of X. However, user 100 may trigger some state-based action in the contract, causing them to do many more actions, and causing their gas cost to be 2X.

Order of operations for multiple transactions to the same contract will be very unpredictable, since it depends on a huge number of random factors like spreading the transaction across the network, the randomness of the miner who finds the right hash, order in which the transactions are packed into the block, etc...

The best thing you may be able to do is look at particular functions, break it down into its different components and possible states, and estimate the gas cost using the details starting on page 25 of the Ethereum yellow paper:

https://ethereum.github.io/yellowpaper/paper.pdf

Answer 2

Do it empirically and compare with other contracts (samples). The most precise measurement you can get.

in core/state_transition.go modify TransitionDb() function to capture current time , then do a difference with the time after the execution of your contract. Theses are the lines you have to modify:

if contractCreation {
    ret, _, st.gas, vmerr = evm.Create(sender, st.data, st.gas, st.value)
} else {
    // Increment the nonce for the next transaction
    st.state.SetNonce(msg.From(), st.state.GetNonce(sender.Address())+1)
    ret, st.gas, vmerr = evm.Call(sender, st.to(), st.data, st.gas, st.value)
}

This is how you would take the time:

start_ts:=time.Now().UnixNano() / int64(time.Millisecond)
if contractCreation {
    ret, _, st.gas, vmerr = evm.Create(sender, st.data, st.gas, st.value)
} else {
    // Increment the nonce for the next transaction
    st.state.SetNonce(msg.From(), st.state.GetNonce(sender.Address())+1)
    ret, st.gas, vmerr = evm.Call(sender, st.to(), st.data, st.gas, st.value)
}
end_ts:=time.Now().UnixNano() / int64(time.Millisecond)
log.Info(fmt.Sprintf("Call() time: %v ms",(end_ts-start_ts)))

The above code has not been tested.