What's the fastest way to mine a transaction so I can test contracts quickly?

Question

I am developing a solidity automated test framework in nodejs that uses events/logging extensively. The use of events eliminates (I believe) all the RPC simulators out there. It's also frustrating to chase down simulator problems, I'd rather use a real server that's modified to go faster.

I've hacked up geth to mine transactions every 4 seconds instead of 15-20 seconds, but it's still too slow. I'd like to run as fast as the VM can execute code.

I'm going to post my patch to geth to do this, but I and others would really like a better answer to this question. It really slows down development to have to wait multiple seconds for each transaction to complete. My test suite of about 60 transactions takes about 4-5 minutes to run, and my test suite keeps getting longer...

Please advise on making this geth mine even faster than the answer I propose, or an alternative suggestion as well.

Answer 1

Patch to geth to make it mine faster. I would really like help in making this about 2x-4x faster than it is now, but it is 5x faster than stock geth:

[March 9, 2016 EDIT - this new patch now mines every second now. My unit tests that runs 78 transactions runs in about 160 seconds. I should put out a Gist for this...]

    diff --git a/eth/fetcher/fetcher.go b/eth/fetcher/fetcher.go
index d88d919..f03fd30 100644
--- a/eth/fetcher/fetcher.go
+++ b/eth/fetcher/fetcher.go
@@ -34,7 +34,7 @@ import (
 const (
    arriveTimeout = 500 * time.Millisecond // Time allowance before an announced block is explicitlrd
    gatherSlack   = 100 * time.Millisecond // Interval used to collate almost-expired announces witfs
-   fetchTimeout  = 5 * time.Second        // Maximum alloted time to return an explicitly requestebk
+   fetchTimeout  = 1 * time.Second        // Maximum alloted time to return an explicitly requestebk
    maxUncleDist  = 7                      // Maximum allowed backward distance from the chain head
    maxQueueDist  = 32                     // Maximum allowed distance from the chain head to queue
    hashLimit     = 256                    // Maximum number of unique blocks a peer may have annoued
diff --git a/eth/peer.go b/eth/peer.go
index 15ba22f..0201057 100644
--- a/eth/peer.go
+++ b/eth/peer.go
@@ -41,7 +41,7 @@ var (
 const (
    maxKnownTxs      = 32768 // Maximum transactions hashes to keep in the known list (prevent DOS)
    maxKnownBlocks   = 1024  // Maximum block hashes to keep in the known list (prevent DOS)
-   handshakeTimeout = 5 * time.Second
+   handshakeTimeout = 1 * time.Second
 )

 // PeerInfo represents a short summary of the Ethereum sub-protocol metadata known
diff --git a/miner/worker.go b/miner/worker.go
index 754a6fc..2b62b59 100644
--- a/miner/worker.go
+++ b/miner/worker.go
@@ -466,7 +466,7 @@ func (self *worker) commitNewWork() {
            tstamp = parent.Time().Int64() + 1
    }
    // this will ensure we're not going off too far in the future
-   if now := time.Now().Unix(); tstamp > now+4 {
+   if now := time.Now().Unix(); tstamp > now {
            wait := time.Duration(tstamp-now) * time.Second
            glog.V(logger.Info).Infoln("We are too far in the future. Waiting for", wait)
            time.Sleep(wait)
diff --git a/p2p/rlpx.go b/p2p/rlpx.go
index aaa7338..ce82eb7 100644
--- a/p2p/rlpx.go
+++ b/p2p/rlpx.go
@@ -57,7 +57,7 @@ const (

    // total timeout for encryption handshake and protocol
    // handshake in both directions.
-   handshakeTimeout = 5 * time.Second
+   handshakeTimeout = 2 * time.Second

    // This is the timeout for sending the disconnect reason.
    // This is shorter than the usual timeout because we don't want
diff --git a/params/protocol_params.go b/params/protocol_params.go
index dcc17e0..21a49c5 100755
--- a/params/protocol_params.go
+++ b/params/protocol_params.go
@@ -29,10 +29,10 @@ var (
    CallNewAccountGas      = big.NewInt(25000)  // Paid for CALL when the destination address didn'e.
    TxGas                  = big.NewInt(21000)  // Per transaction. NOTE: Not payable on data of cas.
    TxDataZeroGas          = big.NewInt(4)      // Per byte of data attached to a transaction that u.
-   DifficultyBoundDivisor = big.NewInt(2048)   // The bound divisor of the difficulty, used in thep.
+   DifficultyBoundDivisor = big.NewInt(1)      // The bound divisor of the difficulty, used in thep.
    QuadCoeffDiv           = big.NewInt(512)    // Divisor for the quadratic particle of the memoryo.
-   GenesisDifficulty      = big.NewInt(131072) // Difficulty of the Genesis block.
-   DurationLimit          = big.NewInt(13)     // The decision boundary on the blocktime duration e.
+   GenesisDifficulty      = big.NewInt(1)      // Difficulty of the Genesis block.
+   DurationLimit          = big.NewInt(1)      // The decision boundary on the blocktime duration e.
    SstoreSetGas           = big.NewInt(20000)  // Once per SLOAD operation.
    LogDataGas             = big.NewInt(8)      // Per byte in a LOG* operation's data.
    CallStipend            = big.NewInt(2300)   // Free gas given at beginning of call.
@@ -57,7 +57,7 @@ var (
    CreateDataGas        = big.NewInt(200)    //
    Ripemd160Gas         = big.NewInt(600)    //
    Ripemd160WordGas     = big.NewInt(120)    //
-   MinimumDifficulty    = big.NewInt(131072) // The minimum that the difficulty may ever be.
+   MinimumDifficulty    = big.NewInt(1)      // The minimum that the difficulty may ever be.
    CallCreateDepth      = big.NewInt(1024)   // Maximum depth of call/create stack.
    ExpGas               = big.NewInt(10)     // Once per EXP instuction.
    LogGas               = big.NewInt(375)    // Per LOG* operation.

Here's my genesis.json. There's lots of other examples out there:

{
    "nonce": "0xdeadbeefdeadbeef",
    "timestamp": "0x0",
    "parentHash":     "0x0000000000000000000000000000000000000000000000000000000000000000",
    "extraData": "0x0",
    "gasLimit": "0x8000000",
    "difficulty": "0x001",
    "mixhash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "coinbase": "0x3333333333333333333333333333333333333333",
    "alloc": {
        "0x662eb875377b1288f07baa6139432553b7b8b03e": {
            "balance": "1000000000000000000000"
         }
    }
}

~

And finally the all important command line. Note I've changed the cost of gas to 50 wei instead of 50Gwei. I probably could have done better to just give a bunch of accounts 10M ether. Also I need to add reading a script that unlocks all the accounts:

#!/bin/bash
bin/geth --datadir /home/Pauls/testing/00 --port 30310 --rpc --rpcport 8110 --networkid 4567890 --dev --lightkdf --nodiscover --maxpeers 0 --vmdebug --verbosity 6 --pprof --genesis /home/Pauls/sandboxes/myproj/bin/genesis.json --gpomin "50" --gpomax "50" --pprofport 6110 console 2> /home/Pauls/testing/00/geth.log

~

Answer 2

Test-RPC is a great application for doing exactly this. It runs a simulated blockchain, which you can connect to via RPC via web3, and doesn't need to mine or network at all.

Answer 3

An alternative is to use pyethereum.tester, pytester for short (and not to be confused with py.test).

pytester is part of pyethereum, the Ethereum Foundation's python client which has undergone a security audit (like Geth) and part of the Ethereum Bounty Program. Agree it's frustrating to chase down simulator problems and I wouldn't classify it as one: it has been very reliable and a bug found using pytester could well be a consensus issue in pyethereum itself and worthy of a bounty.

https://github.com/ethereum/pyethereum/wiki/Using-pyethereum.tester

As you specifically asked about mining, here's an example of mining 105 blocks s.mine(105):

from ethereum import tester as t
def test_make_reports():
    global initial_gas
    initial_gas = 0
    t.gas_limit = 100000000
    s = t.state()
    c = s.abi_contract('functions/output.se')
    gas_use(s)
    c.initiateOwner(1010101)
    c.reputationFaucet(1010101)
    assert(c.submitReportHash(1010101, 3232, -1, 222, 0)==-2), "Nonexistant event check broken"
    event1 = c.createEvent(1010101, "new event", 5, 1, 2, 2)
    bin_market = c.createMarket(1010101, "new market", 2**58, 100*2**64, 184467440737095516, [event1], 1)
    s.mine(105)
    gas_use(s)
    c.incrementPeriod(1010101)
    report_hash = c.makeHash(0, 2**64, event1)
    gas_use(s)
    print c.submitReportHash(1010101, report_hash, 0, event1, 0)
    assert(c.submitReportHash(1010101, report_hash, 0, event1, 0)==1), "Report hash submission failed"
    gas_use(s)
    s.mine(55)
    gas_use(s)
    assert(c.submitReport(1010101, 0, 0, 0, 2**64, event1, 2**64)==1), "Report submission failed"
    gas_use(s)
    print c.getUncaughtOutcome(event1)
    gas_use(s)
    print "Test make reports OK"

pytester also works with Solidity code. The Ethereum Alarm Clock uses pytester to mine many blocks so that it can test, for example, that the clock wakes up far in the future. The wait_for_block is via Populus, a framework which uses pytester; Populus also has an option of testing using an actual Geth node.

Another Solidity example that suggests the power of pytester to test arbitrarily complex scenarios, is with ether_ad, under Ethereum's dapp-bin.