How to send a lot of transactions to ethereum, using js

Question

I'm trying to send a lot of transactions to Ethereum with one program written in js. I use node js, web3 and infura ropsten. The problem is: if i send transactions at once, most of them just disappear. Trying to solve this problem, i send transactions with intervals. It works, but very slowly. I spend hour to send only 100 transactions. Is there any solution how to make it work faster and right? I was thinking about sending transaction after previous one starts pending, but I don't see how can I do that. Function sendRaw gets transaction number only after some time. This code reads the file, gets address, amount and optional data and uses method of smart contract to transfer tokens. Here is code:

function sendRaw(rawTx) {
var privateKey = new Buffer(key, 'hex');
var transaction = new tx(rawTx);
transaction.sign(privateKey);
var serializedTx = transaction.serialize().toString('hex');
web3.eth.sendRawTransaction(
'0x' + serializedTx, function(err, result) {
    if(err) {
        console.log(err);
    } else {

        console.log(result);
        Ntrans=result;
    }
});
}


var nonce = web3.eth.getTransactionCount(address);
var gasPrice = web3.eth.gasPrice;
var gasLimit = 90000;

var fs = require("fs");
var buf1 = new Buffer(1024);
var buf2 = new Buffer(1024);
var buf3 = new Buffer(1024);
var i = 0;
var j = 0;
var k = 0;

var fd = fs.openSync('/home/kate/Desktop/file.txt', 'r+');


function recurs()
{
    if(k==5) return -1;
    j = 0;
    do
    {
          fs.readSync(fd, buf1, j, 1, i);
          i++;
          j++;
    }
    while(buf1[j-1] != 32);
    AddressC = String(buf1.slice(0, j-1))
    console.log(AddressC);
    j = 0;
    do
    {
          fs.readSync(fd, buf2, j, 1, i)
          i++;
          j++;
    }
    while(buf2[j-1]!=32);
    ValueT = Number(buf2.slice(0, j-1))
    console.log(ValueT);
    j = 0;
    do
    {
          fs.readSync(fd, buf3, j, 1, i);
          i++;
          j++;
    }
    while(buf3[j-1]!=10);
    TxC = String(buf3.slice(0, j-1));

    txOptions =
    {
         nonce: web3.toHex(nonce),
         gasLimit: web3.toHex(gasLimit),
         gasPrice: web3.toHex(gasPrice),
         to: contractAddress

    }

    console.log(TxC);
    console.log(txOptions);
    rawTx = txutils.functionTx(interface, 'foreignBuy', [AddressC, 
    ValueT, TxC], txOptions);
    sendRaw(rawTx);
    k++;
    nonce++;
   /* while(web3.eth.getTransactionReceipt(Ntrans)=="null")
    {

    } */
}

setTimeout(function(){
    recurs();
}, 5000);
}
recurs();

Answer 1

Most confusion with handling high volumes of transactions stems from hidden assumptions that will bite at scale.

There are some important and non-obvious facts to understand.

1. Transaction nonce is managed by the sender.
2. Transactions from the same sender will be mined in nonce order.

Implicitly, it follows that:

3. If a transaction isn't mined, transactions with a higher nonce cannot be mined. A gap would violate #2 and this is why an account can seem to be jammed.

Lastly:

4. Miners have full discretion about including transactions in blocks or ignoring them.

It is possible to rapidly submit transactions but it demands precision at the transaction level. Things can go wrong, such as:

1. GasPrice too low
2. Transaction lost (no reason)

Missing transactions are ambiguous. They may indeed be mined if one waits long enough, or they may never be mined. Ambiguity is a problem for non-trivial activities at scale. The process needs to do something pro-active to eliminate the ambiguity.

For each transaction, increment the nonce and send, then wait for the transaction to appear in the chain with confirmations. After a few minutes, start dealing with the failure case.

For everyday use, it can be sufficient to send a cancellation transaction. Here is a handy trick to "unjam" a wallet that sent a transaction with a gasPrice too low. The account should send zero ether to itself specifying with the nonce of the transaction to cancel and a gasPrice higher than the transaction to cancel. When the replacement transaction appears in a block, the original transaction (whatever it was) will not mine, owing to the higher gas price and identical nonce - a confirmed cancellation. It is advisable to wait for several confirmations as you would for any other transaction.

The same principle can be applied to retry. Use the same data and nonce as the missing transaction and a higher gasPrice. This implies that the sender is managing the nonce as it goes and it knows the data, payload, nonce and gasPrice of every transaction it sent so it can address transaction failures with precision.

For more insight into the technical issues, have a look at the Singleton Nonce Manager discussion over here: Concurrency patterns for account nonce

Hope it helps.

Answer 2

Perhaps the web3 1.0 batch methods would work?

var contract = new web3.eth.Contract(abi, address);

var batch = new web3.BatchRequest();
batch.add(web3.eth.getBalance.request('0x0000000000000000000000000000000000000000', 'latest', callback));
batch.add(contract.methods.balance(address).call.request({from: '0x0000000000000000000000000000000000000000'}, callback2));
batch.execute();

Answer 3

This was a common problem in the gaming world, hence why EIP 1155 was formulated. You and others may benefit from either dealing solely with ERC-1155s or by wrapping the tokens of interest (assuming they're ERC-20 or ERC-721) in ERC-1155s.