I have a 3-node quorum setup that has been running from Jan 2019. The nodes were working without a problem until Sept 18 2019 when the network started to throw errors on submitting transactions with the following message in its logs:

WARN [09-18|01:28:34.874] Failed to rotate local tx journal        err="open /home/ubuntu/node1/blockchain/qdata/data/geth/transactions.rlp.new
: too many open files"

I force-killed the geth process and then restarted it. Now when I check the latest block number in geth console, the returned number is 13071

> eth.blockNumber

but when I try to get block by number, I am able to get upto block 148401

> eth.getBlock(148401)
  difficulty: 131072,
  extraData: "0x0000000000000000000000000000000000000000000000000000000000000000f84433b841b7c257ce2e4831391a13c4b69d39dc824f414871c5863f6963f8fc4a4b67a30043dbfc3ec237cf6186539467bc01336e967994de60dcae9321c7ba40aa72410900",
  gasLimit: 800000000,
  gasUsed: 4780949,
  hash: "0xf9e431d0167bef9dd8c0363e04b7c0d58ce74ed11fbc04e37fa555a8a38e32af",
  logsBloom: "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000084000100000000200000000000080000000000000000000000408000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000020000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000",
  miner: "0x0000000000000000000000000000000000000000",
  mixHash: "0x0000000000000000000000000000000000000000000000000000000000000000",
  nonce: "0x0000000000000000",
  number: 148401,
  parentHash: "0x467c9c15ff20af23cecd1f83658ea90e0011446ad8c8a8e335e0756a96cc0fef",
  receiptsRoot: "0x567ee8dc3abe2e6d112ebbf6e4292d481349b0adc9ffff703a1e050d5e4b2487",
  sha3Uncles: "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
  size: 19526,
  stateRoot: "0x22c49a8e9e7231fcb7a0726a68d5e86cbb6a4bda06e508c2a6d6ce6cc7cbdea7",
  timestamp: 1568768769489746200,
  totalDifficulty: 19451215872,
  transactions: ["0xf271425b66a5b62432a61a8d2e732c043c3cfe9f8390323d860db89e74ccd238"],
  transactionsRoot: "0x0e62c8102464cbb6662d491e7a44e1c8d7ebfd64620940ec064695250f2810f7",
  uncles: []

The timestamp 1568768769489746200 of block 148401 corresponds to Sept 18 2019 - the date when the issue started to appear. Now, if I submit a transaction, the transaction will be written on top of block 13071 as is evident from the timestamps:

> eth.getBlock(13070).timestamp
1570525805669591800 (Oct 8, 2019)

> eth.getBlock(13071).timestamp
1570528362009161000 (Oct 8, 2019)

> eth.getBlock(13072).timestamp
1559915961573821400 (Jun 7, 2019)

Probably because of this, web3 is not able to correctly detect new deployed contracts since it is expecting a block added event which is not happening. I made a few attempts to solve this issue including copying the chaindata folder from one node to the other two, but none of that seems to work.

I can see that this is a very specific question to ask, but there is a lot of data stored on the chain and I'd be really greatful if someone can show a way out of this mess with the least loss of data. Thanks a lot.

More info:

$geth version

Version: 1.8.12-stable
Git Commit: 8f678278aab02529cc56d934606e5b275b94f6cf
Quorum Version: 2.2.0
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1337
Go Version: go1.9.7
Operating System: linux

Pre-error Geth start command

PRIVATE_CONFIG="${TESSERA_IPCFILE}" nohup ${QUORUM} --datadir "${QUORUM_DATADIR}" --networkid 10 --raft --rpc --rpcaddr --rpcport 22000 --rpccorsdomain "*" --rpcvhosts "*" --rpcapi admin,db,eth,debug,miner,net,shh,txpool,personal,web3,quorum,raft --emitcheckpoints --raftport 50401 --port 21000 --unlock 0 --password "${QUORUM_PWDFILE}" >> "${QUORUM_LOGFILE}" 2>&1 &

Post-error Geth start command (--gcmode archive added)

PRIVATE_CONFIG="${TESSERA_IPCFILE}" nohup ${QUORUM} --datadir "${QUORUM_DATADIR}" --networkid 10 --gcmode archive --raft --rpc --rpcaddr --rpcport 22000 --rpccorsdomain "*" --rpcvhosts "*" --rpcapi admin,db,eth,debug,miner,net,shh,txpool,personal,web3,quorum,raft --emitcheckpoints --raftport 50401 --port 21000 --unlock 0 --password "${QUORUM_PWDFILE}" >> "${QUORUM_LOGFILE}" 2>&1 &
  • Hi, are you able to upgrade your nodes to the latest release? There are a few fixes in raft implementation between 2.2.0 and 2.3.0 that may help here.
    – fixanoid
    Oct 14, 2019 at 16:02
  • Hi @fixanoid, I posted the issue in slack, and you Quorum folks were super helpful in following up. Tried a few different suggested fixes, none of which seemed to work. So, I archived this version, re-made a new network with 2.3.0 and migrated my data again. It would've been better if there were an option for raft to export data till a particular block similar to geth's export. Any suggestions on how to future-proof the data would also be very welcome.
    – KBhokray
    Oct 16, 2019 at 7:22


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.