Geth Version: 1.8.17-stable Git Commit: 8bbe72075e4e16442c4e28d999edee12e294329e Architecture: amd64 Protocol Versions: [63 62] Network Id: 1 Go Version: go1.10.1 Operating System: linux GOPATH= GOROOT=/usr/lib/go-1.10
Keep the normal signing.
I was running a go-ethereum private network with 6 sealers.
Each sealer is run by:
directory=/home/poa command=/bin/bash -c 'geth --datadir sealer4/ --syncmode 'full' --port 30393 --rpc --rpcaddr 'localhost' --rpcport 8600 --rpcapi='net,web3,eth' --networkid 30 --gasprice '1' -unlock 'someaddress' --password sealer4/password.txt --mine'
The blockchain was running good for about 1-2 months.
Today I found that all the nodes were having issues. Each node was emitting the message "Signed recently, must wait for others"
I check out the logs and i found this message every 1 hour, no more information, the nodes where not mining:
Regenerated local transaction journal transactions=0 accounts=0 Regenerated local transaction journal transactions=0 accounts=0 Regenerated local transaction journal transactions=0 accounts=0 Regenerated local transaction journal transactions=0 accounts=0
Experimenting the same issue with 6 sealers, I restarted each node but now I'm getting stuck in
INFO [01-07|18:17:30.645] Etherbase automatically configured address=0x5Bc69DC4dba04b6955aC94BbdF129C3ce2d20D34 INFO [01-07|18:17:30.645] Commit new mining work number=488677 sealhash=a506ec…8cb403 uncles=0 txs=0 gas=0 fees=0 elapsed=133.76µs INFO [01-07|18:17:30.645] Signed recently, must wait for others
It's weird that some nodes are stuck on the 488677 and others are on 488676. This behaviour was reported on this issue https://github.com/ethereum/go-ethereum/issues/16406, same for the user @lyhbarry
Example: Signer 1
Note that there is no votes pending
So, right now, I shut down and restart each node, i have found that:
- Each node is paired with the others
- Each node is part of clique.getSigners()
Each node is waiting for another to sign...
INFO [01-07|18:41:56.134] Signed recently, must wait for others INFO [01-07|19:41:42.125] Regenerated local transaction journal transactions=0 accounts=0 INFO [01-07|18:41:56.134] Signed recently, must wait for others
So, the synchronization fails but I also can't just start signing again because each node is stuck waiting for the others, that means the network is useless?
The comment of @tudyzhb on that issue mention that:
Ref clique-seal of v1.8.11, I think there is no an effective mechanism to retry seal, when an in-turn/out-of-turn seal fail occur. So our dev network useless easily.
After this problem, i take a look at the logs, each signer has this error messages:
Synchronisation failed, dropping peer peer=7875a002affc775b err="retrieved hash chain is invalid"
INFO [01-02|16:42:10.902] Signed recently, must wait for others WARN [01-02|16:42:11.960] Synchronisation failed, dropping peer peer=7875a002affc775b err="retrieved hash chain is invalid" INFO [01-02|16:42:12.128] Imported new chain segment blocks=1 txs=0 mgas=0.000 elapsed=540.282µs mgasps=0.000 number=488116 hash=269920…afd3c7 cache=5.99kB INFO [01-02|16:42:12.129] Commit new mining work number=488117 sealhash=f7b00c…787d5c uncles=2 txs=0 gas=0 fees=0 elapsed=307.314µs INFO [01-02|16:42:20.929] Successfully sealed new block number=488117 sealhash=f7b00c…787d5c hash=f17438…93ffe3 elapsed=8.800s INFO [01-02|16:42:20.929] 🔨 mined potential block number=488117 hash=f17438…93ffe3 INFO [01-02|16:42:20.930] Commit new mining work number=488118 sealhash=b09b33…1526ba uncles=2 txs=0 gas=0 fees=0 elapsed=520.754µs INFO [01-02|16:42:20.930] Signed recently, must wait for others INFO [01-02|16:42:31.679] Imported new chain segment blocks=1 txs=0 mgas=0.000 elapsed=2.253ms mgasps=0.000 number=488118 hash=763a32…a579f5 cache=5.99kB INFO [01-02|16:42:31.680] 🔗 block reached canonical chain number=488111 hash=3d44dc…df0be5 INFO [01-02|16:42:31.680] Commit new mining work number=488119 sealhash=c8a5e7…db78a1 uncles=2 txs=0 gas=0 fees=0 elapsed=214.155µs INFO [01-02|16:42:31.680] Signed recently, must wait for others INFO [01-02|16:42:40.901] Imported new chain segment blocks=1 txs=0 mgas=0.000 elapsed=808.903µs mgasps=0.000 number=488119 hash=accc3f…44bc4c cache=5.99kB INFO [01-02|16:42:40.901] Commit new mining work number=488120 sealhash=f73978…c03fa7 uncles=2 txs=0 gas=0 fees=0 elapsed=275.72µs INFO [01-02|16:42:40.901] Signed recently, must wait for others WARN [01-02|16:42:41.961] Synchronisation failed, dropping peer peer=7875a002affc775b err="retrieved hash chain is invalid"
I also see some:
INFO [01-02|16:58:10.902] 😱 block lost number=488205 hash=1fb1c5…a41a42
This error about hash chain was just a warning, so the node keeps mining until the 2nd of January, then I saw this on each of the 6 nodes
I was looking that there are a lot of issues about this error, the most similar is the one I posted here but is unresolved.
Most of the issues' workarounds seems to be a restart, but in this case, the chain seems to be in a inconsistent state and the nodes are always waiting for the others.
- Any ideas?
- Any logs that I can provide? I only see the warnings of the error described and the block lost, but nothing when the node stoped to be mining
- Is this PR related? https://github.com/ethereum/go-ethereum/pull/18072
- Could it be related to the comment of @karalabe on this issue https://github.com/ethereum/go-ethereum/issues/16406?
- Will upgrading from 1.8.17 to 1.8.20 solve this?
- In my opinion, it seems like a race condition or something, since I have 2 chains, one running for 2 months, the other one for three months and this is the first time this error happens
This is the other related issue:
https://github.com/ethereum/go-ethereum/issues/16444 (Same issue but I don't have votes pending in my snapshot)