Geth signing stops after a period of time

Question

I have 5 clients setup (1 boot, 1 geth, 3 signers using Clique) - Most of the time they work no problem and mining starts, sometimes mining just stops with no warning nor issue/error message.

Another time I noticed that after coming to consensus on an additional external signer that mining will also stop - I have tried to find information on the order with which signers take turns and what can stop mining from happening, from looking at the code it seems that signers are chosen more so by timing than randomly but can a signer who misses their "turn" for whatever reason cripple or stop the other nodes from signing/validating blocks ? I would hazard a guess that this is not the case but I cant seem to tract down the issue and its is rendering my dev network useless. I know I'm not giving much to go on but there is no errors like usual such "as bad propagated block" all that happens is siging stops and everyone is on "signed recently, waiting to sign again"

for what its worth im running my signers with this

    ```
    geth --datadir ~/test-net/data --mine --cache=1024 --syncmode 'full' --bootnodes="enode://enodeurlgoeshere@ipgoeshere:port" --networkid 47592 --rpc --rpcapi admin,db,eth,debug,miner,net,txpool,personal,web3 --rpcport "8545" --rpcaddr "0.0.0.0" --rpccorsdomain "*" --unlock 914be90b2a0dd6a5b0789c2cee5836dd1f1c030 --password <(echo "password") console --nat "extip:pubIPgoeshere"
    ```

and this is the output from signing when it stops

    ```
    INFO [03-28|18:15:34] Commit new mining work                   number=21271 txs=0 uncles=2 elapsed=315.821µs
    INFO [03-28|18:15:34] Successfully sealed new block            number=21271 hash=0e4d19…b0854b
    INFO [03-28|18:15:34] 🔨 mined potential block                  number=21271 hash=0e4d19…b0854b
    INFO [03-28|18:15:34] Commit new mining work                   number=21272 txs=0 uncles=2 elapsed=434.638µs
    INFO [03-28|18:15:34] Signed recently, must wait for others 
    INFO [03-28|18:23:07] Regenerated local transaction journal    transactions=0 accounts=0
    INFO [03-28|19:23:07] Regenerated local transaction journal    transactions=0 accounts=0
    INFO [03-28|20:23:07] Regenerated local transaction journal    transactions=0 accounts=0
    INFO [03-28|21:23:07] Regenerated local transaction journal    transactions=0 accounts=0
    ```

Answer 1

When you have 3 signers, at least 2 have to be online and have the eth.coinbase account unlocked to be able to continue sealing new blocks. Otherwise the sealing process stops with "Signed recently, must wait for others".

If you're sure that your sealers are online, then the only logical explaination is that your account unlock has ran into a timeout (default in geth is 5min) and the sealer account is locked again. Please check that. You can unlock for unlimted time with personal.unlockAccount(eth.coinbase, "pwd", 0)

See also "How to unlock accounts programmatically after the node has started?"