Mining causes nodes to stop syncing on Besu

Question

I am trying to test my private network by running multiple nodes at one time. I am using a config file that lists the boot node as well as the same genesis file. The boot node is on a linux server, the peer node is on my Windows desktop.

With the boot node running, if I launch the peer node without mining, it syncs to the network just fine and continues importing blocks. However, once synced, if I stop the peer node, and restart it as a mining node, the peer is immediately out of sync with the boot node, and it does not attempt to sync. Both nodes are then mining the same block, or one gets out of sync, and the boot node usually starts throwing exceptions.

If I delete the database on the peer node, and then start is as a mining node from the beginning, it never attempts to sync to the boot node and just starts mining at block #1 as if it's the only node.

The issue seems to be related to the mining process, because that is what triggers the problems. I don't understand why adding a peer mining node is causing the node to not sync with other existing nodes. Both nodes use the same config and genesis file.

Config file:

#TOML config file for starting a node

#Network
bootnodes=["enode://251bb2dd27c06f92dca73dcd56ccd3f6c8044dd542597eacc727c70217e7e34e2c04df5c8d57a4200c9fb8336bd13e9caa28761997b3ff48114fe0bc81e80ee3@IP:30303"]

p2p-host="0.0.0.0"
host-allowlist=["https://IP"]

rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-cors-origins=["all"]
rpc-ws-enabled=true

#Other settings
min-gas-price=0
revert-reason-enabled=true

Genesis:

{
  "config": {
      "byzantiumBlock":0,
      "constantinopleBlock":0,
      "constantinopleFixBlock":0,
      "ethash": {
        "fixeddifficulty": 1000
      },
       "chainID": 2021
   },
  "nonce": "0x0",
  "gasLimit": "0x1fffffffffffff",
  "difficulty": "0x10000"
}

Answer 1

The error was timestamp related. It turns out that the server the private blockchain is hosted on was not properly syncing to the NIST atomic clock, since it is behind a firewall and not 100% internet connected. The local Windows machine is synced to the NIST atomic clock, and thus the timestamps were off. The server was behind NIST by about 31 seconds at the time.

My colleague ran the same chain on two fully internet connected machines and there were no issues.

TL;dr; make sure all machines you're running the chain on are synced to the NIST atomic clock.