I have a private blockchain with 6 nodes, and a total of three miners. When I initialize the blockchain everything looks great and the nodes find each other starting with the bootnodes.
After about 12 hours the connections drop to 3 or 4 peers, and by 48 hours each node has zero peers. I can't figure out why.
The times are synced.
Each one uses the exact same chain specification file (verified by md5sum)
The chain spec has a unique networkid.
The nodes do find each other in the beginning. And they exchange data with each other.
Most of the nodes are on the same local area network.
All machines have 30303 and 30304/udp open and the bootnodes also have 30301 open.
I can manually add nodes back using
api.parity.addReservedPeer('enode://....') but even when I do, the node that I add is the only one that shows up. They do not discover each other even if I add one node in common.
What could I be doing wrong?