I am having an private ethereum network consisting of 49 geth nodes running in docker containers. All geth nodes are miners. My topology is following: it's a tree with one root and having 48 leaves.
I start geth by executing:
geth --rpc --rpcaddr 0.0.0.0 --rpcport 8545 --datadir /data --syncmode "fast" --cache "1024" --rpcapi="db,eth,net,web3,personal,web3" --txpool.pricelimit "0" --miner.gasprice "0" --maxpeers "80" --nodiscover --verbosity 4.
The problem I face is that, after some time none of the nodes is synced with the rest of the network. I'm assuming this because:
docker exec -i nodeX_bc geth --exec "eth.blockNumber" attach ipc://data/geth.ipc- returns a different block number for each node (the numbers differ by at least 50).
docker exec -i nodeX_bc geth --exec "eth.syncing" attach ipc://data/geth.ipc- returns false
docker exec -i nodeX_bc geth --exec "eth.getBlock(XXX).hash" attach ipc://data/geth.ipc- differs if XXX > a value
- The logs of one of the leaves (let's call it node1_bc) says:
INFO [01-03|20:45:28.538] Mining too far in the future wait=2s
DEBUG[01-03|20:45:28.550] Peer discarded announcement peer=503b78bda6977425 number=767 hash=3879d7…02e6c0 distance=-191
DEBUG[01-03|20:45:28.550] Peer discarded announcement peer=503b78bda6977425 number=768 hash=e2d843…ffb5c9 distance=-190
DEBUG[01-03|20:45:28.564] Peer discarded announcement peer=503b78bda6977425 number=769 hash=5b4c52…8d7454 distance=-189
DEBUG[01-03|20:45:28.598] Peer discarded announcement peer=503b78bda6977425 number=767 hash=bdcddc…437d07 distance=-191
DEBUG[01-03|20:45:28.598] Peer discarded announcement peer=503b78bda6977425 number=768 hash=c07110…1d6676 distance=-190
DEBUG[01-03|20:45:28.599] Peer discarded announcement peer=503b78bda6977425 number=767 hash=982f07…15b9e8 distance=-191
DEBUG[01-03|20:45:28.599] Peer discarded announcement peer=503b78bda6977425 number=767 hash=f7704d…065e59 distance=-191
Which as I found in the source code here, means that I am too far 'in front of' the topologies' root (id = 503b78bda6977425), which is the node I want to synchronize at first.
The logs in the root geth node says:
DEBUG[01-03|20:45:56.990] Discarded propagated block, exceeded allowance peer=node1_bc-id number=986 hash=b93deb…05dc20 limit=64
While I assume that I simply have too much miners and each couple of seconds there is a new block generated in each node, and there is no enough time to sync everything, thus I run into the issues. But I am not sure how to interpret the 'peer discarded announcement' message. Does it mean that the node stopped syncing with the rest of the network and generates all blocks local only? Does this mean that the node can't be synced in the future?
Hardware I am using:
- root node: 4 vCPU, 15GB RAM, 60GB HDD
- leaves node: 4 vCPU, 8GB RAM, 5GB SSD
- Experienced the same behavior when using "full" synchronization mode.
An interesting finding:
- I tried to find the most recent transactions in every geth node by using
docker exec -i nodeX_bc geth --exec "eth.getTransaction(XXX)" attach ipc://data/geth.ipc, and I found them in almost all nodes (even if the blockNumber of the nodes differed by more than 500). This should imply that even if the nodes don't have the same block number, they might contain the same transactions, but these might be included in another block.