Losing connection between nodes on private network

Question

I have 3 nodes running on a private network.

After asking this question, I've manually added node1 as peer to node2 and node3, so my network looks like this:

                 _______                    
     ---------> | node2 |
    /           |_______|
   v
 _______
| node1 |
|_______|
   ^
    \            _______
     ---------> | node3 |
                |_______|

I've left the nodes running through last night, but around 10 PM (BRST), node2 and node3 stopped listening to node1 (admin.peers was empty), while node1 was still connected to both of them (admin.peers contained 2 items), but not getting any interaction.

Is it possible that this is a problem with Ethereum protocol or would it be something else?

---

Edit:

I'm running the nodes in the same physical machine, but in different VMs. Those VMs have Ubuntu 12.04 installed.

The VMs are running over a CentOS with VMWare vCenter.

Answer 1

Summary

I would have expected that the bootnodes parameters would allow node 2 and node 3 to find node 1, node 2 and node 3 and then for all the nodes to retain their peer connections.

From Henrique Barcelos's issue, it seems that this configuration is not stable in a private network running the nodes in a VM.

The alternative of using static-nodes.json does not seem to provide a stable connection.

The next configuration to try is using trusted-nodes.json and specifying the --maxpeers parameter

Below are the steps we are taking to solve the problem.

---

An interesting problem.

1. Probability wise, I would say it is highly unlikely that this is a problem with the Ethereum protocol as it is robust enough for the Internet.

2. You have stated that your time is synced to network time. This seems to be a problem regularly encountered and that is why I asked the question above.

3. Let's look at your configuration.

4. Question. Are you running any other Ethereum mainnet nodes within your private network. The reason I am asking this is that there may be a clash with the network port 30303, referring to the first link in your question.

5. From the first link in your question, you are starting up your geth instances on the different nodes with the following command:

   geth --verbosity 4 --autodag --nat any --genesis /opt/blockchain/genesis.json
    --datadir /opt/blockchain/data --networkid 4828 --port 30303 --rpc
    --rpcaddr 10.48.247.25 --rpcport 8545 --rpcapi db,eth,net,web3,admin
    --rpccorsdomain '*' --fast --mine --ipcdisable
   

   Node 1 has been set as your bootnode. Nodes 2 and 3 have the file /opt/blockchain/data/static-nodes.json with the following information:

   ["enode://{node1publickey}@{node1ip}:30303"]
   

6. The information in /opt/blockchain/data/static-nodes.json in 5. above matches the startup information in the node1 geth instance that is displayed using the admin.nodeInfo command in the node1 geth instance.

   ...
   enode: "enode://{node1publickey}@{node1ip}:30303",
   ...
   

   Is this correct?

7. Looking at your geth startup parameters now:
   a. You could try removing the --nat any parameter
   b. Your --networkid is unique from the mainnet "1"
   c. Your --port 30303 should not be a problem unless you are running other mainnet Ethereum nodes within your network, and even so should not be a problem as it will be on a different IP
   d. Your --rpcaddr should be the IP address of each machine. Node 1 should have the node 1 IP address. Node 2 should have the node 2 IP address and node 3 should have the node 3 IP address.
   e. Your --rpcport, --rpcapi and --rpccorsdomain should not affect your network connectivity.
   f. You should not need the --fast parameter as your blockchain should be small anyway, being an internal testnet blockchain.
   g. Your --mine and --ipcdisable parameters should not affect your connectivity.
   
8. What I would try next is to remove the --bootnode parameter, and try the /opt/blockchain/data/static-nodes.json method in 5. above on nodes 2 and 3. Run this for a while and let's hope you don't have the connectivity dropouts. We can cross this off the list if you are still having the connectivity dropouts.
   

Below added 12/04/2016:

9. Henrique Barcelos has been using the static-nodes.json method for the peers to find each other, but the peer connections dropped out. I've suggesting trying trusted-nodes.json as these specify trusted peer nodes to connect to, and would not be blocked by the --maxpeers connection limits.
10. What I have found from my testing is that the static-nodes.json option would not connect unless I add a non-zero --maxpeers parameter, although --maxpeers is meant to default to 25.