I'm running a custom consensus engine and launching a network for it. The nodes I have are making handshakes with other networks, but then not peering since they are not compatible. But, the handshake seems to never end. Technically, the "handshake" status below, is defined by returning nil from PeerInfo, meaning the peer is not in the handler peerSet, and does not mean that a handshake is actually ongoing, I assume, as those should time out after 5 seconds as defined in eth/protocols/eth/handshake.go. So, for some reason, the peer still remains even though it never succeeded to handshake nor form a proper connection. Then, all 50 peer slots fill up with these, preventing my real nodes from peering after that. Anyone know what's going on?
Example peer, there are 50 of these once the limit has filled up:
}, {
caps: ["eth/66", "snap/1"],
enode: "enode://07edbd043c0ab74d81b01d9b8990f360b0e1fbd30d033af14b7347bc8655e4f5596070e00bb3693150afb7705de9d2f0113b28b3fbc88ed5b1fddd8486570381@121.144.172.66:56626",
id: "fbc4e4b929ce17fc76c258fff826dfd51f5ffd52379ce3eec721d220cf6abcce",
name: "Geth/v1.10.18-stable-de23cf91/linux-amd64/go1.18.1",
network: {
inbound: true,
localAddress: "74.207.235.82:30303",
remoteAddress: "121.144.172.66:56626",
static: false,
trusted: false
},
protocols: {
eth: "handshake",
snap: "handshake"
}
}]
Some more context. The peers exist in the run() method go routine in the Server class in p2p/server.go, within the local map peers = make(map[enode.ID]*Peer), and admin_peers RPC call reads from this with the PeersInfo() method in Server class (via Server.Peers()). In PeersInfo(), it looks for the peer within *eth.PeerSet (via eth/handler_eth.go or eth/handler_snap.go) and it cannot find it there, thus labelling it as being in "handshake". To be added to peers in the run(), it looks like it has to pass srv.checkpointAddPeer ("At this point the connection is past the protocol handshake") and srv.addPeerChecks(peers, inboundCount, c) since that is when it is added to peers.
It looks like all the 50 peers are over eth/66, except for the one peer I'm actually supposed to be connected to (and it is over eth/66, eth/67 and eth/68, and not in a "handshake"), see admin_peers output here for my RPC node and here for my validator node. Perhaps the trouble is with eth/66 nodes, and any peer that tried to connect over eth/67 or eth/68 has successfully closed after it was found to be on different chain ID etc.
I'd be very interested in solving this problem since it is preventing peering within my network...
Edit: starting Geth with --eth.protocols 67 68 seems to prevent any eth/66 peers from getting stuck in "handshake", so it is a way for me to get around the problem. Still does not solve it or explain what caused it. Also, with this fix, I'm now able to also gracefully shut down Geth, so this was what was getting stuck in handler.go (see question here). And to be clear with that, I was unable to remove the peers with admin.removePeer, they were simply stuck, preventing peering, preventing shut down, etc. Still interested in understanding what the problem is with eth/66 peers.
