Bootnode Discovery with external IPs instead of local ones

Question

I would like to create a private testnet of mining nodes that external light clients can connect to. The caveat is that I'm doing so on Kubernetes (specifically via the excellent helm chart). So I can get the nodes to manually add themselves to each other via provisioned loadBalancer IP addresses, however the bootnode piece (which worked with just local networking), does not work.

In the logs I see messages along the lines of :

invalid neighbour (10.52.2.49) from [email protected]:30301: is self 

invalid neighbour (10.52.0.50) from [email protected]:30301: LAN address from WAN host 

invalid neighbour (10.52.2.48) from [email protected]:30301: LAN address from WAN host 

I suspect that the bootnode uses the actual local IPs of the nodes to advertise the nodes, which of course is not helpful for external connectivity (although I would think that this should still allow connectivity at least among the full nodes).

I used both the -v5 option on the bootnode as well as the --botnodesv5 and discv option on the nodes. Is there a way in the discovery protocol to specify the IP under which a node should be reachable? I would have assumed that's what a node's enode is for, and all my enodes correctly contain the external IP addresses and not the local ones. (via the --nat=extip:IP option)

Thank you!

Answer 1

So here is how I solved my issue. I host my kubernetes cluster in google cloud, but the same concepts should apply to any provider.

First I created the clusters within a predefined VPC with subnets. Be mindful of secondary ranges for the IP. The gcloud script below illustrates:

sh "gcloud compute networks create ${CLUSTER_NAME}-network --subnet-mode custom"
sh "gcloud compute firewall-rules create ${CLUSTER_NAME}-allowall-firewall --network ${CLUSTER_NAME}-network --allow all --source-ranges 10.0.0.0/8"

sh "gcloud compute networks subnets create ${CLUSTER_NAME}-subnet \
    --project ${PROJ_NAME} \
    --network ${CLUSTER_NAME}-network \
    --region ${ZONE} \
    --range ${RANGE_NODES} \
    --secondary-range range-pods=${RANGE_PODS},range-services=${RANGE_SERVICES}"

sh "gcloud container clusters create ${CLUSTER_NAME} \
    --enable-ip-alias \
    --network ${CLUSTER_NAME}-network \
    --subnetwork ${CLUSTER_NAME}-subnet \
    --cluster-secondary-range-name range-pods \
    --services-secondary-range-name range-services \
    --machine-type=${MACHINE_TYPE} \
    --image-type=COS \
    --num-nodes=${MIN_NODES} \
    --enable-autorepair \
    --enable-autoscaling \
    --enable-autoupgrade \
    --min-nodes=${MIN_NODES} \
    --max-nodes=${MAX_NODES} \
    --node-locations=${NODE_LOCATIONS} \

After that I peer the VPC networks, so they can see each other and live in the same network space. When defining the VPC IP ranges be mindful that the IP addresses from the different clusters don't overlap.

Finaly you can create a deployments and services similar to the ones below. The two golden pieces of information that made this work were:

1 - On google cloud to create an internal load balancer service you need to annotate the service yaml with: cloud.google.com/load-balancer-type: Internal

2 - Set externalTrafficPolicy: Local spec on the service yaml. This prevents the proxy from hiding the source IP.

Bootnode Deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: testnet-bootnode-1-ethereum-network-node
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: testnet-bootnode-1
      app.kubernetes.io/name: ethereum-network-node
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: testnet-bootnode-1
        app.kubernetes.io/name: ethereum-network-node
    spec:
      containers:
      - args:
        - 'bootnode -nodekey /root/bootnode.key -verbosity 9 --addr :30301 '
        command:
        - /bin/sh
        - -c
        image: myGethImage
        imagePullPolicy: Always
        name: ethereum-network-node
        ports:
        - containerPort: 30301
          name: discovery-udp
          protocol: UDP

Bootnode Service

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/load-balancer-type: Internal
  name: testnet-bootnode-1-ethereum-network-node-p2p
spec:
  externalTrafficPolicy: Local
  healthCheckNodePort: 32000
  ports:
  - name: discovery-udp
    nodePort: 30399
    port: 30301
    protocol: UDP
    targetPort: 30301
  selector:
    app.kubernetes.io/instance: testnet-bootnode-1
    app.kubernetes.io/name: ethereum-network-node
  type: LoadBalancer

In theory you could do the same without the VPC, but the problem is that the external load balancer IP is different from the outgoing IP of the nodes, this way. the nodes ping the bootnode with an IP that is different from the service, so the pong does not reach the service back. Perhaps NGINX could help with that. But I haven't explored that path.

The service for the validator nodes are similar (except it should be TCP for P2P)