2

I would like to create a private testnet of mining nodes that external light clients can connect to. The caveat is that I'm doing so on Kubernetes (specifically via the excellent helm chart). So I can get the nodes to manually add themselves to each other via provisioned loadBalancer IP addresses, however the bootnode piece (which worked with just local networking), does not work.

In the logs I see messages along the lines of :

invalid neighbour (10.52.2.49) from 1182aa8c9dbb96cd@35.230.53.216:30301: is self 

invalid neighbour (10.52.0.50) from 1182aa8c9dbb96cd@35.230.53.216:30301: LAN address from WAN host 

invalid neighbour (10.52.2.48) from 1182aa8c9dbb96cd@35.230.53.216:30301: LAN address from WAN host 

I suspect that the bootnode uses the actual local IPs of the nodes to advertise the nodes, which of course is not helpful for external connectivity (although I would think that this should still allow connectivity at least among the full nodes).

I used both the -v5 option on the bootnode as well as the --botnodesv5 and discv option on the nodes. Is there a way in the discovery protocol to specify the IP under which a node should be reachable? I would have assumed that's what a node's enode is for, and all my enodes correctly contain the external IP addresses and not the local ones. (via the --nat=extip:IP option)

Thank you!

  • I am dealing exactly with this issue right now. The problem is not with the bootnode itself (although it could use a protocol that allows sending extra information like FDQN or the IP itself without relying on the information sent by the network). The problem is that whatever reaches the bootnode passes through kube proxy first, so the source IP of the UDP packet will not contain the original IP address. I am still messing with the extra levels of networking that Kubernetes have, if I come up with a solution I will net you know. Also, try setting "externalTrafficPolicy: Local" on the service – Fábio Jun 20 at 11:12
  • As it so happens, I'm also revisiting this problem again. Feel free to reach out to me via Twitter or Keybase if you want to collaborate – nambrot Jun 21 at 12:36
  • Sorry I don't do Twitter and never heard of Keybase haha. You can reach me in Linkedin – Fábio Jun 28 at 14:25
0

So here is how I solved my issue. I host my kubernetes cluster in google cloud, but the same concepts should apply to any provider.

First I created the clusters within a predefined VPC with subnets. Be mindful of secondary ranges for the IP. The gcloud script below illustrates:

sh "gcloud compute networks create ${CLUSTER_NAME}-network --subnet-mode custom"
sh "gcloud compute firewall-rules create ${CLUSTER_NAME}-allowall-firewall --network ${CLUSTER_NAME}-network --allow all --source-ranges 10.0.0.0/8"

sh "gcloud compute networks subnets create ${CLUSTER_NAME}-subnet \
    --project ${PROJ_NAME} \
    --network ${CLUSTER_NAME}-network \
    --region ${ZONE} \
    --range ${RANGE_NODES} \
    --secondary-range range-pods=${RANGE_PODS},range-services=${RANGE_SERVICES}"

sh "gcloud container clusters create ${CLUSTER_NAME} \
    --enable-ip-alias \
    --network ${CLUSTER_NAME}-network \
    --subnetwork ${CLUSTER_NAME}-subnet \
    --cluster-secondary-range-name range-pods \
    --services-secondary-range-name range-services \
    --machine-type=${MACHINE_TYPE} \
    --image-type=COS \
    --num-nodes=${MIN_NODES} \
    --enable-autorepair \
    --enable-autoscaling \
    --enable-autoupgrade \
    --min-nodes=${MIN_NODES} \
    --max-nodes=${MAX_NODES} \
    --node-locations=${NODE_LOCATIONS} \

After that I peer the VPC networks, so they can see each other and live in the same network space. When defining the VPC IP ranges be mindful that the IP addresses from the different clusters don't overlap.

Finaly you can create a deployments and services similar to the ones below. The two golden pieces of information that made this work were:

1 - On google cloud to create an internal load balancer service you need to annotate the service yaml with: cloud.google.com/load-balancer-type: Internal

2 - Set externalTrafficPolicy: Local spec on the service yaml. This prevents the proxy from hiding the source IP.

Bootnode Deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: testnet-bootnode-1-ethereum-network-node
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: testnet-bootnode-1
      app.kubernetes.io/name: ethereum-network-node
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: testnet-bootnode-1
        app.kubernetes.io/name: ethereum-network-node
    spec:
      containers:
      - args:
        - 'bootnode -nodekey /root/bootnode.key -verbosity 9 --addr :30301 '
        command:
        - /bin/sh
        - -c
        image: myGethImage
        imagePullPolicy: Always
        name: ethereum-network-node
        ports:
        - containerPort: 30301
          name: discovery-udp
          protocol: UDP

Bootnode Service

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/load-balancer-type: Internal
  name: testnet-bootnode-1-ethereum-network-node-p2p
spec:
  externalTrafficPolicy: Local
  healthCheckNodePort: 32000
  ports:
  - name: discovery-udp
    nodePort: 30399
    port: 30301
    protocol: UDP
    targetPort: 30301
  selector:
    app.kubernetes.io/instance: testnet-bootnode-1
    app.kubernetes.io/name: ethereum-network-node
  type: LoadBalancer

In theory you could do the same without the VPC, but the problem is that the external load balancer IP is different from the outgoing IP of the nodes, this way. the nodes ping the bootnode with an IP that is different from the service, so the pong does not reach the service back. Perhaps NGINX could help with that. But I haven't explored that path.

The service for the validator nodes are similar (except it should be TCP for P2P)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.