At any given time, how many nodes is my node connected to? As the number of nodes in the network increases, will communication between them be faster?

Question

I have been reading through the DEVp2p documentation and cannot seem to wrap my head around some aspects of the network. Specifically, how many nodes is my node connected to? Does this number change as I perform a transaction? As more nodes become available, does my node connect to the most convenient ones in terms of speed? What other resources can I use to learn more about the underlying infrastructure?

I have been reading over the following pages but cannot seem to wrap my head around it: 1 2

Answer 1

You can see the connected peers by typing admin.peers in the Geth console. The maximum number of peers is set using the -maxpeers n flag in Geth. There is a discovery process based on Kademlia for finding nodes, then a handshaking process by which they determine which devp2p protocols they support (Eth, Bzz, Shh). The P2P layer monitors each node's quality of service . It calculates statistics, drops or even bans bad nodes and try to keep hold of good ones i.e. those with high uptime and which respond quickly to ping messages. The number of peers will thus vary as nodes go offline or service quality changes. So the best thing to do is to stay online in which case the quality of the peers you connect to will gradually improve and also your quality rating will improve.

---

In the second part of your question you ask about the wire protocol. I don't fully understand it myself however I looked though the documentation and source code of various clients. I thus cannot really explain the whole p2p works but here is how I believe the Dev p2p discovery protocol works.

Dev p2p discovery protocol

https://github.com/ethereum/go-ethereum/wiki/Peer-to-Peer

The RLPx Protocol which is used to create the P2P network and provides the basis for protocols like ethereum sync('eth'), whisper("shh"), and swarm("bzz").

So the first thing we need to do is to form a P2P network and find our peers and this is where the discovery protocol comes in. The docs say

RLPx utilizes Kademlia-like routing which has been repurposed as a p2p neighbour discovery protocol. RLPx discovery uses 512-bit public keys as node ids and sha3(node-id) for xor metric. DHT features are not implemented.

The Kademlia like protocol where each peer maintains a routing table that contains the complete set of all the nodes nearest to it, gets exponentially sparser over increasing distance. From a messaging point of view the protocol consists of the following UDP-based RPC functions:

• Ping -Probes a node to see if it is online. The receiver should reply with a Pong packet. If a node receives a ping, it will, after answering with a pong, send it's own ping to the first node.
• Pong - Replies to a ping
• Find_node -Find Node packets are sent to locate nodes close to a given target ID. The receiver should reply with a Neighbors packet containing the k nodes closest to target that it knows about.
• Neighbours -Is the reply to Find Node. It contains up to k nodes that the sender knows which are closest to the requested Target.

Decoding a UDP packet

The peer listens for UDP datagrams on port 30303 by default.

The packets are framed as follows:

|-------|-hash----| signature | packet-type| packet-data   |
|-------|---------| ----------| ---------- |               |
|length | 32 bytes| 65 bytes  | single byte| rest of packet|
|offset | 0       | 32        | 97         | 98            |

Check message integrity and authenticity

Defining message as a byte array we can extract these components as follows: first thing to do is to check at the hash message[:32] equals the SHA3-256 hash (also known as Keccak256) of the rest of the message (message[32:]). If it is not the message is corrupt and we drop the packet.

In python this is:

mdc_hash = message[:32] //bytes zero to 32
the_rest = message[32:] //bytes 32 to length-1
assert mdc == sha3_256(the_rest).digest()

The next thing to check is the signature. The message is signed using the Elliptic Curve Digital Signature Algorithm (ECDSA) specifically secp256k1 where the public key is the node id of the sender. Given the signature (65-byte compact ECDSA signature containing the recovery id as the last element.) and the message that was signed (32 byte sha3 hash of the message), and the knowledge of the curve, it is possible to recover the public key corresponding to the private key used to sign the message:

Note: The python client imports a library from bitcoin for ECDSA. The Go client wraps libsecp256k1 While the java client adapts code from the bitcoinj project.

signature = message[32:97]
signed_data = sha3_256(message[97:])
remote_pubkey = crypto.ecdsa_recover(signed_data, signature)

If we fail to recover a key we drop the message

assert len(remote_pubkey) == 512 / 8

Now we know the sender's Node ID but we still need to verify the signed message and throw an error if message cannot be authenticated.

if not crypto.verify(remote_pubkey, signature, signed_data):
    raise InvalidSignature()

Figure out what kind of message it is and decode it to the appropriate message structure

The message type is given by a single byte message[97] and we simply look this up.

|value | 1   | 2   | 3         | 4         |
|--| --- | --- | ---| ---|
|type | ping| pong| find_node | neighbours|

This can be done with a simple dictionary or enum based switch statement and if the value is not in the range 1-4 the message is dropped.

** Decode it to the appropriate message structure**

The packet data is a Recursive Linear Prefix (RLP) encoded list. Which is an Ethereum specific data structure for serializing arbitrarily nested arrays of binary data e.g

[
  "some string",
  "some bytes",
  ["element1","element2"],
  [["a","b"],["c"]],
]

So first lets understand how encoding works and the specs give us a the following python code as an example.

def rlp_encode(input):
if isinstance(input,str):
    if len(input) == 1 and ord(input) < 128: return input
    else: return encode_length(len(input),128) + input
elif isinstance(input,list):
    output = ''
    for item in input: output += rlp_encode(item)
    return encode_length(len(output),192) + output

def encode_length(L,offset):
if L < 56:
     return chr(L + offset)
elif L < 256**8:
     BL = to_binary(L)
     return chr(len(BL) + offset + 55) + BL
else:
     raise Exception("input too long")

def to_binary(x):
return '' if x == 0 else to_binary(int(x / 256)) + chr(x % 256)

Thus the key point for decoding is to look at the range of the first byte and depending on that you would decode the rest of the payload differently.

|range| meaning | encoding
|-----|----- |
|0x00, 0x7f| Single byte as its self | single byte|
|0x80, 0xb7|string 0-55 bytes long | 0x80 + length of string | rest of string|
|0xb8, 0xbf |string more than 55 bytes long| 0xb7 plus how many bytes are required to represent the length | length of the string as byte array | the string |
|0xc0, 0xf7 | payload e.g. list or list of lists the combined encoded length of which is 0-55 bytes long| length of total rlp encoded payload | RLP encoded payload|
|0xf8, 0xff | payload e.g. list or list of lists the combined encoded length of which is more than 55 bytes long| 0xf7 plus the length in bytes required to represent the length of the payload in binary form | length of the payload as byte array | RLP encoded payload |

** You got a ping packet **

The ping messages have the following structure so you need to read the bytes data into the correct data types.

[
 uint Version, // big-endian encoded 32 bit integer protocol version reject if doesn't match own
 Endpoint to,
  Endpoint from,
  uint32 expirationTimestamp //big-endian encoded 32 bit integer reject if time() > timestamp to prevent replay attacks.
}

For brevity and following the Go code we define the Endpoint type as the following structure:

[
  bytes ip, // big-endian encoded or  4-byte (32-bit)  or 16-byte (128-bit) address (size determines ipv4 vs ipv6)
  uint16 udp-port, //big-endian encoded 16 bit integer
  uint16 tcp-port //big-endian encoded 16 bit integer
]

In the original Kademlia protocol the receiver of the ping message would update the bucket corresponding to the sender. However to protect against ip address spoofing the Ethereum protocol does not. Instead the receiver of a ping message should just reply with a pong message then send its own ping later:

[
  Byte[] replyToken, // This contains the hash of the ping packet.
  Endpoint to, //The endpoint that sent the ping message
  uint32 expirationTimestamp
]

So there are two things to note here:

• The replyToken is used by the receiver of the pong message to link it to their ping message and is simply the mdc_hash of the packet checked earlier.
• The to field should mirror the UDP envelope address of the ping packet not the from endpoint stated in the ping message so as to provide a way to discover the external address (after NAT).

** You got a pong packet **

On receipt of a pong packet the nodechecks if it was solicited i.e That it sent a ping to that node, is waiting for a pong. It then updates the bucket for that node

P2P network

On first joining the network each node generates an ECDSA key pair which is saved and used in subsequent connections enode://<hex node id> for example. The private key will be used to sign messages and the 512 bit public key used to identify the node. It will also know the IP address, UDP port, and node ID of some bootstrap nodes.

Each node maintains a routing table which holds a list of known peers known as contacts.

Routing tables consist of a list(bucket) for each bit of the node ID.

The peer table consists of rows, initially only one, at most 255 (typically much less). Each row contains at most k peers (data structures containing information about said peer such as their peer address, network address, a timestamp, signature by the peer and possibly various other meta-data), where k is a parameter (not necessarily global) with typical values betwen 5 and 20.

Row numbering starts with 0. Each row number i contains peers whose address matches the first i bits of this node's address. The i+1st bit of the address must differ from this nodes address in all rows except the last one.

Every list corresponds to a specific distance from the node thus the n^th bucket differs in the n^th bit of the node ID from the nodeID

In Kademlia the distance is defined and bitwise XOR dist(pubk-A,pubk-e) = pubkA ^ pubkB but because the ECDSA key are not uniformly distributed Devp2p uses dist(pubkA, pubkB) = sha(pubkA) ^ sha(pubkB)

A 4-bit example with a bucket size of 2 is shown below:

The right most bucket (the 0^th bucket) contains the node itself. The left most bucket covers the space nodes which differ in the most significant bit. Notice that the address space potentially covered by the buckets expands exponentially with distance.

|Bucket|4 |3 |2 |1|
|-|- |- |- |-|
|Number of notes it could include if bucket size was unlimited  |8 |4 |2 |1|
|Number of notes it can include when bucket is full |2 |2 |2 |1|

However the buckets have a fixed size (in this example 2) and thus it is clear that the node's routing table has the complete set of all the nodes nearest to it, gets exponentially sparser over increasing distance. This feature is key to Kademlia Algorithm.

bootstrap

1. Add bootstrap nodes. To the table.
   1. bond i.e Ping the remote side and wait for a pong. Give them a chance to ping us. If they already know us they will not send a ping back.
   2. Insert the triple id, IP, discover port, tcp port into the database
   3. Add the given node its corresponding bucket If the bucket has space available, adding the node succeeds immediately and the node is added to the tail. Otherwise, the node is added if the least recently active node i the bucket does not respond to a ping packet.
   4. If the old node does respond keep it (moving it to the tail?) and don't add the new node.
2. Do a self lookup to fill up the buckets.

Lookup

Lookup performs a network search for nodes close to the given target. It approaches the target by querying nodes that are closer to it on each iteration. The given target does not need to be an actual node identifier.

1. Determine the n nodes in its own table that are closest to the given id
2. Pick 3 Kademlia concurrency factor nodes from this list
3. Send findnode message to these remote nodes and wait for k neighbors message.
4. For each of the new nodes bond i.e Ping the remote side and wait for a pong. Then add to correct bucket.
5. Add these new nodes to the list n closest nodes to query.
6. When there are no more nodes to ask stop.

Processing a Find node message.

If the peer is unknown we don't process the packet. We find the n closet nodes Send them as a number of neighbours messages.

neighbors struct {
Nodes      []rpcNode
Expiration uint64
}