How to have a client be agnostic in cluster nodes?

Question

I have been wondering for some time how to handle the situation where the client application of Ethereum blockchain can remain connected when the nodes he connects to via API is down for maintenance.

Say I have 2 or 3 local nodes in my infrastructure and I need to maintain one of them (upgrade to new version, reboot the host ...). If my client app is connected to it, its connection will be lost.

Answer 1

Just because you're looking for different ideas ...

I'd probably approach this by separating concerns and enlisting the IT department to provide an always on "cluster" of ethereum nodes with a virtual IP address that isn't supposed to fail. Conveniently, nodes would tend to stay in sync provided geth is running in all places at all times.

From the apps perspective, you would concentrate on recovering from the loss of callbacks "in flight" when a failover (to another physical node) occurs. It might not be too difficult to habitually retry requests so eventual app recovery is assured immediately after the Ethereum cluster resumes.

The watchers would be interesting. It might not be especially easy to detect a failover if the cluster runs smoothly enough (IT's goal). On the app side, I would look for something detectable that's different between the physical nodes, and re-dispatch the watchers when a failover is detected.

• e-node would be different.

You could have transactional callbacks that persistently retry after timeouts instead of failing altogether. You get stalled requests that eventually work.

You could have periodic checks of e-node ID and wrap your watcher dispatches inside a condition that runs whenever a change of node is detected. So basically, whenever the "node" is different than before, we assume that the watchers need to be dispatched.

To avoid redundant watchers on any node, inform the IT department that when a node is "relieved/demoted" (for whatever reason), restart Geth (or the OS) before the node is "qualified" to take over again in the future. They'll understand the idea of always starting the service with a known clean state.

Indeed, cluster managers can also check the health of the nodes and confirm sync is complete before returning the node to the "pool" of healthy nodes that are ready to serve.

And, you can validate the app design without setting up the whole cluster in every detail. You'd be leaving the IT team with considerable lattitude about how they want to approach it. Your test case would be two nodes, one serving and one on standby. Kill the live node. Reconfigure the IP address on the standby node to "take over". Confirm there is no loss of service, no loss of "in flight" transactions and no confusion about the watchers.

Hope it helps.

Answer 2

So I have designed a small workaround by creating a MultiHttpProvider that acts as a HttpProvider and can replace it during the web3 initialization.

I contains an array of alternate hosts and creates as many HttpProvider instance. Each time it detects a failure in the communication with the host it switches to the next available one.

Only limitation for the moment, the management of recurrent callbacks (filter.watch) needs to be handled at client level to reinitialize the subscription. Any idea to automate this at the level of the provider is welcomed.

Please review and advise on how to improve.

/* USAGE
 import {MultiHttpProvider} from './imports/api/MultiHttpProvider'
 var MultiProvider= new MultiHttpProvider(["http://localhost:8545", "http://localhost:8546"], 0);
 web3 = new Web3(MultiProvider);

 Source: G. de Cadoudal (guenoledc@yahoo.fr)
 */


var MultiHttpProvider = function(hosts, timeout) {
  this.hosts= hosts;
  this.current = -1;
  this.timeout = timeout;
  this.currentProvider=undefined;
  this.providers=hosts.map(function(host) {
                           return new Web3.providers.HttpProvider(host, timeout);});
  // WARNING : This is because of my local Meteor setup where I cannot figure out why XMLHttpRequest and XHR2 are not defined when ethereum_web3.js is loaded.
  // so I forced the setting of these functions via customed defined function inside ethereum_web3
  this.providers.forEach(function(p){
                         p.setXMLHttpRequest( require('xmlhttprequest').XMLHttpRequest );
                         p.setXHR2( require('XHR2').XMLHttpRequest );
                         console.log(p);
                         });
  this.switchToNextConnected();
}


MultiHttpProvider.prototype.prepareRequest = function (async) {
  if(this.currentProvider) {
      return this.currentProvider.prepareRequest(async);
  } else {
      if( this.switchToNextConnected() )
          return this.prepareRequest(async); // we connected so call again
      else // it has not been possible to find a valid connection
          throw new Error('CONNECTION ERROR: Couldn\'t connect to any of the nodes '+ this.hosts +'.');
  }
}

MultiHttpProvider.prototype.send = function (payload) {
  if(this.currentProvider) {
      try {
          return this.currentProvider.send(payload);
      } catch(error) {
          console.log("Error with host "+this.currentProvider.host+". Switching to the next", error);
          if(this.switchToNextConnected())
              return this.send(payload); // try the next connection
          else throw new Error('CONNECTION ERROR: Couldn\'t connect to any of the nodes '+ this.hosts +'.');
      }
  } else {
      if(this.switchToNextConnected())
          return this.send(payload); // try the next connection
      else throw new Error('CONNECTION ERROR: Couldn\'t connect to any of the nodes '+ this.hosts +'.');
  }
}


MultiHttpProvider.prototype.sendAsync = function (payload, callback) {
  // Caveat: When the callback is supposed to be called several times over time (eg eth.filter, eth.watch)
  // the node can be put down and then the callback returns an error.
  // the end user must process that error to renew the subscription on a different node.

  if(this.isConnected()) // first check that there is a connection in synchronous mode
      this.currentProvider.sendAsync(payload, callback);
  else callback(new Error('CONNECTION ERROR: Couldn\'t connect to any of the nodes '+ this.hosts +'.'), undefined);
}


MultiHttpProvider.prototype.isConnected = function() {
  if(this.currentProvider)
      if(this.currentProvider.isConnected()) return true;
      else { this.switchToNextConnected(); return this.currentProvider!=undefined;}
  else { this.switchToNextConnected(); return this.currentProvider!=undefined;}
}

MultiHttpProvider.prototype.switchToNextConnected = function() {
  // if there is no available provider the loop does nothing
  for(var i=1; i<=this.providers.length; i++ ) {
      this.current++; // move to next one. the first time it moves from -1 to 0;
      // if we have gone over the nb of provider revert to 0;
      if(this.current>=this.providers.length) this.current=0;
      if(this.providers[this.current].isConnected()) {
          this.currentProvider=this.providers[this.current];
          return true;
      }
  }
  this.currentProvider=undefined;
  console.log("MultiHttpProvider could not find an active node in ", this.hosts);
  return false;
}



export { MultiHttpProvider };

Answer 3

I have been unsatisfied with this situation of loosing my subscriptions (Filters) when I lose the geth node and having to implement everywhere a resubscribe mecanism.

So for all having the same problem, I have packaged a MultiHttpProvider npm module that does it all for you and does the work nicely.

Please report any issue.