Just because you're looking for different ideas ...
I'd probably approach this by separating concerns and enlisting the IT department to provide an always on "cluster" of ethereum nodes with a virtual IP address that isn't supposed to fail. Conveniently, nodes would tend to stay in sync provided geth is running in all places at all times.
From the apps perspective, you would concentrate on recovering from the loss of callbacks "in flight" when a failover (to another physical node) occurs. It might not be too difficult to habitually retry requests so eventual app recovery is assured immediately after the Ethereum cluster resumes.
The watchers would be interesting. It might not be especially easy to detect a failover if the cluster runs smoothly enough (IT's goal). On the app side, I would look for something detectable that's different between the physical nodes, and re-dispatch the watchers when a failover is detected.
- e-node would be different.
You could have transactional callbacks that persistently retry after timeouts instead of failing altogether. You get stalled requests that eventually work.
You could have periodic checks of e-node ID and wrap your watcher dispatches inside a condition that runs whenever a change of node is detected. So basically, whenever the "node" is different than before, we assume that the watchers need to be dispatched.
To avoid redundant watchers on any node, inform the IT department that when a node is "relieved/demoted" (for whatever reason), restart Geth (or the OS) before the node is "qualified" to take over again in the future. They'll understand the idea of always starting the service with a known clean state.
Indeed, cluster managers can also check the health of the nodes and confirm sync is complete before returning the node to the "pool" of healthy nodes that are ready to serve.
And, you can validate the app design without setting up the whole cluster in every detail. You'd be leaving the IT team with considerable lattitude about how they want to approach it. Your test case would be two nodes, one serving and one on standby. Kill the live node. Reconfigure the IP address on the standby node to "take over". Confirm there is no loss of service, no loss of "in flight" transactions and no confusion about the watchers.
Hope it helps.