Restarting geth node, doesn't work properly with cron nor Ubuntu upstart service

Question

Restarting geth node, doesn't work properly with cron nor Ubuntu upstart service

My geth node fails once in a day or so. Tried everything, but still keeps failing.

Geth 1.6.1 hangs throwing "unexpected fault address" fatal error

Upgraded geth to last version, rebuilt chain from scratch, but keeps failing once in a day.

So a restarting system was needed in order to keep the node up & running.

First try was creating a cron entry like this:

*/1 * * * * /home/my_local_username/ethereum/launch.sh

that checks whether geth is running or not. If not, it launchs geth again:

!/bin/sh

env - cat /home/my_local_username/my_env.sh /bin/sh

ps auxw | grep geth | grep -v grep > /dev/null

if [ $? != 0 ] then /home/my_local_username/ethereum/geth --testnet --rpc --rpcapi "db,eth,net,web3,personal" --rpcport "8545" --mine --etherbase "0x8c6..." --gasprice "1000000000" --bootnodes "enode://20c..." 2> /home/my_local_username/ethereum/nohup.out fi

The environment file /home/my_local_username/my_env.sh was created using this command:

env > /home/my_local_username/my_env.sh

This way, when geth fails it starts over after 1 minute as much. And it works and it mines, but not properly. I mean, when geth is relaunched with the cron it has a weird behaviour. It doesn´t let me open a terminal like this:

$ ./geth --testnet attach ipc:/home/my_local_username/.ethereum/testnet/geth.ipc

It fails with the error:

Fatal: Unable to attach to remote geth: dial unix /home/my_local_username/.ethereum/testnet/geth.ipc: connect: connection refused

And when sending commands through the JSON-RPC link, it fails too with the error:

Unknown account

When launching geth manually from a shell terminal, those two things work properly. When relauched from cron, they don't.

Second try was creating a service on Ubuntu for geth, as explained at https://ethereum.stackexchange.com/a/2249/2426

The result, the same weird behaviour: unable to connect to geth through terminal and unknown account error when sending commands through JSON-RPC port.

This is the log from geth when starting as a cron process:

INFO [09-14|08:20:02] Starting peer-to-peer node               instance=Geth/v1.6.5-stable-cf87713d/linux-amd64/go1.8.3
INFO [09-14|08:20:02] Allocated cache and file handles         database=/home/bitnami/.ethereum/testnet/geth/chaindata cache=128 handles=1024
INFO [09-14|08:20:06] Initialised chain configuration          config="{ChainID: 3 Homestead: 0 DAO: <nil> DAOSupport: true EIP150: 0 EIP155: 10 EIP158: 10 Metropolis: 9223372036854775807 Engine: ethash}"
INFO [09-14|08:20:06] Disk storage enabled for ethash caches   dir=/home/bitnami/.ethereum/testnet/geth/ethash count=3
INFO [09-14|08:20:06] Disk storage enabled for ethash DAGs     dir=/home/bitnami/.ethash                       count=2
INFO [09-14|08:20:06] Initialising Ethereum protocol           versions="[63 62]" network=3
INFO [09-14|08:20:06] Loaded most recent local header          number=1658958 hash=2eacdb…cdfa00 td=1833602778380180
INFO [09-14|08:20:06] Loaded most recent local full block      number=1658958 hash=2eacdb…cdfa00 td=1833602778380180
INFO [09-14|08:20:06] Loaded most recent local fast block      number=1658958 hash=2eacdb…cdfa00 td=1833602778380180
WARN [09-14|08:20:07] Blockchain not empty, fast sync disabled 
INFO [09-14|08:20:07] Starting P2P networking 
INFO [09-14|08:20:10] UDP listener up                          self=enode://d01d74c3b76f6bcf57c548d36b977b49745328566e2a12c00dded0b4ad0970518a5615cde8fd833f8f2a93b35482cf373479baf8e0c634b44123d6c8067a2615@[::]:30303
INFO [09-14|08:20:10] RLPx listener up                         self=enode://d01d74c3b76f6bcf57c548d36b977b49745328566e2a12c00dded0b4ad0970518a5615cde8fd833f8f2a93b35482cf373479baf8e0c634b44123d6c8067a2615@[::]:30303
INFO [09-14|08:20:10] IPC endpoint opened: /home/bitnami/.ethereum/testnet/geth.ipc 
INFO [09-14|08:20:10] HTTP endpoint opened: http://127.0.0.1:8545 
INFO [09-14|08:20:10] Starting mining operation 
INFO [09-14|08:20:11] Commit new mining work                   number=1658959 txs=0 uncles=0 elapsed=424.726ms
INFO [09-14|08:24:21] Block synchronisation started 
INFO [09-14|08:24:21] Mining aborted due to sync 
INFO [09-14|08:24:32] Imported new chain segment               blocks=1 txs=10 mgas=1.612 elapsed=10.313s   mgasps=0.156 number=1658959 hash=5b879b…1d974e

I am quite puzzled about all these errors. Any hint on it will be very helpful for me. Thx!

Refs:

Restart Geth from another process

Answer 1

It sounds like the socket is not connected to your geth instance. I saw here that geth.ipc could cause problems, you can try:

rm -f /home/my_local_username/.ethereum/testnet/geth.ipc

I don't see the bug myself but this socket stays when hard killing geth and could be in a bad state for you. You should also check the logs on starting geth to see if any socket warnings or errors are raised.

Answer 2

That is the strangest way I have seen of keeping a Geth instance running :)

I suggest that you do the following:

• Firstly, If you start Geth in your terminal, and then close the terminal you will also close Geth.

You will want to keep your node running in a terminal accessible on ssh'ing into the server. I recommend you use screen.

• Next create a service configuration in /etc/systemd/system. Name it appropriately - node.service for example.

Use a configuration like this:

[Unit]
Description=Geth Node

[Service]
ExecStart=/usr/bin/screen -DmS node-screen -c /path/node.screenrc
ExecStop=/usr/bin/screen -S node-screen -X quit
User=root
Group=root
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target

This tells the service to create a screen (called node-screen) and run it using the configuration found in node.screenrc (a screen configuration file).

• The configuration file should contain something like:

  screen -t 'Geth node' geth --fast --your-other-config-here

On restarting your server the service will be run and the Geth node will restart.

• If your node crashes randomly you can restart it with:

service node restart

• If you want it to constantly run you could change the screen configuration to run a shell script (instead of starting the node directly) such as the following:

I.E. Put the Geth start code within a constantly executing while loop such that if it crashes it gets restarted.

#!/bin/bash

while true
do
    geth --fast --other-config
done