Why my geth fast node suddenly stopped running?

Question

Fatal: Error starting protocol stack: gap (#3) in the chain between ancients and leveldb

when I run geth node for ropsten by :

geth --testnet --syncmode "fast" --cache=512 --rpc --ws --shh console

It throws Error:

INFO [09-11|21:33:55.437] Maximum peer count                       ETH=50 LES=0 total=50
INFO [09-11|21:33:55.437] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[09-11|21:33:55.438] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-11|21:33:55.438] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-11|21:33:55.438] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-11|21:33:55.438] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-11|21:33:55.438] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-11|21:33:55.438] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [09-11|21:33:55.440] Starting peer-to-peer node               instance=Geth/v1.9.3-stable-cfbb969d/linux-amd64/go1.11.5
INFO [09-11|21:33:55.440] Allocated trie memory caches             clean=128.00MiB dirty=128.00MiB
INFO [09-11|21:33:55.440] Allocated cache and file handles         database=/root/.ethereum/testnet/geth/chaindata cache=256.00MiB handles=524288
INFO [09-11|21:33:55.641] Opened ancient database                  database=/root/.ethereum/testnet/geth/chaindata/ancient
Fatal: Error starting protocol stack: gap (#3) in the chain between ancients and leveldb

I tried to delete ancient directory from chaindata but got this:

 Error starting protocol stack: ancient chain segments already extracted, please set --datadir.ancient to the correct path

Answer 1

https://github.com/ethereum/go-ethereum/issues/20383

Geth persists data after 1 hour worth of block processing (so for example if your blocks are processed in 100ms, that's 10/s * 60s * 60m = after 36000 blocks). If your network is super light (i.e. mostly empty blocks), it may happen that it takes a very very long time until blocks are flushed.

An ungraceful shutdown should still eventually sync up with the network when connections are made to old nodes that are still healthy, so it should eventually self heal itself.

Currently there is no way to configure the period for these persistency rounds because every piece of data pushed to disk cannot be garbage collected any more, so the more frequent, the more junk. You can hack around the issue currently with your workaround of restarting Geth periodically.

We have an alternative idea in the works that would persist the in-memory state not to the database, rather to a flat file. That will allow users to configure wall-clock timeouts for the backups without adding junk to their databases. Not sure when that will ship however, it's not a high priority feature.