syncing forever

Question

I'm trying to mine using the commands geth --rpc and ethminer on an ubuntu 16.04 machine. Sometimes it works, but often after I restart my PC, I get the error "Network syncing, will start miner afterwards". This happens even after letting it run for days, and even when it had been working before on the same day. That means synchronization does not work, because the additional data it has to download for the meantime is so tiny, it can't normally take hours. It should take minutes at most.

As a work around, I then delete all the chain data and start syncing from scratch. That takes a complete day, but then it works again. Then after trying it out several times and restarting, the problem comes back and I need to download or recreate all the 30GB again. This is a very nasty problem.

What can I do about this?

Here is the last 10 lines of geth's output:

...
INFO [11-17|16:10:23] Imported new chain segment               blocks=2  txs=118  mgas=8.346  elapsed=11.620s mgasps=0.718  number=4018787 hash=b33409…30e233
INFO [11-17|16:10:37] Imported new chain segment               blocks=3  txs=137  mgas=13.701 elapsed=13.448s mgasps=1.019  number=4018790 hash=e652f8…f20c4e
INFO [11-17|16:10:45] Imported new chain segment               blocks=2  txs=174  mgas=10.659 elapsed=8.115s  mgasps=1.313  number=4018792 hash=5cedef…eaf7ed
INFO [11-17|16:10:53] Network syncing, will start miner afterwards 
INFO [11-17|16:10:55] Imported new chain segment               blocks=2  txs=135  mgas=8.080  elapsed=10.302s mgasps=0.784  number=4018794 hash=cc3e31…f5fff2
INFO [11-17|16:10:55] Network syncing, will start miner afterwards 
INFO [11-17|16:10:57] Network syncing, will start miner afterwards 
INFO [11-17|16:11:04] Imported new chain segment               blocks=1  txs=105  mgas=6.709  elapsed=8.731s  mgasps=0.768  number=4018795 hash=9053b0…379cf6
INFO [11-17|16:11:12] Imported new chain segment               blocks=1  txs=171  mgas=6.669  elapsed=8.369s  mgasps=0.797  number=4018796 hash=4af69b…9418a5
INFO [11-17|16:11:23] Imported new chain segment               blocks=2  txs=155  mgas=10.575 elapsed=10.816s mgasps=0.978  number=4018798 hash=41a86c…fc801d

corresponding ethminer output:

$ ethminer 
miner  16:10:53.876|ethminer  Getting work package...
JSON-RPC problem. Probably couldn't connect. Retrying in 1... 
miner  16:10:55.878|ethminer  Getting work package...
JSON-RPC problem. Probably couldn't connect. Retrying in 1... 
miner  16:10:57.879|ethminer  Getting work package...
^CON-RPC problem. Probably couldn't connect. Retrying in 2... 

Update:
The system has 16GB of RAM, a fast SSD and it's on a 100MBit/s connection. Speed test for the connection has good results.

Update:
I had it synced and mining today. Now I read that current block number should be 4570731, but my download is at 4018737, so more than half a million blocks are missing. I think my database is corrupted. Could that be the reason? If yes, why does it corrupt again and again?

Update:
I had it running since yesterday with --cache 8192. It's now at block 4136031, meaning it progressed by about 120000 blocks in 24h. That means it will take more than 5 further days to complete sync.

Answer 1

The miner will only work if you have a node that is in sync up to the last block, therefore your node needs to complete the sync before the miner starts.

If a node starts with empty database, a fast sync is usually done. The fast sync does not load the complete chain, which makes it faster than a full sync.

If your node is not running for a while, it will loose sync to the blockchain, and has to sync when you turn it on again. Once the database is not empty, however, a fast sync is no longer possible. The node then does a full sync. A full sync requires more resources than the fast sync, especially RAM seems to be relevant for a carrying our a full sync.

Therefore, and based on my experience, I recommend the following:

• make sure you have sufficient RAM on the machine running your node, 8GB is minimum, better more.
• give geth a larger cache by using option --cache cache_size_in_MB
• select the cache size as large as possible but according to your RAM size. My experience is that you should set the cache size to about 30% to 40% of the total RAM, otherwise geth is likely to run into out of memory issues later if you are operating the node for longer time. So for the 8GB RAM try --cache 3072

Note that if you have allocated sufficient cache to geth, you will see that geth imports several 10 blocks per per "Imported new chain segment" message.

Another experience I made is that geth cannot follow with updating the local copy of the block chain, if you do not provide sufficient cache size to geth. So I recommend keep the cache setting after sync.

Answer 2

To start out, there are many very different problems with syncing and what you experienced is most likely the overlapping symptoms of more than one problem, so it's near to impossible to diagnose it here. The best option is to try syncing and if it doesn't complete within a day, delete the chaindata and start a fast sync from scratch (see below why).

There are two kinds of syncs, a normal sync and a fast sync. A fast sync can only be done from scratch, e.g. after you deleted your chaindata dir. A normal sync can be done from an in between state.

To date, the fast sync takes several hours on good hardware and excellent connection. The normal sync, however, is awfully slow even compared to that. If you didn't sync for up to some hours, it makes sense to use the normal sync, because it can continue from the previous state and it has so little to do to catch up. But if you didn't sync for a few months, then doing the normal sync is impractical. It could take more than two weeks until synced, so in this case you are better off deleting the chaindata dir and starting a fast sync, which takes a few hours on good equipment, or maybe a day on average equipment and connection.

Note that when this answer gets old (e.g. you are reading this in 2018 or later), the estimated times may not reflect the current situation anymore, but the concept is still the same: If you have synced more than a few weeks ago, delete the chaindata dir and start a fast sync.