Ethereum wallet/Geth incredibly slow to sync, help?

Question

So I've noticed some people having the issue similar to mine, entitling their post as "very VERY slow to sync" and other variations of that, however they're complaining at around 100 blocks every minute.

I have the issue where Geth is telling me that it's getting 1 block every 1 to 6 minutes.

I only have 48,806 blocks left to go (I had it fully synced a few months ago), and they don't exactly seem to be "problem" blocks like back in the 200,000 range.

I did have the issue last year of where it stopped syncing for some reason, so I've tried all the fixes that may help with that such as deleting all my chain data and trying a different hard drive, nothing making any real difference...

I'm on; Windows 10 Geth 1.7.3 Ethereum Wallet 0.9.3

Any help with this matter would be much appreciated.

Answer 1

I managed to have my wallet synced around the time you asked the question, running on a dedicated home server. Then I forgot to run geth for some hours after a power failure, and I was ~3000 blocks behind. Over the next 2 weeks, the "blocks left" in the UI has been varying, as low as 1200 up to about 4000.

Then a few days ago I upgraded to 1.8.0 and the following day to 1.8.1 of geth, and it has since slowed down significantly and I am now 7800 blocks behind.

My conclusion is; Either geth is a piece of junk and should not be used (I have not seen anything significantly better), OR (perhaps more likely) Ethereum has such major design flaw that it is doomed to succeed. If a high-ish end, Ethereum-dedicated home server can not catch up, then there is no future in Ethereum or any block-chain with similar traits.

The fact that the server spends 45% waiting on I/O, and that 10s of TB has been written to the file system, tells me that a. it is most likely that the developers don't know what they are doing, b. that even SSDs are not really a solution, as the writes will wear them down too quickly.

I have spent far too much money and far too much effort trying to get this to work. "Proof-of-Work" has suddenly a different meaning!

I have also no clue why there is so little responses on this, why this is not the most important topic to discuss in Ethereum and that every single person involved spend all there effort trying to solve it, UNLESS it is deemed unsolvable and it is now a matter of them making a smooth exit. Who knows?

In any event, I am not a happy camper and I am certain that there are many people like me.

Answer 2

There's a lot of chat about how bad Geth synchronization is. As the network scales, synchronization is not scaling as well. While last summer you could fast sync the blockchain in a matter of hours, now you need some hardware to get through a Geth sync quickly. It was discouraging enough to turn me off of Ethereum development for a while, but I've pushed through and cracked the sync case for myself.

If you think it's not bad to sync - even the developers joked about it with the release of Geth 1.8 "Iceberg" -- "Because [Geth 1.7] was un-sync-able."

As of April 2018, the hardware requirements for a full node assume big solid-state-backed hardware. One developer used an Azure Standard_L16s, a storage-optimized server and got a fast sync in about 8.5 hours in April 2018. The Geth team tested synchronization on the m4.2xlarge AWS instance which is $0.40 USD per hour to run. On AWS hardware, a full sync takes about 20 days and a couple of terabytes.

I've been running a cluster of Geth nodes across Digital Ocean, AWS, and OVH for a while, and I use a NUC on my desk for LAN development. I have run into nearly every Geth sync problem filed on GitHub at this stage, since I have to run full nodes for some of the development work I am doing.

If you're having trouble synchronizing full nodes, here are some tips:

1. Use SSD-backed hardware with provisioned IOPS. Fast syncs download and query the Ethereum state trie, which requires a lot of random disk I/O. Geth developer karalabe says you will not get the throughput on HDD. I can affirm this - using a Seagate 2TB SSHD there was no hope of getting sync to happen fast enough - even though I tried for nearly a month straight. I had OK luck with a fast sync to a WD Velociraptor RAID set, which barely provided the needed IOPS. The AWS i3 nodes work well for this. Any SSD-backed dedicated server works. If you have bare-metal for an archive node, you can probably get away with bcache.

2. Open port 30303. Geth on non-30303 ports for connections don't seem to work that great. Keeping 30303 open will allow you to peer and appears to help with "Synchronisation failed" errors.

3. Bootstrap geth with others' mainnet nodes. A developer named Rocky Fikki often seems to posts gists of the peers his geth nodes are connected to. If you're having trouble getting your geth node to peer to people that can give you the blocks you need, try seeding from his nodes.

4. Periodically restart your nodes. Geth has a lot of strange failure states and bugs that the developers are fixing or trying to fix. Sometimes Geth will drop all its peers and not reconnect to any. Sometimes for some reason I will always see successive "Synchronisation failed, dropping peer" and "Node data write error", which resolve after a restart. Sometimes Geth will have peers, talk to peers, but never actually sync anything.

   The sync mechanism in Geth is still somewhat black magic to a lot of people. Instead of trying to fix Geth's syncing problems, I just wrote myself a watchdog application called Quarian. Whenever Geth reaches a failure state I've seen before or lags too much, I just have it restart Geth and the problems usually resolve themselves.

5. Don't use your laptop as a primary dev node. Always keep something running that is 24x7 connected to the network. For dev, I bought an Intel NUC that contains a Samsung Enterprise PM863 SSD which can take the abuse. When I'm on my laptop, I will just connect to it, or on the go rsync the chaindata folder from it.

Remember how I said the thing about the full node? For most Ethereum transactions, you probably just need a light node. These sync fast on just about anything, and I'd recommend trying the light sync model if you can get away with it. If you don't need Geth-specific functionality, as early as last week (May 2018) I was able to get a parity --warp sync to finish and work fine in a day or so on a 2TB FireCuda SSHD using --db-compaction=hdd and -snapshot-peers=100.