Why does geth (fast) sync beome slower?

Question

I've been running geth 1.9.21-stable-0287d548 in Ubuntu 20.04 (VPS with 2CPU, 8G RAM, and 400G SSD) for close to 24 hours.

geth is run in default fast sync mode --

geth --http --http.addr 0.0.0.0 --datadir /mnt/goethereum

It seems to be stepping back-and-forth around the last 100ish blocks --

$ geth attach http://localhost:8545
> eth.syncing.highestBlock - eth.syncing.currentBlock
192
> eth.syncing.highestBlock - eth.syncing.currentBlock
72
> eth.syncing.highestBlock - eth.syncing.currentBlock
74
> eth.syncing.highestBlock - eth.syncing.currentBlock
108
> eth.syncing.highestBlock - eth.syncing.currentBlock
124
> eth.syncing.highestBlock - eth.syncing.currentBlock
65
> eth.syncing.highestBlock - eth.syncing.currentBlock
72
> eth.syncing.highestBlock - eth.syncing.currentBlock
95
> eth.syncing
{
  currentBlock: 10943724,
  highestBlock: 10943830,
  knownStates: 369296404,
  pulledStates: 369063841,
  startingBlock: 10941014
}

I don't find anything useful in the geth log --

Sep 27 16:33:33 geth-mainnet geth[541]: INFO [09-27|16:33:33.849] Imported new block headers               count=1    elapsed=33.211ms    number=10943558 hash="f6e7e1…9cc02f" age=1m7s
Sep 27 16:33:34 geth-mainnet geth[541]: INFO [09-27|16:33:34.186] Imported new state entries               count=384  elapsed="158.345µs" processed=364890530 pending=103974 trieretry=0    coderetry=0 duplicate=749 unexpected=5371
Sep 27 16:33:42 geth-mainnet geth[541]: INFO [09-27|16:33:42.965] Imported new block headers               count=1    elapsed=32.726ms    number=10943559 hash="56b61c…4b4f50"
Sep 27 16:33:43 geth-mainnet geth[541]: INFO [09-27|16:33:43.224] Imported new state entries               count=384  elapsed="2.403µs"   processed=364890914 pending=104364 trieretry=0    coderetry=0 duplicate=749 unexpected=5371
Sep 27 16:33:52 geth-mainnet geth[541]: INFO [09-27|16:33:52.320] Imported new state entries               count=384  elapsed="3.086µs"   processed=364891298 pending=104749 trieretry=0    coderetry=0 duplicate=749 unexpected=5371
Sep 27 16:33:55 geth-mainnet geth[541]: INFO [09-27|16:33:55.005] Imported new block headers               count=1    elapsed=23.788ms    number=10943560 hash="4fe922…a04038"
Sep 27 16:33:58 geth-mainnet geth[541]: INFO [09-27|16:33:58.038] Imported new block headers               count=1    elapsed=15.386ms    number=10943561 hash="efe84b…6ec63f"

What's wrong here and how can I sync to the head?

P.S. Close to 10 hours after geth start, there was a drastic downturn in CPU, disk, and network usage --

Answer 1

All credits go to @holiman on Discord "Go Ethereum" --

@jerryji the network charts look like that because the data-heavy header/block-body download is done, what's left is remaining state download. It fetches in total ~600M entries, with max 384 items per request, so there are quite a lot of requests that need to be made. The network latency + peer latency is the limiting factor, most likely, and the CPU might be pretty idle. Sometimes though, with good peers, the local IO might be a bottleneck too, because there's a heavy read pressure when checking if entries are already present or not (this last check goes orders of magnitudes faster if the state bloom is fully loaded, hence restarting fast-sync incurs a large penalty as the statebloom needs to be re-initialized)