Why did the geth node stop syncing?

Question

to be honest, I'm already tired of fighting with the eternal geth bugs. Now, after a hosting failure, the node, which had been working normally for weeks, stopped syncing and hung on "Syncing: state healing in progress". In the geth console, web3.eth.blockNumber returns 0. Prysm beacon client is fine, and continues approving blocks. I studied this problem on the Internet, and tried everything that was offered to me, including changing geth builds to unstable. None of this helped me. I notice that only block headers are imported, and block receipts are imported a little bit at node startup, but then I only see such messages:

Feb 19 10:25:27 Imported new block headers               count=1    elapsed="364.699µs" number=16,661,727 hash=ee51c8..aabdb1
Feb 19 10:25:27 Downloader queue stats                   receiptTasks=0    blockTasks=0    itemSize=276.22KiB throttle=950
Feb 19 10:25:35 Syncing: state healing in progress       accounts=19734@1.09MiB    slots=88381@6.82MiB   codes=1597@11.07MiB nodes=18,720,858@4.89GiB pending=14404
Feb 19 10:25:37 Forkchoice requested sync to new head    number=16,661,728 hash=e33a23..70c586
Feb 19 10:25:39 Imported new block headers               count=1    elapsed=4.820ms     number=16,661,728 hash=e33a23..70c586
Feb 19 10:25:43 Syncing: state healing in progress       accounts=19851@1.09MiB    slots=88597@6.83MiB   codes=1597@11.07MiB nodes=18,722,242@4.89GiB pending=14328
Feb 19 10:25:48 Forkchoice requested sync to new head    number=16,661,729 hash=55444d..ef9584
Feb 19 10:25:51 Imported new block headers               count=1    elapsed="404.091µs" number=16,661,729 hash=55444d..ef9584
Feb 19 10:25:51 Syncing: state healing in progress       accounts=20027@1.10MiB    slots=88831@6.85MiB   codes=1598@11.08MiB nodes=18,723,875@4.89GiB pending=14227

here is the command I use to launch the node:

/geth/geth/geth --cache 2048 --maxpeers 16 --syncmode full --datadir /ethereum/execution --http --http.api eth,net,engine,admin,web3

Answer 1

You are currently starting Geth as a full node, from geth docs source:

A full block-by-block sync generates the current state by executing every block starting from the genesis block. A full sync independently verifies block provenance as well as all state transitions by re-executing the transactions in the entire historical sequence of blocks. Only the most recent 128 block states are stored in a full node - older block states are pruned periodically and represented as a series of checkpoints from which any previous state can be regenerated on request. 128 blocks is about 25.6 minutes of history with a block time of 12 seconds. To create a full node pass --syncmode full at startup.

Also, by default, full sync mode have a internal cache default value equals to 4096 and you are lowering it, it can reduce the performance.

Executing every block could take really a long time, did you really need to start a node as a full node? Try the default "snap" sync mode, you can find more on the source page linked before