Geth full node stuck on "Migrating ancient blocks". Normal?

Question

I'm resurrecting my geth node after being offline since late 2018. I copied the old data dir to my server and compiled the latest stable geth. Everything was fine at first, but now sync seems to have stalled.

cli: geth console --datadir /mnt/san-data/blockchains/eth/blockchain/ --syncmode full --cache 16384 --http --ws --nat extip:[redacted]

exert from output:

INFO [04-26|16:56:33.882] Migrating ancient blocks                 count=101661 elapsed=15h58m42.135s
> eth.blockNumber
0
> eth.syncing
{
  currentBlock: 4000261,
  highestBlock: 12312841,
  knownStates: 251868014,
  pulledStates: 251786330,
  startingBlock: 4000261
}
> INFO [04-26|16:57:34.617] Writing clean trie cache to disk         path=/mnt/san-data/blockchains/eth/blockchain/geth/triecache threads=1
INFO [04-26|16:57:34.618] Regenerated local transaction journal    transactions=0 accounts=0
INFO [04-26|16:57:34.655] Persisted the clean trie cache           path=/mnt/san-data/blockchains/eth/blockchain/geth/triecache elapsed=38.401ms
INFO [04-26|16:57:48.442] Migrating ancient blocks                 count=101662 elapsed=15h59m56.695s
INFO [04-26|16:59:03.649] Migrating ancient blocks                 count=101663 elapsed=16h1m11.901s
INFO [04-26|17:00:17.920] Migrating ancient blocks                 count=101664 elapsed=16h2m26.173s
... ======= 5+ hour cut =============
INFO [04-26|22:25:29.949] Migrating ancient blocks                 count=102084 elapsed=21h27m38.202s
INFO [04-26|22:26:53.417] Migrating ancient blocks                 count=102085 elapsed=21h29m1.670s
INFO [04-26|22:28:17.714] Migrating ancient blocks                 count=102086 elapsed=21h30m25.967s
INFO [04-26|22:29:41.789] Migrating ancient blocks                 count=102087 elapsed=21h31m50.042s

> eth.syncing
{
  currentBlock: 4000261,
  highestBlock: 12312841,
  knownStates: 251868014,
  pulledStates: 251786330,
  startingBlock: 4000261
}
> eth.blockNumber
0

$ geth version
Geth
Version: 1.10.2-stable
Git Commit: 97d11b0187b4695ccf44e3b71b54155fe405a36f
Git Commit Date: 20210408
Architecture: amd64
Go Version: go1.16.3
Operating System: linux

disk space:

$ df -h /mnt/san-data/
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/san-data   60T  3.9T   53T   7% /mnt/san-data

iotop in accumulative mode notes that geth wrote 8.4G and read 56K of disk in that 5+ hour time window, and hardly any current activity (<100k/s). All open files (via procfs /proc/{pid}/fd) are under the data dir as expected and no other disks are running out of room. netstat shows 50 active connections to geth, 34 of them inbound. nload show spurts of activity (probably from other things running on the server) but plenty of available bandwidth. top shows geth has a single core monopolized. Did it actually stop syncing to do something to every block, 1/3rd of the way through syncing? If so, is there any way to parallelize whatever it's doing?

e: Server is a dl580 chassis with 4 sockets each with a Intel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz for 80 logical cpus; 640gb ram; san is a PowerVault MD1200 with 60TB effective storage. Please spare me the lecture about using 10+ yo hardware for a hobby project. Running Ubuntu 20.04.2 LTS server on 5.4.0-72-generic kernel

ee update: Per Ismael's suggestion in the comments, I looked into creating an issue report in git. Per git issue posting guidelines, I attempted to restart the node before filing an issue. This resulted in failure to restart due to apparent db corruption. So now I'm syncing from scratch. I'll update once it catches up to where it was, to say whether it has the same issue again.

Answer 1

Long story short, I accidentally corrupted the chain state db while trying to resolve this issue, and ended up re-syncing from scratch. That has blown past the block it was previously stuck (now at block 4539217) in less time than it had spent stuck before I broke it. I conclude that there was something wrong with the old chain data, and the easiest (if not best) solution is to re-sync. I do not know if the chaindata was already corrupt for having sat on cheap storage for 2+ years, or if this is some compatibility issue between very different geth versions.