I've been building the full blockchain with geth. I resorted to using ZFS snapshots to recover from repeated failures caused by corrupted blocks according to geth reports. Whenever unrecoverable corruption was detected, I issued rollback in the hope that the corruption was recent and the process could continue. This worked for awhile but ultimately failed, and I assume this is because very early blocks are corrupt. So I've moved geth to another system in the hope that unreliable RAM was the cause of the trouble. I'm considering building a system with ECC RAM. But that is a side issue. To my question:
Now I'm observing progress on system 3 which seems to be doing well, albeit more slowly on HD instead of SSD as before, but I'm curious about a peculiarity seen with each sync attempt:
The snapshots below are named according to the block number that was complete at the time of the snapshot. I have adjusted the snapshot interval periodically so as to evenly space them out in terms of perceived work completed, or essentially, time. The curious part is the wildly varying ratios between of blockchain blocks logged vs. apparent space modified in the filesystem.
As most readers probably know, like most snapshot support, ZFS uses copy-on-write to preserve modified disk blocks. The space allocated to a snapshot is thus proportional to the dispersal of modification activity since the snapshot. This can be seen in the USED column of the zfs list report:
root@merle:/blockchain/eth# zfs list
NAME USED AVAIL REFER MOUNTPOINT
naked3t 586G 2.06T 19K /naked3t
naked3t/blockchain 108G 2.06T 19K /blockchain
naked3t/blockchain/eth 108G 2.06T 27.8G /blockchain/eth
naked3t/blockchain/eth@2278805 2.56G - 5.63G -
naked3t/blockchain/eth@2598969 14.4G - 24.5G -
naked3t/blockchain/eth@2636231 14.4G - 24.7G -
naked3t/blockchain/eth@2675264 18.8G - 25.0G -
naked3t/blockchain/eth@2680721 20.9G - 26.4G -
naked3t/blockchain/eth@2682986 3.38G - 27.8G -
Note that geth apparently raced through ~37K blocks and required 14.4G of space to maintain the prior snapshot. But later, it slowed to a crawl, and I triggered a snapshot after only 2300 blockchain blocks, yet 20.9G had been consumed in maintaining the prior state to that point.
My questions are about what accounts for the differential, and to further explain, if this is predictable, as if there is something about the activity in certain period of the history that causes this, or if it is non-deterministic and I can expect to see different behavior if I were to restart the process. FWIW, the current slow period in process is around November, 2016.
Insights?
