I'm trying to run a geth full node with archiving for development\learning and custom analytics. I have a physical Windows server running HyperV where I carved out the following VM specs:

  • 24 vCPUs
  • 128 GB DDR3 RAM
  • 80GB OS drive on local NVME storage
  • 8TB Tiered Storage Drive (see below)*
  • Ubuntu 20.04
  • Geth 1.10.2-stable

According to etherscan stats a full node with archive today is about 7.2TB of disk.

*Unfortunately I don't have 8TB worth of SSD, but I have a NAS with 24TB of HDD in a raid which can throughput 116MB\s (~about max speed of 1GB nic), and another locally attach 1TB nvme. Previous posts indicate that running a full node on HDD is not sufficient or possible to keep up, but what about tiered storage?

Since my host is windows, I'm using storage spaces to combine the local 1TB nvme with 8TB iscsi LUN with SSD,HDD tiering. I then attached the tiered disk to VM as /eth1 to host the chaindata.

I'm running geth via following command:

./geth --datadir /eth1/data/ --datadir.ancient /eth1/data/geth/chaindata/ancient --mainnet --syncmode full --gcmode archive --txlookuplimit=0 --cache.preimages --cache=8192

Other thoughts

The tiering functionality is a bit of a black box to me, and i'm not sure when certain chaindata is considered hot or cold to storage spaces. I think all initial writes go to LUN. I'd be nice if I could just configure geth to say use 950GB of recent data on drive X, and store\move all other chain data to drive Y, which I assume is kind of how the freezer works? I saw there is a datadir.ancient parameter but not sure when and how that data is moved there.

  • 1
    It has been tried multiple times and it won’t work as writing is distributed equally all over the database. Tired storage is only for backup no longer updating. However, if you don’t want to update your nodes you can use archivenode.net. Please note that low end ssd just fast enough to keep the node synced but not catch up costs $800 (for 7.6Tb). May 28 '21 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.