4

I am planning to run a new full node, from scratch, using Parity.

From what I understand, it is imperative to use SSD disk(s) in order to do be able to do initial sync and keep up with the network.

However, I am unable to find any references to feasibility/performance of syncing + running a full node with 4 SSD disks (Samsung SM863a) in a RAID 10 setup.

Would it be able to handle the I/O load, or it is better to run single SSD? Would RAID setup actually wear the disks more/sooner?

I have tried running a test setup, on a server (32GB RAM, of which I allocated most to caching options in Parity, with 16GB cache-size) with 4 HGST HDDs, in RAID 10, but after 3 days of running it is around block 2,700,000 and is slowing down, so it would take around 60 days at this rate to fully sync.

Many posts that I found always refer to HDDs being a problem and apparently SSD disk is the way to go, but I really don't want to invest into a new setup just to find out that it will again sync at the very slow speed.

Any experiences and/or advices are very welcome :)

1 Answer 1

1

TLDR; you probably just want to save some money by replacing/adding a single SSD to your array instead of running it in RAID 10.

With a high enough mirroring level and a large enough disk cache, you could probably sync the Ethereum blockchain on a HDD, but one SSD will do due to the low latency of random seeks; there's no need to put SSDs into RAID.

You haven't mentioned whether you're doing software or hardware RAID, but in any case, I'm guessing your system is more than capable of handling RAID 10 without any negative effects on syncing if it just boils down to the use of RAID itself.

Will see more wear? Yes: you are writing twice as much data due to the mirrored portion of your RAID 10 (the RAID 1 part). So it's an increase by at least (and close to) a factor of two. But the wear is spread out over four disks, so I think you'd see a per-disk life expectancy of roughly double what you'd see in a RAID 1 or JBOD setup (i.e., you'll be doing less swapping out of disks due to wear); OTOH, at present rates, it'll probably take a very long time to wear out the disks (on the order of years/decades, if not more, to attain the number of write cycles even the lowliest of SSDs are rated for).

Thus, the answer to whether your described setup would work is yes. But I have a different question: why are you thinking of RAID10? (I'll assume a 2x2 array) Is it because you need/want RAID 10 for some other purpose? The only reason I can think of needing RAID 10 is because perhaps you plan on resilvering an existing pool to switch to SSDs with zero downtime (if you're on ZFS, there are probably better ways around this). Here's why I think RAID 1 is more suitable than RAID 10 for this upgrade: If you've got a RAID stripe (RAID 0), then I'm guessing this is to increase your read throughput and/or latency and/or total addressable capacity and it's already meeting your needs. But a not-particularly-speedy SSD can give you more than 2x throughput than an HDD -- and thus provides at least as much throughput as a striped HDD volume. It also gives you far more IOPs than a 2-disk striped HDD volume (two or more orders of magnitude). Because Ethereum is essentially a giant database in which there are many reads/writes, the low disk access latency of SSDs provides a HUGE advantage in terms of blockchain sync speed. Unless you need to RAID your SSDs to achieve the single-volume capacity you need, you can save your money and skip striping your SSDs.

I also think you should skip mirroring this disk as well, unless you plan on keeping things other than the blockchain on it or you want to achieve higher uptime due to online redundancy (e.g., in a production server environment). I would assume you'd actually want node-level redundancy in a high-availability setup anyway, so disk mirroring wouldn't entirely address high availability, anyway. You can always resync the blockchain if/when your disk goes kaput -- no real data loss (I'm assuming you back up your keys separately -- redundancy is not backup and untested copies are not backups!). To reduce sync time, if your volumes/disks support snapshotting you can always snapshot and dump your data to a lower-cost medium like your existing HDDs. Then a restore is as simple as restoring from the snapshots and you only need to do a partial resync. In this case, the IOPs advantage of SSDs is moot -- you just need throughput to restore your backup which your HDDs provide more than enough of. The full Ethereum blockchain (instead of pruned syncs) is about 350GB at the moment. You could restore that amount of data to an SSD from a single HDD in less than an hour (assuming 120MB/s HDD transfer rate). So... no need for mirroring, either!

That being said, one of my Ethereum nodes is running on a 2x512GiB RAID1 ZFS volume made up of SSDs. This is because I plan on putting other data that needs low latency on those disks at some later point in time (right now, the disks are mostly empty). Instead of dedicating a non-redundant disk to the task, I used the mirrored volume because I ran out of hot-swap disk bays and didn't want to "waste" a bay. Thus, I can see several use cases for RAID10 or other RAID levels being used for the blockchain -- I just don't think there are that many. Hopefully I saved you some $$$ :)

More performance data with Parity/v1.7.7-stable-eb7c648-20171015/x86_64-macos/rustc1.20.0: I just did a brand new sync in pruned mode (which should do at least as much I/O as archive mode) on an NVMe drive; it's only 256 GiB, so I couldn't do an archive test if I wanted. My network connection is ~200Mbps up and down. Parity seems to be much more read-IOPS-intensive than write-IOPS, but does about 2x as much writing in terms of volume of data, so RAID1 seems to provide more performance benefits than RAID0. Also, I'm only seeing 5-10k IOPS (<5% utilization) and 200MB/s throughput. and thus seem to be CPU constrained: ~2 cores maxed out on an i7 6700HQ (2016 15" MBP). The process took 2h29m.

5
  • Thank you for informative answer. I am mostly interested in RAID10 for reliability purposes (no downtime in the future due to HDD failure). However, I am still lost as to what are exacty requirements to run a full node from scratch. The server I am trying it on is a very decent one, disks are quite fast as well, but getting to 50% in 3 days and then watching it slow down is worrying me. Apparently, it processes 4 blocks in approximately 10 seconds, which means that I would need around 60 days to sync fully. So, if it is a HDD issue, I was considering switching to SSDs.
    – user21855
    Oct 29, 2017 at 16:55
  • If you're only interested in reliability, then a 2-disk RAID 1 would better suit your needs than a 4-disk RAID 10. It is cheaper, slightly less performant (but probably not meaningfully different for most purposes), and more reliable. Of course, a 4-disk RAID1 would be even more reliable and give you better read performance than a 2-disk RAID1. If you're managing a ZFS pool with a lot of disks and you're buying 4 disks anyway, you're probably better off using a 3-disk RAID1 SSD array for Ethereum and put in one extra hard disk drive as a hot-spare (assuming the rest of your array is HDDs).
    – lungj
    Oct 29, 2017 at 17:18
  • The Ethereum blockchain is a giant database. Because the database is updated and read every block, there is a lot of random read/write access. Especially if you have block pruning on. Because HDDs have slower random access time, it takes longer to do all the read/write operations on the database. If the amount of time taken to do the reads and writes exceeds 30s (or whatever the block time is), you'll never be able to sync the chain. This is possible on slower volumes if there is a lot of activity on the blockchain. SSDs have fast random access/writes and thus often better suited for DB use.
    – lungj
    Oct 29, 2017 at 17:28
  • RAID10 is simply for performance reasons, since it will have better write performance than RAID1. As I said, it is actually very hard to find definitive answer on required hardware to run a full node, so I just wanted to be on the safe side. Considering that you are apparently running a full node on 2 SSD drives in RAID1, I will consider that I have got an answer that I was looking for. Real-life experience with a full node, basically :) Thank you.
    – user21855
    Oct 29, 2017 at 17:40
  • I got a powerful machine with 1 SSD and enough RAM for a full node. It's impressible how io-bound syncing is when bandwidth and CPU power are not an issue. ATM Parity is using about 20G of RAM.
    – arhuaco
    Aug 7, 2018 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.