Parity pruning=archive never syncd

Question

We are having an issue with parity, we are running it with archive pruning and sealing trying to keep the complete state of the blockchain (because we are using the rpc to query ALL aspects of it for export to a sql db). However recently we have found that it is not up to sync and is syncing so slow that it currently is not able to catch up. I ran the iotop utility and found parity to be using nearly 100% of the I/O capacity on a dedicated server with specs:

CPU: Intel Xeon E3-1230v6 - 4c/8t - 3.5GHz /3.9GHz RAM: 16GB DDR4 ECC 2400 MHz HDD: Sata 4TB

Output of iotop:

TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                                                                                            
4740 be/4 ethereum  766.54 K/s    0.00 B/s  0.00 % 99.99 % parity --relay-set strict --force-sealing --cache-size-db 4096 --jsonrpc-hosts all --pruning=archive --tracing=on --notify-work=http://localhost:3030 [IO Worker #3]
4757 be/4 ethereum  581.78 K/s    0.00 B/s  0.00 % 77.99 % parity --relay-set strict --force-sealing --cache-size-db 4096 --jsonrpc-hosts all --pruning=archive --tracing=on --notify-work=http://localhost:3030 [IO Worker #2]
4758 be/4 ethereum  283.03 K/s    0.00 B/s  0.00 % 50.83 % parity --relay-set strict --force-sealing --cache-size-db 4096 --jsonrpc-hosts all --pruning=archive --tracing=on --notify-work=http://localhost:3030 [IO Worker #3]
4755 be/4 ethereum  169.03 K/s    0.00 B/s  0.00 % 42.55 % parity --relay-set strict --force-sealing --cache-size-db 4096 --jsonrpc-hosts all --pruning=archive --tracing=on --notify-work=http://localhost:3030 [IO Worker #0]
4756 be/4 ethereum  176.89 K/s    0.00 B/s  0.00 % 32.75 % parity --relay-set strict --force-sealing --cache-size-db 4096 --jsonrpc-hosts all --pruning=archive --tracing=on --notify-work=http://localhost:3030 [IO Worker #1]

Some things that are odd, parity Syncs one block multiple times, the output seems to be missing some blocks in some cases, and the output is exactly every 10 seconds. At this rate parity will never fully sync though and will not catch up to the most recent blocks. Output of parity originally at 1Gb (--cache-size-db 1024) cache size:

2017-12-12 10:14:53  Syncing #4705029 0933…3695     0 blk/s   13 tx/s   0 Mgas/s      0+  640 Qed  #4705673   32/50 peers      7 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:15:03  Syncing #4705029 0933…3695     0 blk/s    0 tx/s   0 Mgas/s      0+  640 Qed  #4705673   31/50 peers      7 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:15:13  Syncing #4705030 7a7c…ec0a     0 blk/s   10 tx/s   0 Mgas/s      0+  640 Qed  #4705673   31/50 peers      6 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:15:23  Syncing #4705030 7a7c…ec0a     0 blk/s    0 tx/s   0 Mgas/s      0+  640 Qed  #4705673   31/50 peers      6 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:15:33  Syncing #4705031 f29b…2cb2     0 blk/s   12 tx/s   0 Mgas/s      0+  640 Qed  #4705673   34/50 peers      3 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:15:43  Syncing #4705031 f29b…2cb2     0 blk/s    0 tx/s   0 Mgas/s      0+  640 Qed  #4705673   39/50 peers      4 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:15:53  Syncing #4705031 f29b…2cb2     0 blk/s    0 tx/s   0 Mgas/s      0+  640 Qed  #4705673   41/50 peers      5 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:16:13  Syncing #4705033 8e92…a680     0 blk/s    8 tx/s   0 Mgas/s      0+  636 Qed  #4705673   39/50 peers      6 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:16:23  Syncing #4705033 8e92…a680     0 blk/s    0 tx/s   0 Mgas/s      0+  636 Qed  #4705673   39/50 peers      9 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:16:33  Syncing #4705034 c39e…55e3     0 blk/s   11 tx/s   0 Mgas/s      0+  636 Qed  #4705673   42/50 peers      7 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:16:43  Syncing #4705034 c39e…55e3     0 blk/s    0 tx/s   0 Mgas/s      0+  636 Qed  #4705673   42/50 peers      7 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:16:53  Syncing #4705035 c822…6a33     0 blk/s   19 tx/s   0 Mgas/s      0+  636 Qed  #4705673   43/50 peers      5 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:17:03  Syncing #4705035 c822…6a33     0 blk/s    0 tx/s   0 Mgas/s      0+  636 Qed  #4705673   43/50 peers      7 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:17:13  Syncing #4705036 0159…63bf     0 blk/s   13 tx/s   0 Mgas/s      0+  636 Qed  #4705673   47/50 peers      6 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:17:23  Syncing #4705036 0159…63bf     0 blk/s    0 tx/s   0 Mgas/s      0+  636 Qed  #4705673   49/50 peers      9 MiB chain   18 MiB db   67 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:17:33  Syncing #4705037 40e1…cebc     0 blk/s   13 tx/s   0 Mgas/s      0+  632 Qed  #4705673   49/50 peers      8 MiB chain   18 MiB db   66 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:17:43  Syncing #4705037 40e1…cebc     0 blk/s    0 tx/s   0 Mgas/s      0+  632 Qed  #4705673   50/50 peers      5 MiB chain   18 MiB db   66 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:17:53  Syncing #4705038 950c…3a1b     0 blk/s   11 tx/s   0 Mgas/s      0+  632 Qed  #4705673   48/50 peers      6 MiB chain   18 MiB db   66 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:18:03  Syncing #4705038 950c…3a1b     0 blk/s    0 tx/s   0 Mgas/s      0+  632 Qed  #4705673   48/50 peers      5 MiB chain   18 MiB db   66 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:18:13  Syncing #4705039 6019…beea     0 blk/s    9 tx/s   0 Mgas/s      0+  632 Qed  #4705673   49/50 peers      6 MiB chain   18 MiB db   66 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:18:23  Syncing #4705039 6019…beea     0 blk/s    0 tx/s   0 Mgas/s      0+  632 Qed  #4705673   50/50 peers      7 MiB chain   18 MiB db   66 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:18:33  Syncing #4705040 770a…eb95     0 blk/s   14 tx/s   0 Mgas/s      0+  632 Qed  #4705673   49/50 peers      8 MiB chain   18 MiB db   66 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 10:18:43  Syncing #4705040 770a…eb95     0 blk/s    0 tx/s   0 Mgas/s      0+  632 Qed  #4705673   48/50 peers      6 MiB chain   18 MiB db   66 MiB queue   27 MiB sync  RPC:  0 conn,  0 req/s,   0 µs

Output of parity at 4Gb cache:

2017-12-12 18:32:23  Syncing #4707265 8d23…c30a     0 blk/s   22 tx/s   0 Mgas/s      0+  541 Qed  #4707810   32/50 peers      7 MiB chain   18 MiB db   63 MiB queue   13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 18:32:33  Syncing #4707268 e01b…ffa0     0 blk/s   24 tx/s   2 Mgas/s      0+  541 Qed  #4707810   35/50 peers      6 MiB chain   18 MiB db   63 MiB queue   13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 18:32:43  Syncing #4707268 e01b…ffa0     0 blk/s    0 tx/s   0 Mgas/s      0+  541 Qed  #4707810   36/50 peers      8 MiB chain   18 MiB db   63 MiB queue   13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 18:32:53  Syncing #4707270 aa17…5c58     0 blk/s   22 tx/s   1 Mgas/s      0+  537 Qed  #4707810   38/50 peers      8 MiB chain   18 MiB db   62 MiB queue   13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 18:33:03  Syncing #4707272 20a9…44e7     0 blk/s   19 tx/s   1 Mgas/s      0+  537 Qed  #4707810   39/50 peers      6 MiB chain   18 MiB db   62 MiB queue   13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 18:33:13  Syncing #4707273 2db9…8c5f     0 blk/s   16 tx/s   0 Mgas/s      0+  533 Qed  #4707810   38/50 peers      6 MiB chain   18 MiB db   62 MiB queue   13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 18:33:23  Syncing #4707275 56df…7e73     0 blk/s   34 tx/s   1 Mgas/s      0+  533 Qed  #4707810   42/50 peers      8 MiB chain   18 MiB db   62 MiB queue   13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 18:33:33  Syncing #4707277 8037…22cc     0 blk/s   31 tx/s   1 Mgas/s      0+  529 Qed  #4707810   42/50 peers      6 MiB chain   18 MiB db   61 MiB queue   13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 18:33:43  Syncing #4707277 8037…22cc     0 blk/s    0 tx/s   0 Mgas/s      0+  529 Qed  #4707810   42/50 peers      3 MiB chain   18 MiB db   61 MiB queue   13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 18:33:53  Syncing #4707279 dc6b…a350     0 blk/s   25 tx/s   1 Mgas/s      0+  529 Qed  #4707810   39/50 peers      4 MiB chain   18 MiB db   61 MiB queue   13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs
2017-12-12 18:34:03  Syncing #4707281 8a32…73e7     0 blk/s   25 tx/s   1 Mgas/s      0+  525 Qed  #4707810   22/25 peers      9 MiB chain   18 MiB db   61 MiB queue   13 MiB sync  RPC:  0 conn,  0 req/s,   0 µs

Answer 1

I'm sorry to inform you: There is no hope to synchronize an Ethereum node with any HDD. Well, maybe it is with some enterprise-grade HDDs in some insane Raid-2 style configurations.

But it's more efficient to just get an SSD. SSD is the only way to synchronize the Ethereum chain, unfortunately, regarding I/O. See also this core dev discussion:

• https://github.com/ethereum/pm/issues/28#issuecomment-351614908

One other thing that would be really really nice, but probably quite difficult to achieve, is make it possible to do a sync in an HDD. I have tried to do mainnet syncs in HDDs many times. Fast/warp works fine (after many many retries), but after finishing it an HDD just can't keep up with the network with neither parity nor geth.