Geth disk and memory performance analysis

Question

A week ago I started a synchronization of a geth node with these parameters:

--fast --cache=1024 --rpc --rpcaddr "0.0.0.0" --ws --wsaddr "0.0.0.0" --wsorigins="*"

The result that I have obtained is that after a week running it has spent 69.9049 GB. The fast synchronization needed a total of 39.1735 GB (More info about Ethereum ChainData Size Growth w/FAST Sync) so the normal use of geth for a week has needed 30.7314 GB.

This table shows the evolution of the free space on the disk at specific points:

| DATE          | SIZE(GB)    | DIFF (GB)   |
|---------------|-------------|-------------|
| 27/12/17 8:30 | 93.23209763 |             |
| 28/12/17 8:30 | 46.39630127 | 46.83579636 |
| 29/12/17 8:30 | 41.57238007 | 4.823921204 |
| 30/12/17 8:30 | 36.69866943 | 4.873710632 |
| 31/12/17 8:30 | 31.91919327 | 4.779476166 |
| 1/1/18 8:30   | 27.54598618 | 4.373207092 |
| 2/1/18 8:30   | 25.38702011 | 2.158966064 |
| 3/1/18 8:30   | 23.32711411 | 2.059906006 |

and this is the graph of the evolution:

With these results ~3GB/day (~90GB/month) it seems very difficult to keep a production node synchronized in a VPS without having to perform resynchronizations to clean space or spend a lot of money in disk space

Do you get similar results to me?

Do you know if there is any way to optimize it?

Does geth plan to optimize this behavior?

UPDATE #1:

About the behavior of Geth Péter Szilágyi commented the following on a go-ethereum issue: why my chaindata size up to 400GB?

After it's initial sync, Geth switches to "full sync" where all historical state form that point onward is retained. If you resync, then only the latest state is downloaded. The latest state with the blockchain data is worth about 50GB, but since we don't have state pruning yet, after a sync the data just keeps accumulating.

UPDATE #2:

I'm currently having trouble synchronizing with the most recent block. It seems that the performance has worsened due to the increasing number of transactions in the last days. After one hour running it has only imported 228 blocks:

chain_1  | INFO [01-09|10:05:48] Imported new chain segment               blocks=1 txs=256 mgas=7.973 elapsed=15.996s mgasps=0.498 number=4852965 hash=219fed…eed6cc
chain_1  | INFO [01-09|11:05:54] Imported new chain segment               blocks=1 txs=338 mgas=7.993  elapsed=28.800s mgasps=0.278 number=4853193 hash=44e3bd…1bb02f

Assuming that a new block is generated every 15 seconds approx. Involve that 240 blocks are generated per hour, which means that every hour the node will be 12 blocks behind the network.

I think the problem is due to the lack of ram because it is using the whole system ram plus 3.00 GB of SWAP

As a conclusion it seems that nowadays a VPS with two processors at 2.2GHz and 4GB of RAM does not seem enough to have a node synchronized with geth running inside a docker.

Does anyone experience a similar behavior?

Do you have the same problems?

NOTE: Detailed characteristics of the node:

Digital Ocean droplet:

# cat /proc/cpuinfo | grep "processor\|model name"
processor   : 0
model name  : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
processor   : 1
model name  : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz

# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G        3.7G        112M        108K         85M         26M
Swap:          8.0G        3.7G        4.3G

# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"

# df -klh | grep "^/dev/"
/dev/vda1        58G   18G   41G  31% /
/dev/vda15      105M  3.4M  102M   4% /boot/efi
/dev/sda         99G   83G   12G  89% /mnt/mounted-drive

# docker version
Client:
 Version:   17.12.0-ce
 API version:   1.35
 Go version:    go1.9.2
 Git commit:    c97c6d6
 Built: Wed Dec 27 20:11:19 2017
 OS/Arch:   linux/amd64

Server:
 Engine:
  Version:  17.12.0-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.2
  Git commit:   c97c6d6
  Built:    Wed Dec 27 20:09:53 2017
  OS/Arch:  linux/amd64
  Experimental: false

# docker ps | grep geth
f9cdb05cd876        ethereum/client-go:v1.7.3   "geth --fast --cache…"   2 hours ago         Up 2 hours          0.0.0.0:8545-8546->8545-8546/tcp, 0.0.0.0:30303->30303/tcp, 30303-30304/udp   installpath_chain_1

    command:
      --fast --cache=1024 --rpc --rpcaddr "0.0.0.0" --ws --wsaddr "0.0.0.0" --wsorigins="*"

Answer 1

Redundancy

First thing to consider, is that blockchain is all about decentralisation and stability. But VPS is all about centralisation. Imagine if there is some MEGA VPS, running every server on single core of some imaginary CPU. You, and I, and then everyone store their blockchains and keystores there. Suddenly comes whatever.. thieves, tsunami, powercut - crypto is dead.

Consequence: Your wallet and node could easily be compromised, or shut down.

I/O:

Second thing - is that geth is not so heavily related and bound to CPU, you may easily run that on a couple of low-end cores. Its mostly I/O dependent - fast access to DB storage produces thoughts of RAMdisk, and that native wish for mining "to be the first one" - to be the first to promote your result and actual blocks to most of the nodes - require insane throughput and great latency. You have to:

• deliver as many small packets as you can
• to as many nodes as you can reach
• as fast as possible.

That proposes to use some direct 10G fiber optics, and running node JUST on top of that. Putting it in VM is more like putting you in a prison cell and requesting to run a marathon. Yes, maybe you could...

Consequence: Your wallet will be out of sync, sending wrong outdated data, abusing the internet and others.

Quality

Cheap things are bad, expensives are good, maybe, not always. But there are not only cost efficiency - that is a bit outdated. There is also "thing's quality" and "stability for era" terms. So putting geth in VPS could only describe your's bad intentions, and that wish "to use, but not to pay for it and support it". That is all regarding to ethereum blockchain, community and users, which gave that a value that you use. So let blockchain decide your destiny, according to your approach to it, support and investements in it ;-)

Consequence: Cheap hardware will burn, or someone will pay more for VPS, and they will simply cut you out of a plug, or raise a rent.

---

Conclusion

Buy your OWN GOOD hardware, invest in blockchain; Stay NATIVE and performant; ignore centralized wallets, exchanges, and other massive storing points (related to [massive failures] and single person "access ownership"2! I am doing that for you too! Or quit now and use fiat and central bank, it will be your best VPS..

Answer 2

In short, when node is syncing it executes transactions, that are program code, that needs to read and write data (variables, arrays etc.), when executes. The data is stored at disk, and read\write accesses are random in general (depens of what code transactions include). As bigger blockchain, as more area to perform random reads\writes. I'm not sure, but I think make syncing slower when blockchain growing. So the first problem of syncing is IO. When I was syncing my node I had about 1.2K read and 0.15K write operations per second in average (SSD). I tryed to use HDD first, but syncing was too slow and I read that many people could not sync their nodes at all with HDD. The problem for me was that in production I have only HDD.

I've solved my problem with two steps:

1. I made synchronization with good hardware (big SSD, good CPU)
2. I copied blockchain directory to target host with HDD.
3. Profit

P.S. I can not say anything abot RAM beacause I have 32GB and did not monitor it. You can give more info about RAM usage by geth process and by docker.

Answer 3

My experiences with geth node sync:

Until recently I was able to run a geth full node synchronized to the blockheight of roughly 4.6 million blocks. I was using a 2011 Apple MacBook Pro with 8GB RAM and large enough SSD. I also ran a parity node in parellel, which performs way faster and syncs faster. I prefer parity, but I also need geth for development purposes. It seemed to me that my MacBook Pro was not fast enough to stay in sync with geth, I had given up HDD sync before. The geth chain folder at that time was around 280GB !! I ended up making yet another copy of the full node and currently trying to install it on faster (Apple) hardware.

Import/Export:

I did an export with geth export into a single file. It is around 17GB. But the geth import did not really performed well. I aborted the geth import process at around 2.6 million blocks after a couple of hours. Perhaps, it might be a good idea to export and import multiple smaller portions, say 100.000 blocks at a time, I couldn't find time to figure it out. It's worth a trial yet.

geth node cache:

Assigning more cache to a geth client never really helped me out. The one that is stuck around 2.6 million blocks now, does use around 10GB of RAM and it reserves 531 (!!!) GB of virtual memory. I usually assign 1024 or 2048 MB of cache, but geth will blow to whatever it wants anyway.

Performance of geth binaries

Today, I try to experiment with another compiled binary of geth 1.7.3 stable. (v1.7.3-stable/darwin-amd64/go1.9.2)

mjdillon wrote about it, a couple of days ago: https://github.com/ethereum/go-ethereum/issues/15001

I'm running geth binaries that were downloaded from ethereum against geth binaries that were brewed with brew. I'll probably build from scratch without any package manager or wrap-around, directly from source.