0

Some background:

I setup private Ethereum nodes on my local docker system and built some modules that use the web3 Ethereum client to exercise these nodes. I've been testing on it for the last 3-4 months with no issues. (atleast with Ethereum)

Recently, we decided to deploy our Ethereum nodes on an AWS platform instead of using local docker instances. I setup an EC2 instance with 2 CPU's and deployed the Ethereum nodes on it.

Initially, once the nodes are up, it looks fine.I create an account and mine gas. I am able to invoke add/get requests from my code and add/fetch data from the Ethereum backend. In other words, the functionality works as expected.

However, I noticed that after 2-3 hours, the Ethereum nodes get into a bad state where every transaction to add data to ethereum (I am using a send() call to do that) gives the following error:

(node:52) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 4)
(node:52) UnhandledPromiseRejectionWarning: Error: Returned error: insufficient funds for gas * price + value
    at Object.ErrorResponse (/node_modules/web3-core-helpers/src/errors.js:29:16)
    at /node_modules/web3-core-requestmanager/src/index.js:140:36
    at XMLHttpRequest.request.onreadystatechange (/node_modules/web3-providers-http/src/index.js:91:13)
    at XMLHttpRequestEventTarget.dispatchEvent (/node_modules/xhr2-cookies/dist/xml-http-request-event-target.js:34:22)
    at XMLHttpRequest._setReadyState (/node_modules/xhr2-cookies/dist/xml-http-request.js:208:14)
    at XMLHttpRequest._onHttpResponseEnd (/node_modules/xhr2-cookies/dist/xml-http-request.js:318:14)
    at IncomingMessage.<anonymous> (/node_modules/xhr2-cookies/dist/xml-http-request.js:289:61)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

I did:

eth.getBlock("latest").gasLimit

and it returns a value that is sufficiently high for the transaction. (I mean it worked for similar transactions before so this has to work. The gas limit hasnt gone down either.

Ultimately, the Ethereum node crashes with the following msg:

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xf7eb77, 0x16)
        /usr/lib/go-1.10/src/runtime/panic.go:616 +0x81
runtime.sysMap(0xc4ea190000, 0x10000000, 0x0, 0x1bb99b8)
        /usr/lib/go-1.10/src/runtime/mem_linux.go:216 +0x20a
runtime.(*mheap).sysAlloc(0x1b9ffe0, 0x10000000, 0x7f9d1430cbb0)
        /usr/lib/go-1.10/src/runtime/malloc.go:470 +0xd4
runtime.(*mheap).grow(0x1b9ffe0, 0x8000, 0x0)
        /usr/lib/go-1.10/src/runtime/mheap.go:907 +0x60
runtime.(*mheap).allocSpanLocked(0x1b9ffe0, 0x8000, 0x1bb99c8, 0x7f9d0cd03d88)
        /usr/lib/go-1.10/src/runtime/mheap.go:820 +0x301
runtime.(*mheap).alloc_m(0x1b9ffe0, 0x8000, 0x420101, 0xc41f92dcff)
        /usr/lib/go-1.10/src/runtime/mheap.go:686 +0x118
runtime.(*mheap).alloc.func1()
        /usr/lib/go-1.10/src/runtime/mheap.go:753 +0x4d
runtime.(*mheap).alloc(0x1b9ffe0, 0x8000, 0x7f9d0c010101, 0x429afc)
        /usr/lib/go-1.10/src/runtime/mheap.go:752 +0x8a
runtime.largeAlloc(0x10000000, 0x460101, 0x7f9d1629c6c8)
        /usr/lib/go-1.10/src/runtime/malloc.go:826 +0x94
runtime.mallocgc.func1()
        /usr/lib/go-1.10/src/runtime/malloc.go:721 +0x46
runtime.systemstack(0x0)
 /usr/lib/go-1.10/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
        /usr/lib/go-1.10/src/runtime/proc.go:1175

However, when I check the memory usage, it seems to be good:

# free
              total        used        free      shared  buff/cache   available
Mem:        3800968      178628     3220016        5664      402324     3401868
Swap:             0           0           0

Even docker stats also reports sufficient memory available.

However, I see that the CPU % is always hovering around close to 100%. Could that be an issue?

a few things I tried:

  1. I checked the memory and CPU % on the dev system I had (using just dockers on a local Linux machine) That also had around 95% CPU most of the time. But that system had 4 CPU's.

  2. I had a suspicion that the process of mining is using up too much CPU or memory and therefore updated the gasLimit on the genesis file and the --targetgaslimit to a much lower value.(I had set targetgaslimit to around 900000000 before and the genesis file gasLimit to 4500000 - got both down to 1500000) The lower gas limit is enough for what I do.

I expected to see the mining stop after the targetgaslimit is reached. however, I see messages going down on the Geth console after that also. I see this even if I explicitly do: miner.stop()

  1. Before, the crash I also saw messages like:

    Failed account unlock attempt

This is surprising since I unlocked the account for an infinite time.

A few questions: 1. I am trying to find out how to increase the initial gas value in the system. Mine is a private ethereum network and hence am fine if there is a constant value of gas. No need to mine. Is this a good idea? If not, what are the challenges?

  1. To do #1, I tried to edit the geth code to change the pre-defined value of gas limit. However, the new value doesnt get picked up. I changed the variable MinGasLimit to a value of 1500000 (baseline value in the repository is 5000)

  2. Will increasing CPU/memory help? I feel it will only delay the issue and not solve it.

  3. any other thoughts/inputs?

Thanks in advance!

1 Answer 1

1

A few thoughts:

  • Is your clock absolutely synced with the time servers? I would think so otherwise you did not get this far.
  • Disk space. This all takes alot of space. Did you check?
  • There have been recent updates to the GETH and Parity codebases to reflect changes that came about from the staged Constantinople upgrade coming up. Have you updated to this? You might end up syncing to the point where the new blocks are coming in and getting this problem, perhaps?
2
  • I am sure about #1. Disk space - yes, that is something I ensured is good by checking df. I have tried running this on an EC2 instance with more CPU's and memory and will see how it goes.
    – Omi
    Feb 11, 2019 at 4:26
  • I ended up using an EC2 instance with more memory and more CPU's. That solved the issue. The system has been running for almost 2 days with no issues. Earlier no matter what tweaks I did, it would crash in atmost 3-4 hours
    – Omi
    Feb 12, 2019 at 7:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.