Some background:
I setup private Ethereum nodes on my local docker system and built some modules that use the web3 Ethereum client to exercise these nodes. I've been testing on it for the last 3-4 months with no issues. (atleast with Ethereum)
Recently, we decided to deploy our Ethereum nodes on an AWS platform instead of using local docker instances. I setup an EC2 instance with 2 CPU's and deployed the Ethereum nodes on it.
Initially, once the nodes are up, it looks fine.I create an account and mine gas. I am able to invoke add/get requests from my code and add/fetch data from the Ethereum backend. In other words, the functionality works as expected.
However, I noticed that after 2-3 hours, the Ethereum nodes get into a bad state where every transaction to add data to ethereum (I am using a send() call to do that) gives the following error:
(node:52) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 4)
(node:52) UnhandledPromiseRejectionWarning: Error: Returned error: insufficient funds for gas * price + value
at Object.ErrorResponse (/node_modules/web3-core-helpers/src/errors.js:29:16)
at /node_modules/web3-core-requestmanager/src/index.js:140:36
at XMLHttpRequest.request.onreadystatechange (/node_modules/web3-providers-http/src/index.js:91:13)
at XMLHttpRequestEventTarget.dispatchEvent (/node_modules/xhr2-cookies/dist/xml-http-request-event-target.js:34:22)
at XMLHttpRequest._setReadyState (/node_modules/xhr2-cookies/dist/xml-http-request.js:208:14)
at XMLHttpRequest._onHttpResponseEnd (/node_modules/xhr2-cookies/dist/xml-http-request.js:318:14)
at IncomingMessage.<anonymous> (/node_modules/xhr2-cookies/dist/xml-http-request.js:289:61)
at emitNone (events.js:111:20)
at IncomingMessage.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1064:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
I did:
eth.getBlock("latest").gasLimit
and it returns a value that is sufficiently high for the transaction. (I mean it worked for similar transactions before so this has to work. The gas limit hasnt gone down either.
Ultimately, the Ethereum node crashes with the following msg:
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0xf7eb77, 0x16)
/usr/lib/go-1.10/src/runtime/panic.go:616 +0x81
runtime.sysMap(0xc4ea190000, 0x10000000, 0x0, 0x1bb99b8)
/usr/lib/go-1.10/src/runtime/mem_linux.go:216 +0x20a
runtime.(*mheap).sysAlloc(0x1b9ffe0, 0x10000000, 0x7f9d1430cbb0)
/usr/lib/go-1.10/src/runtime/malloc.go:470 +0xd4
runtime.(*mheap).grow(0x1b9ffe0, 0x8000, 0x0)
/usr/lib/go-1.10/src/runtime/mheap.go:907 +0x60
runtime.(*mheap).allocSpanLocked(0x1b9ffe0, 0x8000, 0x1bb99c8, 0x7f9d0cd03d88)
/usr/lib/go-1.10/src/runtime/mheap.go:820 +0x301
runtime.(*mheap).alloc_m(0x1b9ffe0, 0x8000, 0x420101, 0xc41f92dcff)
/usr/lib/go-1.10/src/runtime/mheap.go:686 +0x118
runtime.(*mheap).alloc.func1()
/usr/lib/go-1.10/src/runtime/mheap.go:753 +0x4d
runtime.(*mheap).alloc(0x1b9ffe0, 0x8000, 0x7f9d0c010101, 0x429afc)
/usr/lib/go-1.10/src/runtime/mheap.go:752 +0x8a
runtime.largeAlloc(0x10000000, 0x460101, 0x7f9d1629c6c8)
/usr/lib/go-1.10/src/runtime/malloc.go:826 +0x94
runtime.mallocgc.func1()
/usr/lib/go-1.10/src/runtime/malloc.go:721 +0x46
runtime.systemstack(0x0)
/usr/lib/go-1.10/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
/usr/lib/go-1.10/src/runtime/proc.go:1175
However, when I check the memory usage, it seems to be good:
# free
total used free shared buff/cache available
Mem: 3800968 178628 3220016 5664 402324 3401868
Swap: 0 0 0
Even docker stats also reports sufficient memory available.
However, I see that the CPU % is always hovering around close to 100%. Could that be an issue?
a few things I tried:
I checked the memory and CPU % on the dev system I had (using just dockers on a local Linux machine) That also had around 95% CPU most of the time. But that system had 4 CPU's.
I had a suspicion that the process of mining is using up too much CPU or memory and therefore updated the gasLimit on the genesis file and the --targetgaslimit to a much lower value.(I had set targetgaslimit to around 900000000 before and the genesis file gasLimit to 4500000 - got both down to 1500000) The lower gas limit is enough for what I do.
I expected to see the mining stop after the targetgaslimit is reached. however, I see messages going down on the Geth console after that also. I see this even if I explicitly do: miner.stop()
Before, the crash I also saw messages like:
Failed account unlock attempt
This is surprising since I unlocked the account for an infinite time.
A few questions: 1. I am trying to find out how to increase the initial gas value in the system. Mine is a private ethereum network and hence am fine if there is a constant value of gas. No need to mine. Is this a good idea? If not, what are the challenges?
To do #1, I tried to edit the geth code to change the pre-defined value of gas limit. However, the new value doesnt get picked up. I changed the variable MinGasLimit to a value of 1500000 (baseline value in the repository is 5000)
Will increasing CPU/memory help? I feel it will only delay the issue and not solve it.
any other thoughts/inputs?
Thanks in advance!
