Preamble
This might or might not be a duplicate of Dangling hash node ref: too many open files. In any case, this problem still exists and there should be a solution rather than the - though well explained - workarounds found in the other question's answers. I believe there is a bug in geth. Let's clarify this and file a bug report if there is.
Question
geth fails to run continuously on a fresh Ubuntu 16.04 installation with all updates.
I think I've downloaded the blockchain (~ 25GB in ~/.ethereum). Now when running geth again, it only runs for about a minute and then quits with the error "too many open files". This happens again and again, no matter how often I restart geth. The particular file that the error is reported on changes almost every time. It is not a good option for me to delete the chain data and download it again, because this will take most of a day and I will potentially run into just the same problem again afterwards.
While running geth on one console, on another console we can see that geth in fact has roughly 20k files open:
~$ lsof | awk '{print $1}' | uniq -c | sort -rn | head
19476 geth
4296 pool
3972 JS\x20Hel
2317 ImgDecode
1324 mozStorag
776 pool
662 DataStora
537 gmain
537 gdbus
537 dconf\x20
This console copy-paste is also helpful:
~$ uname -a
Linux x782x 4.8.0-58-generic #63~16.04.1-Ubuntu SMP Mon Jun 26 18:08:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
~$ du -sh ~/.ethereum
25G /home/x/.ethereum
~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63518
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 63518
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
~$ date --utc
Fr 7. Jul 10:29:13 UTC 2017
~$ lsof | awk '{print $1}' | uniq -c | sort -rn | head
4296 pool
3972 JS\x20Hel
2317 ImgDecode
1324 mozStorag
776 pool
662 DataStora
537 gmain
537 gdbus
537 dconf\x20
537 compiz
~$ date --utc; geth; date --utc
Fr 7. Jul 10:29:41 UTC 2017
INFO [07-07|12:29:42] Starting peer-to-peer node instance=Geth/v1.6.6-stable-10a45cb5/linux-amd64/go1.8.1
INFO [07-07|12:29:42] Allocated cache and file handles database=/home/x/.ethereum/geth/chaindata cache=128 handles=1024
INFO [07-07|12:29:42] Initialised chain configuration config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis: 9223372036854775807 Engine: ethash}"
INFO [07-07|12:29:42] Disk storage enabled for ethash caches dir=/home/x/.ethereum/geth/ethash count=3
INFO [07-07|12:29:42] Disk storage enabled for ethash DAGs dir=/home/x/.ethash count=2
INFO [07-07|12:29:42] Initialising Ethereum protocol versions="[63 62]" network=1
INFO [07-07|12:29:42] Loaded most recent local header number=3987489 hash=229174…bfb77d td=455799413675928101636
INFO [07-07|12:29:42] Loaded most recent local full block number=0 hash=d4e567…cb8fa3 td=17179869184
INFO [07-07|12:29:42] Loaded most recent local fast block number=3986648 hash=fbc0e2…e05b36 td=454949784312079869125
INFO [07-07|12:29:43] Starting P2P networking
INFO [07-07|12:29:45] UDP listener up self=enode://775ea4726decc703ca8fb66923bd329bac28bfa30938fbd2b818664419355733e307acc8fac07255d309f4957bc43f5ed49432ddd7d5d4922883b8cf75115243@91.67.186.41:30303
INFO [07-07|12:29:45] RLPx listener up self=enode://775ea4726decc703ca8fb66923bd329bac28bfa30938fbd2b818664419355733e307acc8fac07255d309f4957bc43f5ed49432ddd7d5d4922883b8cf75115243@91.67.186.41:30303
INFO [07-07|12:29:45] IPC endpoint opened: /home/x/.ethereum/geth.ipc
INFO [07-07|12:29:45] Mapped network port proto=udp extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
INFO [07-07|12:29:45] Mapped network port proto=tcp extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
INFO [07-07|12:29:55] Block synchronisation started
WARN [07-07|12:30:06] Synchronisation failed, retrying err="block download canceled (requested)"
INFO [07-07|12:30:15] Imported new state entries count=1 flushed=0 elapsed=13.398ms processed=1 pending=17 retry=0 duplicate=0 unexpected=0
INFO [07-07|12:30:16] Imported new state entries count=13 flushed=0 elapsed=165.953ms processed=14 pending=225 retry=2 duplicate=0 unexpected=0
INFO [07-07|12:30:17] Imported new block headers count=0 elapsed=6.823ms number=3987032 hash=2dee86…a1a715 ignored=384
INFO [07-07|12:30:17] Imported new block headers count=0 elapsed=2.924ms number=3987224 hash=0d338a…023d52 ignored=192
INFO [07-07|12:30:17] Imported new state entries count=93 flushed=0 elapsed=1.197s processed=107 pending=1713 retry=0 duplicate=0 unexpected=0
INFO [07-07|12:30:17] Imported new block receipts count=0 elapsed=6.386ms number=3986650 hash=733728…e408ba ignored=2
INFO [07-07|12:30:17] Imported new block headers count=0 elapsed=2.565ms number=3987416 hash=f20661…e10ded ignored=192
INFO [07-07|12:30:18] Imported new block receipts count=0 elapsed=21.190ms number=3986660 hash=7e7e77…c8c88e ignored=10
INFO [07-07|12:30:18] Imported new block receipts count=10 elapsed=65.403ms number=3986673 hash=fd2417…99268a ignored=3
INFO [07-07|12:30:18] Imported new block headers count=60 elapsed=704.929ms number=3987549 hash=c159b6…8524dd ignored=73
INFO [07-07|12:30:19] Imported new state entries count=69 flushed=0 elapsed=997.560ms processed=176 pending=2636 retry=2 duplicate=0 unexpected=0
INFO [07-07|12:30:19] Imported new block receipts count=3 elapsed=17.730ms number=3986676 hash=4c9170…1d3733 ignored=0
INFO [07-07|12:30:19] Imported new block receipts count=21 elapsed=72.690ms number=3986697 hash=5ee4fa…4ef760 ignored=0
INFO [07-07|12:30:20] Imported new block receipts count=39 elapsed=105.088ms number=3986736 hash=ed392f…b725de ignored=0
INFO [07-07|12:30:21] Imported new block receipts count=34 elapsed=99.869ms number=3986770 hash=94f9fc…5776aa ignored=0
INFO [07-07|12:30:21] Imported new block receipts count=4 elapsed=29.266ms number=3986774 hash=7d4b4a…99905c ignored=0
INFO [07-07|12:30:22] Imported new state entries count=207 flushed=0 elapsed=2.897s processed=383 pending=5207 retry=0 duplicate=0 unexpected=0
INFO [07-07|12:30:24] Imported new state entries count=153 flushed=0 elapsed=2.190s processed=536 pending=5863 retry=3 duplicate=0 unexpected=0
INFO [07-07|12:30:24] Imported new state entries count=17 flushed=0 elapsed=242.006ms processed=553 pending=5931 retry=73 duplicate=0 unexpected=0
INFO [07-07|12:30:25] Imported new state entries count=93 flushed=47 elapsed=321.475ms processed=646 pending=5982 retry=2 duplicate=0 unexpected=0
INFO [07-07|12:30:25] Imported new state entries count=8 flushed=2 elapsed=24.063ms processed=654 pending=5998 retry=73 duplicate=0 unexpected=0
INFO [07-07|12:30:26] Imported new state entries count=1 flushed=0 elapsed=10.696ms processed=655 pending=5999 retry=1 duplicate=0 unexpected=0
INFO [07-07|12:30:26] Imported new block receipts count=62 elapsed=251.859ms number=3986836 hash=4fe030…9e78c3 ignored=0
INFO [07-07|12:30:27] Imported new state entries count=272 flushed=144 elapsed=702.709ms processed=927 pending=6259 retry=0 duplicate=0 unexpected=0
INFO [07-07|12:30:27] Imported new state entries count=166 flushed=10 elapsed=545.969ms processed=1093 pending=6676 retry=1 duplicate=0 unexpected=0
INFO [07-07|12:30:27] Imported new state entries count=7 flushed=4 elapsed=5.219ms processed=1100 pending=6676 retry=2 duplicate=0 unexpected=0
INFO [07-07|12:30:28] Imported new state entries count=9 flushed=12 elapsed=4.879ms processed=1109 pending=6665 retry=5 duplicate=0 unexpected=0
INFO [07-07|12:30:29] Imported new state entries count=11 flushed=11 elapsed=9.919ms processed=1120 pending=6658 retry=73 duplicate=0 unexpected=0
INFO [07-07|12:30:29] Imported new state entries count=63 flushed=67 elapsed=25.589ms processed=1183 pending=6603 retry=16 duplicate=0 unexpected=0
INFO [07-07|12:30:31] Imported new state entries count=4 flushed=8 elapsed=3.359ms processed=1187 pending=6595 retry=76 duplicate=0 unexpected=0
INFO [07-07|12:30:31] Imported new state entries count=32 flushed=17 elapsed=61.327ms processed=1219 pending=6623 retry=135 duplicate=0 unexpected=0
INFO [07-07|12:30:32] Imported new state entries count=104 flushed=16 elapsed=324.452ms processed=1323 pending=6860 retry=40 duplicate=0 unexpected=0
INFO [07-07|12:30:33] Imported new state entries count=220 flushed=246 elapsed=297.035ms processed=1543 pending=6686 retry=0 duplicate=0 unexpected=0
INFO [07-07|12:30:35] Imported new block receipts count=34 elapsed=146.697ms number=3986870 hash=c48f87…7d0189 ignored=0
INFO [07-07|12:30:35] Imported new state entries count=152 flushed=150 elapsed=87.589ms processed=1695 pending=6591 retry=115 duplicate=0 unexpected=0
CRIT [07-07|12:30:38] Failed to store receipts err="open /home/x/.ethereum/geth/chaindata/226710.ldb: too many open files"
Fr 7. Jul 10:30:38 UTC 2017
~$
I think that 20k open files is wrong, because it doesn't work on a fresh Ubuntu 16.04 installation with all updates. Is there a special reason it has to be this many files, or is this a bug?
ulimit
modifications and I've also tweakedgeth
, but no joy. Hopefully someone knowledgeable will chime in. In the meantime I'll run inlight
mode. – youri Aug 11 '18 at 12:58