geth “too many open files” on ubuntu

Question

Preamble

This might or might not be a duplicate of Dangling hash node ref: too many open files. In any case, this problem still exists and there should be a solution rather than the - though well explained - workarounds found in the other question's answers. I believe there is a bug in geth. Let's clarify this and file a bug report if there is.

Question

geth fails to run continuously on a fresh Ubuntu 16.04 installation with all updates.

I think I've downloaded the blockchain (~ 25GB in ~/.ethereum). Now when running geth again, it only runs for about a minute and then quits with the error "too many open files". This happens again and again, no matter how often I restart geth. The particular file that the error is reported on changes almost every time. It is not a good option for me to delete the chain data and download it again, because this will take most of a day and I will potentially run into just the same problem again afterwards.

While running geth on one console, on another console we can see that geth in fact has roughly 20k files open:

~$ lsof | awk '{print $1}' | uniq -c | sort -rn | head
  19476 geth
   4296 pool
   3972 JS\x20Hel
   2317 ImgDecode
   1324 mozStorag
    776 pool
    662 DataStora
    537 gmain
    537 gdbus
    537 dconf\x20

This console copy-paste is also helpful:

~$ uname -a
Linux x782x 4.8.0-58-generic #63~16.04.1-Ubuntu SMP Mon Jun 26 18:08:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
~$ du -sh ~/.ethereum
25G /home/x/.ethereum
~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63518
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63518
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
~$ date --utc
Fr 7. Jul 10:29:13 UTC 2017
~$ lsof | awk '{print $1}' | uniq -c | sort -rn | head
   4296 pool
   3972 JS\x20Hel
   2317 ImgDecode
   1324 mozStorag
    776 pool
    662 DataStora
    537 gmain
    537 gdbus
    537 dconf\x20
    537 compiz
~$ date --utc; geth; date --utc
Fr 7. Jul 10:29:41 UTC 2017
INFO [07-07|12:29:42] Starting peer-to-peer node               instance=Geth/v1.6.6-stable-10a45cb5/linux-amd64/go1.8.1
INFO [07-07|12:29:42] Allocated cache and file handles         database=/home/x/.ethereum/geth/chaindata cache=128 handles=1024
INFO [07-07|12:29:42] Initialised chain configuration          config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Metropolis: 9223372036854775807 Engine: ethash}"
INFO [07-07|12:29:42] Disk storage enabled for ethash caches   dir=/home/x/.ethereum/geth/ethash count=3
INFO [07-07|12:29:42] Disk storage enabled for ethash DAGs     dir=/home/x/.ethash               count=2
INFO [07-07|12:29:42] Initialising Ethereum protocol           versions="[63 62]" network=1
INFO [07-07|12:29:42] Loaded most recent local header          number=3987489 hash=229174…bfb77d td=455799413675928101636
INFO [07-07|12:29:42] Loaded most recent local full block      number=0       hash=d4e567…cb8fa3 td=17179869184
INFO [07-07|12:29:42] Loaded most recent local fast block      number=3986648 hash=fbc0e2…e05b36 td=454949784312079869125
INFO [07-07|12:29:43] Starting P2P networking 
INFO [07-07|12:29:45] UDP listener up                          self=enode://775ea4726decc703ca8fb66923bd329bac28bfa30938fbd2b818664419355733e307acc8fac07255d309f4957bc43f5ed49432ddd7d5d4922883b8cf75115243@91.67.186.41:30303
INFO [07-07|12:29:45] RLPx listener up                         self=enode://775ea4726decc703ca8fb66923bd329bac28bfa30938fbd2b818664419355733e307acc8fac07255d309f4957bc43f5ed49432ddd7d5d4922883b8cf75115243@91.67.186.41:30303
INFO [07-07|12:29:45] IPC endpoint opened: /home/x/.ethereum/geth.ipc 
INFO [07-07|12:29:45] Mapped network port                      proto=udp extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
INFO [07-07|12:29:45] Mapped network port                      proto=tcp extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
INFO [07-07|12:29:55] Block synchronisation started 
WARN [07-07|12:30:06] Synchronisation failed, retrying         err="block download canceled (requested)"
INFO [07-07|12:30:15] Imported new state entries               count=1 flushed=0 elapsed=13.398ms processed=1 pending=17 retry=0 duplicate=0 unexpected=0
INFO [07-07|12:30:16] Imported new state entries               count=13 flushed=0 elapsed=165.953ms processed=14 pending=225 retry=2 duplicate=0 unexpected=0
INFO [07-07|12:30:17] Imported new block headers               count=0  elapsed=6.823ms   number=3987032 hash=2dee86…a1a715 ignored=384
INFO [07-07|12:30:17] Imported new block headers               count=0  elapsed=2.924ms   number=3987224 hash=0d338a…023d52 ignored=192
INFO [07-07|12:30:17] Imported new state entries               count=93 flushed=0 elapsed=1.197s    processed=107 pending=1713 retry=0 duplicate=0 unexpected=0
INFO [07-07|12:30:17] Imported new block receipts              count=0  elapsed=6.386ms   number=3986650 hash=733728…e408ba ignored=2
INFO [07-07|12:30:17] Imported new block headers               count=0  elapsed=2.565ms   number=3987416 hash=f20661…e10ded ignored=192
INFO [07-07|12:30:18] Imported new block receipts              count=0  elapsed=21.190ms  number=3986660 hash=7e7e77…c8c88e ignored=10
INFO [07-07|12:30:18] Imported new block receipts              count=10 elapsed=65.403ms  number=3986673 hash=fd2417…99268a ignored=3
INFO [07-07|12:30:18] Imported new block headers               count=60 elapsed=704.929ms number=3987549 hash=c159b6…8524dd ignored=73
INFO [07-07|12:30:19] Imported new state entries               count=69 flushed=0 elapsed=997.560ms processed=176 pending=2636 retry=2 duplicate=0 unexpected=0
INFO [07-07|12:30:19] Imported new block receipts              count=3  elapsed=17.730ms  number=3986676 hash=4c9170…1d3733 ignored=0
INFO [07-07|12:30:19] Imported new block receipts              count=21 elapsed=72.690ms  number=3986697 hash=5ee4fa…4ef760 ignored=0
INFO [07-07|12:30:20] Imported new block receipts              count=39 elapsed=105.088ms number=3986736 hash=ed392f…b725de ignored=0
INFO [07-07|12:30:21] Imported new block receipts              count=34 elapsed=99.869ms  number=3986770 hash=94f9fc…5776aa ignored=0
INFO [07-07|12:30:21] Imported new block receipts              count=4  elapsed=29.266ms  number=3986774 hash=7d4b4a…99905c ignored=0
INFO [07-07|12:30:22] Imported new state entries               count=207 flushed=0 elapsed=2.897s    processed=383 pending=5207 retry=0 duplicate=0 unexpected=0
INFO [07-07|12:30:24] Imported new state entries               count=153 flushed=0 elapsed=2.190s    processed=536 pending=5863 retry=3 duplicate=0 unexpected=0
INFO [07-07|12:30:24] Imported new state entries               count=17  flushed=0 elapsed=242.006ms processed=553 pending=5931 retry=73 duplicate=0 unexpected=0
INFO [07-07|12:30:25] Imported new state entries               count=93  flushed=47 elapsed=321.475ms processed=646 pending=5982 retry=2  duplicate=0 unexpected=0
INFO [07-07|12:30:25] Imported new state entries               count=8   flushed=2  elapsed=24.063ms  processed=654 pending=5998 retry=73 duplicate=0 unexpected=0
INFO [07-07|12:30:26] Imported new state entries               count=1   flushed=0  elapsed=10.696ms  processed=655 pending=5999 retry=1  duplicate=0 unexpected=0
INFO [07-07|12:30:26] Imported new block receipts              count=62  elapsed=251.859ms number=3986836 hash=4fe030…9e78c3 ignored=0
INFO [07-07|12:30:27] Imported new state entries               count=272 flushed=144 elapsed=702.709ms processed=927 pending=6259 retry=0  duplicate=0 unexpected=0
INFO [07-07|12:30:27] Imported new state entries               count=166 flushed=10  elapsed=545.969ms processed=1093 pending=6676 retry=1  duplicate=0 unexpected=0
INFO [07-07|12:30:27] Imported new state entries               count=7   flushed=4   elapsed=5.219ms   processed=1100 pending=6676 retry=2  duplicate=0 unexpected=0
INFO [07-07|12:30:28] Imported new state entries               count=9   flushed=12  elapsed=4.879ms   processed=1109 pending=6665 retry=5  duplicate=0 unexpected=0
INFO [07-07|12:30:29] Imported new state entries               count=11  flushed=11  elapsed=9.919ms   processed=1120 pending=6658 retry=73 duplicate=0 unexpected=0
INFO [07-07|12:30:29] Imported new state entries               count=63  flushed=67  elapsed=25.589ms  processed=1183 pending=6603 retry=16 duplicate=0 unexpected=0
INFO [07-07|12:30:31] Imported new state entries               count=4   flushed=8   elapsed=3.359ms   processed=1187 pending=6595 retry=76 duplicate=0 unexpected=0
INFO [07-07|12:30:31] Imported new state entries               count=32  flushed=17  elapsed=61.327ms  processed=1219 pending=6623 retry=135 duplicate=0 unexpected=0
INFO [07-07|12:30:32] Imported new state entries               count=104 flushed=16  elapsed=324.452ms processed=1323 pending=6860 retry=40  duplicate=0 unexpected=0
INFO [07-07|12:30:33] Imported new state entries               count=220 flushed=246 elapsed=297.035ms processed=1543 pending=6686 retry=0   duplicate=0 unexpected=0
INFO [07-07|12:30:35] Imported new block receipts              count=34  elapsed=146.697ms number=3986870 hash=c48f87…7d0189 ignored=0
INFO [07-07|12:30:35] Imported new state entries               count=152 flushed=150 elapsed=87.589ms  processed=1695 pending=6591 retry=115 duplicate=0 unexpected=0
CRIT [07-07|12:30:38] Failed to store receipts                 err="open /home/x/.ethereum/geth/chaindata/226710.ldb: too many open files"
Fr 7. Jul 10:30:38 UTC 2017
~$

I think that 20k open files is wrong, because it doesn't work on a fresh Ubuntu 16.04 installation with all updates. Is there a special reason it has to be this many files, or is this a bug?