Eth2 Client Teku missing proposals: MissingDepositsException

Question

My Eth2 validator occasionally encounters a MissingDepositsException.

For the second time now my validator missed a block proposal with the exception: tech.pegasys.teku.validator.coordinator.MissingDepositsException

The setup:

Chain	Client	Version
Ethereum 1	Geth	1.10.3
Ethereum 2 / Beaconchain	Teku	21.4.1

This exception occurs only sometimes. Two and a half months ago, I encountered this error the first time. Then I had 9 block proposals without issue and now I got the error again.

This exception only affects block proposals, attestation performance stayed at 100% throughout the missed proposal.

Any hints as to what might be causing this exception to occur would be greatly apprechiated!

---

Logs of the exception:

tech.pegasys.teku.validator.coordinator.MissingDepositsException: Unable to create block because ETH1 deposits are not available. Missing deposits 124010 to 165920
        at tech.pegasys.teku.validator.coordinator.MissingDepositsException.missingRange(MissingDepositsException.java:31) ~[teku-validator-coordinator-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.validator.coordinator.DepositProvider.checkRequiredDepositsAvailable(DepositProvider.java:159) ~[teku-validator-coordinator-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.validator.coordinator.DepositProvider.getDeposits(DepositProvider.java:138) ~[teku-validator-coordinator-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.validator.coordinator.BlockFactory.createUnsignedBlock(BlockFactory.java:117) ~[teku-validator-coordinator-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.validator.coordinator.ValidatorApiHandler.createBlock(ValidatorApiHandler.java:269) ~[teku-validator-coordinator-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.validator.coordinator.ValidatorApiHandler.lambda$createUnsignedBlock$11(ValidatorApiHandler.java:252) ~[teku-validator-coordinator-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$thenApplyChecked$28(SafeFuture.java:410) ~[teku-infrastructure-async-21.4.1.jar:21.4.1]
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
        at tech.pegasys.teku.dataproviders.generators.CachingTaskQueue.lambda$completePendingTask$9(CachingTaskQueue.java:204) ~[teku-ethereum-dataproviders-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.fromRunnable(SafeFuture.java:154) ~[teku-infrastructure-async-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.infrastructure.async.AsyncRunner.lambda$runAsync$0(AsyncRunner.java:23) ~[teku-infrastructure-async-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:73) ~[teku-infrastructure-async-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.infrastructure.async.ScheduledExecutorAsyncRunner.lambda$createRunnableForAction$1(ScheduledExecutorAsyncRunner.java:119) ~[teku-infrastructure-async-21.4.1.jar:21.4.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]

Answer 1

This error suggests that at some point the eth1 node has lost or failed to provide the logs for a deposit event when queried. That's pretty unusual for Geth but might be possible if it was still syncing.

Missing deposits is only an issue when creating a block if there are pending deposits that haven't already been included. That's quite a small number of blocks so you've been very unlucky to hit it twice.

It's very unlikely this error will be able to resolve itself at this point because Teku doesn't query backwards in the chain to find old deposits since the APIs should have provided those deposits initially and querying large ranges of the chain is quite expensive.

I'd suggest upgrading to the latest Teku (currently 21.5.0) and resyncing the beacon node to force it to scan for deposit events from the start of the chain again - if Geth is still not returning them the latest Teku should produce a very clear error as part of that resync. Fortunately the resync can be done in just a couple of minutes using the snapshot sync feature. And as an additional benefit you'll migrate the database over to LevelDB which uses less memory. Just follow the instructions at https://docs.teku.consensys.net/en/latest/HowTo/Migrate-Database/