0

My Eth2 validator occasionally encounters a MissingDepositsException.

For the second time now my validator missed a block proposal with the exception: tech.pegasys.teku.validator.coordinator.MissingDepositsException

The setup:

Chain Client Version
Ethereum 1 Geth 1.10.3
Ethereum 2 / Beaconchain Teku 21.4.1

This exception occurs only sometimes. Two and a half months ago, I encountered this error the first time. Then I had 9 block proposals without issue and now I got the error again.

This exception only affects block proposals, attestation performance stayed at 100% throughout the missed proposal.

Any hints as to what might be causing this exception to occur would be greatly apprechiated!

Logs of the exception:

tech.pegasys.teku.validator.coordinator.MissingDepositsException: Unable to create block because ETH1 deposits are not available. Missing deposits 124010 to 165920
        at tech.pegasys.teku.validator.coordinator.MissingDepositsException.missingRange(MissingDepositsException.java:31) ~[teku-validator-coordinator-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.validator.coordinator.DepositProvider.checkRequiredDepositsAvailable(DepositProvider.java:159) ~[teku-validator-coordinator-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.validator.coordinator.DepositProvider.getDeposits(DepositProvider.java:138) ~[teku-validator-coordinator-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.validator.coordinator.BlockFactory.createUnsignedBlock(BlockFactory.java:117) ~[teku-validator-coordinator-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.validator.coordinator.ValidatorApiHandler.createBlock(ValidatorApiHandler.java:269) ~[teku-validator-coordinator-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.validator.coordinator.ValidatorApiHandler.lambda$createUnsignedBlock$11(ValidatorApiHandler.java:252) ~[teku-validator-coordinator-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$thenApplyChecked$28(SafeFuture.java:410) ~[teku-infrastructure-async-21.4.1.jar:21.4.1]
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
        at tech.pegasys.teku.dataproviders.generators.CachingTaskQueue.lambda$completePendingTask$9(CachingTaskQueue.java:204) ~[teku-ethereum-dataproviders-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.fromRunnable(SafeFuture.java:154) ~[teku-infrastructure-async-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.infrastructure.async.AsyncRunner.lambda$runAsync$0(AsyncRunner.java:23) ~[teku-infrastructure-async-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:73) ~[teku-infrastructure-async-21.4.1.jar:21.4.1]
        at tech.pegasys.teku.infrastructure.async.ScheduledExecutorAsyncRunner.lambda$createRunnableForAction$1(ScheduledExecutorAsyncRunner.java:119) ~[teku-infrastructure-async-21.4.1.jar:21.4.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Improve this question

1 Answer 1

Reset to default
1

This error suggests that at some point the eth1 node has lost or failed to provide the logs for a deposit event when queried. That's pretty unusual for Geth but might be possible if it was still syncing.

Missing deposits is only an issue when creating a block if there are pending deposits that haven't already been included. That's quite a small number of blocks so you've been very unlucky to hit it twice.

It's very unlikely this error will be able to resolve itself at this point because Teku doesn't query backwards in the chain to find old deposits since the APIs should have provided those deposits initially and querying large ranges of the chain is quite expensive.

I'd suggest upgrading to the latest Teku (currently 21.5.0) and resyncing the beacon node to force it to scan for deposit events from the start of the chain again - if Geth is still not returning them the latest Teku should produce a very clear error as part of that resync. Fortunately the resync can be done in just a couple of minutes using the snapshot sync feature. And as an additional benefit you'll migrate the database over to LevelDB which uses less memory. Just follow the instructions at https://docs.teku.consensys.net/en/latest/HowTo/Migrate-Database/

Improve this answer
2
  • Thank you, I'll try the update and resync. One more thing, against the recommendation I use an HDD instead of an SSD (but both Geth and Teku are synced), could that have affected this issue? I don't think so, except for after a reboot Geth might have been out of sync for a minute longer than usual.
    – Felix Dolderer
    Jun 7, 2021 at 8:09
  • I doubt the HDD is an issue, but its hard to say exactly what went wrong. We've added some extra checks in Teku to detect when bad deposit information is received earlier so it can stop at the time instead of winding up in this kind of situation, but I suspect you originally hit the problem before that was in.
    – ajsutton
    Jun 8, 2021 at 9:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.