What caused the forks in the Schlesi testnet?

Question

The Schlesi testnet was the first multiclient testnet for ETH2 clients. Lighthouse, Prysm, Teku were in consensus before it forked.

What were the causes of the forks? Is there a technical post mortem report on it?

Source

Answer 1

What were the causes of the forks?

There were some issues relating to rewards/penalties calculation. Both Lighthouse and Prysm were incorrectly computing the validator balances.

Interestingly, Lighthouse, Prysm and ZCLI (reference spec implementation in Go) all computed different values, the result of multiple unrelated issues. Teku stayed faithful to the specification.

Client Fixes

• Lighthouse
  • https://github.com/sigp/lighthouse/pull/1166
• ZCLI
  • https://github.com/protolambda/zrnt/commit/ee4c8af2fb45301fdff6c7c72638e7ee677ec8fe
• Prysm
  • https://github.com/prysmaticlabs/prysm/pull/5909
  • https://github.com/prysmaticlabs/prysm/pull/5903

Steps taken to resolve:

• All clients have fixed the bug.
• The latest spec v0.12.0 has refactored balance calculations so they're easier to test. More tests have been added.
• The offending blocks and states have been added to beacon-fuzz (eth2 fuzzer) to help protect against regression and find related bugs.

Side note: this fork highlighted an interesting balance calculation quirk in the spec when the network isn't reaching finality. The specification was changed here.