What were the causes of the forks?

There were some issues relating to rewards/penalties calculation. Both Lighthouse and Prysm were incorrectly computing the validator balances.

Interestingly, Lighthouse, Prysm and ZCLI (reference spec implementation in Go) all computed different values, the result of multiple unrelated issues. Teku stayed faithful to the specification.

Client Fixes

• Lighthouse: https://github.com/sigp/lighthouse/pull/1166
• ZCLI: https://github.com/protolambda/zrnt/commit/ee4c8af2fb45301fdff6c7c72638e7ee677ec8fe

Steps taken to resolve:

• All clients have fixed the bug.
• The latest spec v0.12.0 has refactored balance calculations so they're easier to test. More tests have been added.
• The offending blocks and states have been added to beacon-fuzz (eth2 fuzzer) to help protect against regression and find related bugs.

Side note: this fork highlighted an interesting balance calculation quirk in the spec when the network isn't reaching finality. The specification was changed here.

What were the causes of the forks?

There were some issues relating to rewards/penalties calculation. Both Lighthouse and Prysm were incorrectly computing the validator balances.

Interestingly, Lighthouse, Prysm and ZCLI (reference spec implementation in Go) all computed different values, the result of multiple unrelated issues. Teku stayed faithful to the specification.

Client Fixes

• Lighthouse
  • https://github.com/sigp/lighthouse/pull/1166
• ZCLI
  • https://github.com/protolambda/zrnt/commit/ee4c8af2fb45301fdff6c7c72638e7ee677ec8fe
• Prysm
  • https://github.com/prysmaticlabs/prysm/pull/5909
  • https://github.com/prysmaticlabs/prysm/pull/5903

Steps taken to resolve:

• All clients have fixed the bug.
• The latest spec v0.12.0 has refactored balance calculations so they're easier to test. More tests have been added.
• The offending blocks and states have been added to beacon-fuzz (eth2 fuzzer) to help protect against regression and find related bugs.

Side note: this fork highlighted an interesting balance calculation quirk in the spec when the network isn't reaching finality. The specification was changed here.

What were the causes of the forks?

There were some issues relating to rewards/penalties calculation. Both Lighthouse and Prysm were incorrectly computing the validator balances.

Interestingly, Lighthouse, Prysm and ZCLI (reference spec implementation in Go) all computed different values, the result of multiple unrelated issues. Teku stayed faithful to the specification.

Here's the PR we (Lighthouse) created to fix our bug. It also contains some links to historical Discord messages (including the offending block and state): https://github.com/sigp/lighthouse/pull/1166

Steps taken to resolve:

• All clients have fixed the bug.
• The latest spec v0.12.0 has refactored balance calculations so they're easier to test. More tests have been added.
• The offending blocks and states have been added to beacon-fuzz (eth2 fuzzer) to help protect against regression and find related bugs.

Side note: this fork highlighted an interesting balance calculation quirk in the spec when the network isn't reaching finality. The specification was changed here.

What were the causes of the forks?

There were some issues relating to rewards/penalties calculation. Both Lighthouse and Prysm were incorrectly computing the validator balances.

Interestingly, Lighthouse, Prysm and ZCLI (reference spec implementation in Go) all computed different values, the result of multiple unrelated issues. Teku stayed faithful to the specification.

Client Fixes

• Lighthouse: https://github.com/sigp/lighthouse/pull/1166
• ZCLI: https://github.com/protolambda/zrnt/commit/ee4c8af2fb45301fdff6c7c72638e7ee677ec8fe

Steps taken to resolve:

• All clients have fixed the bug.
• The latest spec v0.12.0 has refactored balance calculations so they're easier to test. More tests have been added.
• The offending blocks and states have been added to beacon-fuzz (eth2 fuzzer) to help protect against regression and find related bugs.

Side note: this fork highlighted an interesting balance calculation quirk in the spec when the network isn't reaching finality. The specification was changed here.