What's the difference between invariant and fuzz testing?

Question

echidna is a popular property-based fuzz tester, and Foundry comes packed with invariant tests.

What is the difference between them?

Answer 1

Summary

Fuzz testing, also known as fuzzing, involves providing random data as inputs during testing.

Invariant tests are tests that focus on verifying the conditions that must always hold in a system.

In Foundry, an invariant test involves making random function calls with random data inputs to break some invariant (some assertion that must always hold.)

This means, an invariant test is also a kind of fuzz test (more widely known as "stateful fuzzing") since an invariant test uses random data.

Fuzz Example

When writing tests, it's hard to come up with every input a user might do, so fuzzing automatically runs through examples.

For example, in our Vault contract someone should not be able to guess the password:

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;

contract Vault {
    bool public s_locked;
    bytes32 private s_password;

    constructor(bytes32 password) {
        s_locked = true;
        s_password = password;
    }

    function unlock(bytes32 password) external {
        if (s_password == password) {
            s_locked = false;
        }
    }
}

We could setup a fuzz test that passes random (or semi-random) values to get the answer.

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.7;

import "../../Vault.sol";

contract VaultFuzzTest is Vault {
    constructor() Vault("123asd123") {}

    function echidna_test_find_password() public view returns (bool) {
        return s_locked == true;
    }
}

Echidna is a "property-based" fuzz tester, which means it's somewhat intelligent with the random values it uses. It sees 123asd123 is used in storage, and may try that as one of the starting random values.

The line return s_locked == true; tells Echidna that the Vault should always be locked, and that if it finds an example where it unlocks the Vault, it should consider the test failed. Running a fuzz test on the above results in something like this:

   ┌─────────────────────────────────────────────────────Echidna 1.7.2────────────────────────────────────────────────────┐    
   │ Tests found: 1                                                                                                       │    
   │ Seed: 5536602576911304448                                                                                            │    
   │ Unique instructions: 154                                                                                             │    
   │ Unique codehashes: 1                                                                                                 │    
   │ Corpus size: 1                                                                                                       │    
   │─────────────────────────────────────────────────────────Tests────────────────────────────────────────────────────────│    
   │ echidna_test_find_password: FAILED!                                                                                  │    
   │                                                                                                                      │    
   │ Call sequence:                                                                                                       │    
   │ 1.unlock("123asd123\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL")    │    
   └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘    
                                            Campaign complete, C-c or esc to exit                                                                                      

Invariant Example

What's interesting, is the above example is actually also an invariant test. Invariant testing makes sure certain conditions always hold true. In our above example, if we assume our vault can never be unlocked, we want to test this condition will always hold true.

However, an invariant test can be more than just providing random data to functions, it can also be calling random functions in random orders.

In foundry, we could write an invariant test for the above Vault in Foundry.

// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.13;

import "forge-std/Test.sol";
import "../src/Vault.sol";

contract CounterTest is Test {
    Vault public vault;

    function setUp() public {
        vault = new Vault("123asd123");
    }

    function invariant_CannotUnlockVault() public {
        assert(vault.s_locked());
    }
}

And running forge test will result in:

Running 1 test for test/Vault.t.sol:CounterTest
[FAIL. Reason: Assertion violated]
        [Sequence]
                sender=0x0000000000000000000000000000000000000010 addr=[src/Vault.sol:Vault]0x5615deb798bb3e4dfa0139dfa1b3d433cc23b72f calldata=unlock(bytes32), args=[0x3132336173643132330000000000000000000000000000000000000000000000]

 invariant_CannotUnlockVault() (runs: 8, calls: 115, reverts: 0)
Test result: FAILED. 0 passed; 1 failed; finished in 17.89ms

Failing tests:
Encountered 1 failing test in test/Vault.t.sol:CounterTest
[FAIL. Reason: Assertion violated]
        [Sequence]
                sender=0x0000000000000000000000000000000000000010 addr=[src/Vault.sol:Vault]0x5615deb798bb3e4dfa0139dfa1b3d433cc23b72f calldata=unlock(bytes32), args=[0x3132336173643132330000000000000000000000000000000000000000000000]

 invariant_CannotUnlockVault() (runs: 8, calls: 115, reverts: 0)

When you get more advanced with invariant testing, you can select a group of functions you'd want the invariant tester to try to loop through, making random function calls with random data.

So invariant tests and fuzz testing have a bit of overlap as you can see,