I am trying to sample a random number without replacement until the pool of possible numbers runs out, using Solidity. In pseudo-code, it should be something like this:

1. Initial pool of numbers: (1, 2, 3, 4)
2. Attempt 1: one of (1, 2, 3, 4) are possible, 3 is sampled
3. Attempt 2: one of (1, 2, 4) are possible, 4 is sampled
4. Attempt 3: one of (1, 2) are possible, 1 is sampled
5. Attempt 4: one of (2,) are possible, 2 is sampled
6. End

Note that each Attempt is a different call to the contract (by a different user/sender/wallet).

So far, the below code gets me a single random number within a range from 1 to 50:

uint256 public randomResult;

function fulfillRandomness(bytes32 requestId, uint256 randomness) internal override {
    randomResult = (randomness % 50) + 1;
}

I am stuck as to how do I get the pseudo-code's logic into a smart contract using Solidity. Preferably would like to avoid solutions that require iterations to check if number has been seen or not. Any help is appreciated. Thanks.