Mapping vs Arrays: duplicating variables gas consumption

Question

I have a contract that has a struct including address and unit, and mapping of a struct, like this.

  struct User {
    uint count;
    address userAddress;
  }
  mapping (address => User) users;

There is a point at which I need to make a copy of this mapping and clear out the entries of the old one, like this pseudo code:

  users_backup = users;
  users = [];

Of course this code won't work, so I learned you can loop through it like this:

Assignment of mapping in Solidity

But that seems expensive, especially after reading:

Copying a mapping from Contract A to contract B

So How do I do this efficiently?

With a mapping of 100k users or 1 million or even 100 million entries would this looping method be feasible? Or would it be better to change my architecture?

uint[] userCounts;
address[] userAddresses;

I could use a series of arrays instead of maps. Then, would I be able to copy one array to a new variable and clear the old variable out in one or two lines with less gas? The Hard thing is, even the arrays will have to be variable arrays because I can't know beforehand how many users will use it.

What would you do to minimize gas consumption? I'm just not sure what the best pattern is, thanks!

Answer 1

It's not easy to conjure up a trivial example. Based on the comments, I think the main concerns are approximately:

1. Clean up expired information responsibly and capture the gas refund offered for releasing unneeded storage.
2. Avoid iteration. https://blog.b9lab.com/getting-loopy-with-solidity-1d51794622ad
3. Access information in O(1), by user, by epoch, globally by epoch, or globally overall.

You may have more paths to consider. You have to anticipate all queries the contract needs to make against its own data and work out a way to access the answer with O(1) complexity.

This approach is to give you some ideas:

1. You'll need data organization that easily contends with using id sets that refer to mappings of structs. It usually works best to have a global mapping of everything that exists, with lists of pointers that satisfy each case the contract needs for its own lookup logic. For example, a global list of all transaction IDs that exist and similar lists for each user, in each epoch. You can use address(this) as a "User" to represent the global set. You will write keys in multiple places but the records only once.
2. You will want to add another dimension to some parts of the structure to logically group things into epochs (e.g. a year). That which should be logically cleared out goes in the Epoch layer and initializes to zero, naturally. That which should not reset every year goes in the layer above so that it appears to carry forward. To be clear, it is imperative that the data stays at rest because large-scale reorganization is not feasible or desirable.
3. If you are done with the data you should be a good citizen and do garbage collection. This will also help you economize on gas cost for using the system, because each time a user deletes a non-zero value, they get a gas refund. You have to do that without iteration. A solution to that problem is to use the structure that exists to inspect the expired Epoch. If data still exists, delete one record. You can do that every time a user supplies gas for any purpose by attaching a modifier to state-changing functions. This should not objectionable because users get a gas refund if there is something to delete.

You can make life a lot easier with these Set libraries that attend to CRUD operations. https://github.com/rob-Hitchens/SetTypes. In case the method being used isn't clear, check out this series: https://medium.com/robhitchens/solidity-crud-epilogue-e563e794fde

You can do all sorts of data handling as libraries - linked lists, sorted lists, etc.. For something as complex as your projects sounds, I recommend separating those concerns from the line-of-business application to reduce repetition and improve readability and probably reliability.

The garbage collection method suggested is based on the notion of work amortization explained over here: https://medium.com/@weka/dividend-bearing-tokens-on-ethereum-42d01c710657. The current Epoch and the proper functioning of the contract is the primary concern, of course. It should be irrelevant if obsolete data is cleaned up, but, it is good policy to do so, and you can make it so users are rewarded for chipping away at it until the cleanup process is complete.

Roughly:

modifier garbageCollection {
  if(previousEpochStillExists) {
    // get to work removing one txn/record and all references to it
  }
  if(nothingLeft)
    // remove the epoch itsef
  }
  _;
}

As an after-thought, think carefully about minimizing state storage. You can immortalize inputs with event logs (way cheaper). A good heuristic is to limit state storage to values the contract needs for its own internal logic. Never for memre record-keeping because a client might be interested. There are more efficient ways to do that.

Hope it helps.