What is the best practice to store and retreive large data in solidity smart contracts?

Question

Assume, I have an event creation page with some form fields in UI. And I have a smart contract to store the event in events mapping or the public array in some struct format.

struct event{
      bytes32 name;
      uint time;
}

event[] public events;

or

mapping(uint=>event) public events;

I'm submitting the UI and data is stored in the contract.

I have a list events page, which will display all the events created in the contract.

After searching many forums, I got some info, can someone validate this.

Approach 1:

For every submits or save in contract, an event is called and UI is having the watcher and stored in the local storage and displays in the UI. The events data can be a mapping or array here.

I don't want to have any intermediate storage for the application, so whenever the UI refresh the event watcher will receive all the state data and again update the local storage and display in the UI.

Approach 2:

For event submit new event object is created and pushed into the events array.

And writing the get events method, to iterate the events array and returns the list of string elements for each key in the object and then again concatenating and destructure the strings in an array of objects in UI to bind with the UI.

I think the computation for the second approach will be high because of the iteration. Since we are developing the Dapp I don't want to have any intermediate database again which become as a centralised system.

Can someone explain the various possibilities of storing and retrieving data from the smart contract which is a preferable way so far.

It will help me a lot. Thanks in advance.

Answer 1

event is a keyword in solidity that allows you to easily retrieve data that a contract generated in a Javascript frontend. It is cheaper in terms of gas to write an event than to write into a storage variable (like your struct array). You can then .watch for these events and pull them into your UI. This is the recommended way to go. For performance reasons you might want to cache these event data somehow because parsing through all blocks on every reload is fairly inefficient.

Also, do not try to fit all your data into your smart contract. The job of a well-designed Dapp is to contain nothing more than the absolutely required elements and pointers to external structures but keep your storage-layer separated from the blockchain. A storage layer could be implemented via IPFS or Swarm.

Answer 2

As Sebastian stated, you want to be cautious of what data you want to store on the blockchain in the contract.

That being said, the gas cost for storing the data will land on the transaction issuer for creating an event. Likewise, any "transaction" based call that needs to iterate this data will become very costly depending on the amount of data and the iterations you need to do.

If a function is marked constant and does not require modification of the data, it can be called without incurring a gas cost and therefore iterating a list of events although expensive in computation if executed as a transaction is basically free to call.

A good practice would be to add a range that you want to list, so that you can limit the processing required on each call and page the results in your dApp.

Here is an example of a contract that can this, it's by no means complete, but copy paste in Remix will work to show basic functionality:

pragma solidity ^0.4.0;

contract EventManager {

    struct Event {
        bytes32 name;
        uint time;
    }

    uint totalEvents;

    mapping(uint => Event) EventList;

    event EventAdded(address indexed _senderAddress, uint _eventId);

    function addEvent(bytes32 _name, uint _time) returns(uint eventId) {
        eventId = totalEvents++;
        EventList[eventId] = Event(_name, _time);
        EventAdded(msg.sender, eventId);
    }

    function listEvents(uint _start, uint _count) constant returns(uint[] eventIds, bytes32[] eventNames, uint[] eventTimes) {

        uint maxIters = _count;
        if( (_start+_count) > totalEvents) {
            maxIters = totalEvents - _start;
        }

        eventIds = new uint[](maxIters);
        eventNames = new bytes32[](maxIters);
        eventTimes = new uint[](maxIters);

        for(uint i=0;i<maxIters;i++) {
            uint eventId = _start + i;

            Event memory e = EventList[eventId];
            eventIds[i] = eventId;
            eventNames[i] = e.name;
            eventTimes[i] = e.time;
        }
    }
}

Answer 3

TL/DR: Use a mapping with a dynamic array:

mapping(address => DataStruct[]) public dataStructsOfOwner;

If someone could guide me on where to better post this answer, please go ahead.

I had another task: Save contracts mapped to an owner. I have done a short simulation with these two options.

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;

struct Contract {
    string name;
    address contractAddress;
}

/**
 * @dev This Database saves an array of contracts for each owner
 */
contract Database1 {
    mapping(address => Contract[]) public contractsOfOwner;

    function addContract(
        address owner,
        string memory name,
        address contractAddress
    ) public {
        contractsOfOwner[owner].push(Contract(name, contractAddress));
    }

    function getContractsOfOwner(address owner)
        external
        view
        returns (Contract[] memory contracts)
    {
        return contractsOfOwner[owner];
    }
}

and

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;

struct Contract {
    string name;
    address contractAddress;
    address owner;
}

/**
 * @dev This Database saves an array of contracts and iterates over them to get
 * contacts by owner
 */
contract Database2 {
    address[] contracts;
    mapping(address => Contract) contractDetails;
    mapping(address => uint256) balanceOf;

    function addContract(
        address owner,
        string memory name,
        address contractAddress
    ) public {
        contracts.push(contractAddress);
        contractDetails[contractAddress] = Contract(
            name,
            contractAddress,
            owner
        );
        balanceOf[owner] += 1;
    }

    function getContractsOfOwner(address owner)
        external
        view
        returns (Contract[] memory returncontracts)
    {
        uint256 count = balanceOf[owner];
        Contract[] memory contractsOfOwner = new Contract[](count);
        uint256 i;
        uint256 resultIndex;
        for (i = 0; i < contracts.length; i++) {
            if (contractDetails[contracts[i]].owner == owner) {
                contractsOfOwner[resultIndex] = contractDetails[contracts[i]];
                resultIndex++;
            }
        }
        return contractsOfOwner;
    }
}

Surprising result:

    ·--------------------------------------|---------------------------|-------------|-----------------------------·
|         Solc version: 0.8.10         ·  Optimizer enabled: true  ·  Runs: 200  ·  Block limit: 30000000 gas  │
·······································|···························|·············|······························
|  Methods                             ·              111 gwei/gas               ·       3983.14 eur/eth       │
·················|·····················|·············|·············|·············|···············|··············
|  Contract      ·  Method             ·  Min        ·  Max        ·  Avg        ·  # calls      ·  eur (avg)  │
·················|·····················|·············|·············|·············|···············|··············
|  Database1     ·  addContract        ·      72947  ·      90059  ·      81500  ·            4  ·      36.03  │
·················|·····················|·············|·············|·············|···············|··············
|  Database2     ·  addContract        ·     122359  ·     156559  ·     135187  ·            4  ·      59.77  │
·················|·····················|·············|·············|·············|···············|···························|···············|··············
|  Deployments                         ·                                         ·  % of limit   ·             │
·······································|·············|·············|·············|···············|··············
|  Database1                           ·          -  ·          -  ·     406176  ·        1.4 %  ·     179.58  │
·······································|·············|·············|·············|···············|··············
|  Database2                           ·          -  ·          -  ·     437581  ·        1.5 %  ·     193.47  │
·······································|·············|·············|·············|···············|··············
·--------------------------------------|-------------|-------------|-------------|---------------|-------------·