Is there a limit to the function call output?

Question

I have a simple contract for storing string tuple of ipfs hashes data in the array

pragma solidity ^0.4.24;
pragma experimental ABIEncoderV2;

contract Store {
    string[] public objects;

    function setData(string memory x, string memory y) public {
        objects.push(x);
        objects.push(y);
    }

    function getDataAtIndex(uint256 index) public returns (string memory, string memory) {
        return (objects[2 * index], objects[2 * index + 1]);
    }

    function getCount() public returns (uint count) {
        return objects.length / 2;
    }

    function getAllData() constant returns (string[]) {
        return objects;
    }
}

I am using getAllData() to get the whole array at once. When the size of the array is small, everything works just fine. But when I have large array, it raise an error

web3.exceptions.BadFunctionCallOutput: Could not decode contract function call getAllData return data b'' for output_types ['string[]']

Do you know what causes the problem?

Is there better way to store large amount of ipfs hashes in one contract? Or is it better to store it in more contracts and search through all of them?

Answer 1

Generally, "return everything" is unnecessary and unreliable.

It is similar in spirit to SELECT * FROM OBJECTS but it is not especially well-adapted to the EVM. The database-centric world is one of a centralized database and a relatively cheap (cost and performance) operation. The EVM consists of nodes that have replicated copies of the state and a different cost base and performance profile.

So, why isn't it necessary?

You can proceed on the assumption that all nodes (all interested observers) can and should be aware of all state changes. That is, all insert, update and delete operations. This is because all transactions are observable by all interested parties (no such assumption applies in a database-centric setting). You can make it more convenient to observe important state changes with events.

event LogSetData(string value);

function setData(...
   object.push(...
   LogSetData(x);
   ...

Subscribers can "listen" to the event log from the block in which the contract was deployed to the present block, and into the future such that no insertion transaction will escape their notice. A client can, therefore, see to it that they are already aware of the state of the array. What then would be the point of asking?

You already have a discoverable length and a way to inspect the data at a specific index (a little strange, but it's there). That would be suitable when the client is another contract. Another contract would not be interested in replicating the entire array (too costly) so, again, no need to access the entire array in one gulp.

Why is it unreliable?

Gas accounting is in play for read-only operations and gasLimit rules apply. Therefore, there is an upper bound on the size of the array. This implies a limit to scalability. If the array is too large, the function will fail.

It's true that the gas will be refunded (never actually spent) in a read-only operation. This is a confusing topic. More info: https://blog.b9lab.com/getting-loopy-with-solidity-1d51794622ad

Since we know that there are limits on the size of arrays for both inputs and outputs, it's always a good idea to provide an alternative. If one codes the alternative, one often finds the bulky function isn't needed. Your getDataAtIndex() and getCount() functions enable a client to iterate over the objects. Strictly speaking, getAllData() isn't needed, which is a good thing because it's not going to work with large arrays.

You may have noticed that dynamic arrays are sometimes used in batch operations. As inputs for, say, airdrops, arrays can squeeze a few drops of gas out of operations that are slightly more expensive done one row at a time. Importantly, the sender has control of the batch size in that case, so the sender can govern themselves accordingly having found the maximum batch size they can safely send.

The same logic doesn't apply to your read function because it returns them all. This won't be anywhere near as fast as a database, because the node needs to fish around in the blockchain to find all the elements. It's just not optimized for that. It has no way to limit the rows returned to a magnitude that finishes before the block gasLimit is exceeded (has no from, to), so it will simply fail when the list gets too big.

Hope it helps.

Answer 2

All nodes in the network replicate all the data in the network so your node also contains all the required data. Therefore you can just read the data directly from your node without interacting with the network. This also means that reading data in Ethereum is free and does not cost any gas. (Exception is that if the reading is done in a non-readonly transaction which interacts with the blockchain - in that case you have to pay all the typical gas even for the read operations.)

So there is no need to optimize your read operations. They are fast enough as you get the data directly from your node and they are free.

In fact, you don't have to include any reading functionality in your code as all data is accessible even without explicit code. In the case of public variables, a public getter method is automatically generated and can be accessed with objects(index) where index is the index in the array. More about this for example at https://solidity.readthedocs.io/en/v0.4.24/types.html#arrays .

If you don't know the index (or how long the array is) you have to do a bit more work - easiest is to include a function which gives the length (get array.length without a getter from other contract?) or then interpret it for example from transactions (how many times data has been input in the array).