Searching data on a smart contract?

Question

I have a smart contract which contains some article data stored inside a struct, which is mapped with an id which uniquely identifies it with a mapping, like this:

struct Foo {
    string a;
    string b;
}

mapping (uint64 => Foo) public articles;

(Obviously the above code is simplied, but the structure is the same, although the real struct has 7 variables instead of 2).

Now I have to be able to search using any of the 7 attributes. The blockchain is going to be private, so gas usage is not a concern.

My first solution was to add a search method to the smart contract and, within it, traverse the entire mapping and check which items satisfy the condition. Obviously this is O(n) and it was taking around 8 seconds for 2000 elements, which although isn't bad, I hope for a better solution.

Then I thought about having 7 more mappings which would act as search indexes, mapping each attribute value to the index of the article that satisfies it, such as:

mapping (string => uint64) public searchIndex;

(Note that right now it isn't required to check substrings, only equality)

And finally, I was thinking about if it would be good to do keep these indexes on Elastic Search rather than on the blockchain itself, although I would prefer a pure blockchain-based solution if possible.

So, what of these options (of any other better you know) do you think is more adequate?

Thank you.

Answer 1

It is clear you can't directly get the object you want without knowing it's key. The idea of the 7 mappings is actually the most obvious and the least complicated one and it works in most cases not only yours.

The most useful idea I can come up with is that you look for patterns in your struct attributes, patterns that help you organize a search schema in case you have 10 or 20 thousand elements. e.g. ; if string a values are linked to a known timeline like a == "X12" for today and the next day another entry would havea == "X13". So when you want to search for structs with a == X32 you will have better understanding of where could that struct be and what are the search index limits.

Thus making searching in 20 thousand elements a search in only 2 thousand.

hope it was clear.

Answer 2

One thing I have done to solve this same type of problem is to mirror the data coming into the contract in a Redis or Mongo database to create a much more queryable dataset. Then, whenever I query that database, I verify the data in the blockchain using a getter function to ensure it hasn't been tampered with. I realize this is not the purest blockchain-only method, but it works without risking gas-heavy loops or multiple maps that multiply storage cost for what should be simple queries on non-key attributes.

Given your simplified contract:

struct Foo {
    string a;
    string b;
}

mapping (uint64 => Foo) public articles;

You can create an event to emit whenever a new addition is created like this that returns the values and the unique ID:

event newArticleCreated(string a, string b, index uint64);

Then, using the Ethers.js library (or another like Web3) you can set up a listener to catch any instances of this event and handle the data however you wish from there. In this case, you would pass the data to a service that adds the data to Mongo or Redis with the key being the uint64 key in your contract:

    //import the ethers.js library
    const ethers = require('ethers');

    //set your provider and create a new instance of your contract by providing 
    //the ABI and deployed address 
    let provider = new ethers.providers.JsonRpcProvider(<RPC URL for your node>); 
    contractInstance = new ethers.Contract(<contractAddress>, <abi>, provider);

    //listen for events matching this name
    contractInstance.on("newArticleCreated", (a, b, index) => {
        //do stuff when the event fires

        //set the values in the event to a JSON object structure
        let chainObj = {
            id: index, 
            a: a
            b: b
        }

        //add the data to the database
        dataService.addDataToDatabase(chainObj);

I have personally had success with this type of setup and it has a lot of flexibility. It allows you to query data mirrored from the chain, but always verify that data back with the contract using that index/unique ID.

Thanks!