I am learning Ethereum using Geth. I have a private blockchain and I am saving hashes through a REST API (python web service). I have readed I can save hash values using maps (mapping data type) and using logs. I am using the logs option and it works fine, but I want to know if it is the most efficient way, or if I should use a map or other kind of variable (like a string or byte array). I need to save millions of hashes (about 1 per second) so I am worried about the long-term scalability, mainly related to the search. By now I have a service saveHash(string hexHashValue) returning the transaction hash, and a service verifyHash(string hexHashValue, string transactionHash).

In python using web3.py I have this functions, I don't now if this can be improved, if using a map I can save millions of values in a better way or if I can find a specific string value in the contract using other methods. So my question is if the best way is logs, maps, strings or if the difference is trivial.

The contract:

pragma solidity ^0.5.0;

contract HashSave {                   
   event log_current_hash(bytes hashArg);

   constructor() public {                       

   function set_hash(bytes memory objectHash) public {
       emit log_current_hash(objectHash);

My current test methods using logs (event log_current_hash):

def save_new_hash(hash_bytes):
    tx_hash = contract_instance.functions.set_hash(hash_bytes).transact()   
    log_test_file.write("Hash: 0x%s Tx: %s\r\n" % (hash_bytes.hex(), tx_hash.hex()))
    return tx_hash.hex()

def check_transaction_exists(tx_hex_hash, obj_hash_bytes):
        tx_data = w3.eth.getTransaction(tx_hex_hash)
        if tx_data is not None:
            print("Transaction is in the blockchain")
            if find_obj_hash_in_transaction(tx_data.blockNumber, obj_hash_bytes) is True:
                return "Data validated ok"
                return "Data is not valid"
    except Exception:
        return "Transaction does't exists"

def find_obj_hash_in_transaction(block_num, obj_hash_bytes):
    hash_filter = contract_instance.events.log_current_hash.createFilter(fromBlock=block_num, toBlock=block_num,
                                                                       argument_filters={'hashArg': obj_hash_bytes})
    if hash_filter is None:
        return False
    hash_events = hash_filter.get_all_entries()
    if len(hash_events) > 0:
        return True
        return False
  • 1
    Current Ethereum clients are not well suited to store large amounts of data. An option is to store data off-chain in a merkle-tree like structure so you can generate a proof to validate if the data was tampered with. And store this proof in a smart contract.
    – Ismael
    Jan 15 '20 at 17:45
  • @Ismael thanks for your response. I thought one transaction per second wasn't large amounts of data for a private blockchain (e.g. 5 PCs in a room, about 5-10 MB transactions per day), but maybe you're right. I could save daily or hourly data in some database and like you say save some hash of all the info or a merkle tree root in the blockchain.
    – RobertGG
    Jan 15 '20 at 18:09
  • It doesn't appear to be much right now, but your client will be creating and storing a cryptographic proof every time. The other issue is that you need other tools to make useful queries to your data, like running a block explorer or something similar.
    – Ismael
    Jan 16 '20 at 14:02
  • Ok @Ismael I understand you. Now about your suggestion of using tools, I suppose you are talking from a management perspective and not about my functions. Until now I though blocks and network explorers where mainly for admin purposes (I'm currently using eth-netstats), but my idea was to call functions (like check_transaction_exists) from a remote location (e.g. using a web service). When you say "tools to make useful queries", are you talking about management and good practices right? because I can't connect a web service to a tool, I need a custom function for that. Thanks in advance.
    – RobertGG
    Jan 19 '20 at 3:47

I know this is old, but ... depending on you hash length, as the hash is generated outside the blockchain you should consider using uint256 (2 for a 512 bits hash) instead of bytes, in other words, save the hash not the representation of the hash.

  • uint256 and bytes32 are the same size, as both will take 32 bytes/256 bits. bytes without length specifier is usually a bit more inefficient. Nov 27 '21 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.