26

Following Q/A (What datatype should I use for an IPFS address hash?) recommend us to use bytes to store IPFS hash.

I was using following example(https://github.com/AdrianClv/ethereum-ipfs/blob/master/NotSoSimpleStorage.sol), which uses string to store IPFS hash that costs around 110,000 gas price, which seems pretty expensive.

[Q] Does using bytes instead of string in order to store IPFS hash cost cheaper? I observe that storing bytes instead of string costs very close to string (110,000 gas). Since both datatypes storage seems expensive should I use events to store them?

Is there any example/tutorial related to store IPFS hash using bytes?

Would this work:

myContract.insertHash("QmWmyoMoctfbAaiEs2G46gpeUmhqFRDW6KWo64y5r581Vz");

contract Example_bytes {
    bytes[] list;
    function insertHash(bytes ipfsHash) {
       list.push(ipfsHash); //costs around 110,000 gas. 
    }
}

contract Example_string {
    struct hashes{
         string hash;
    }

    hashes[] list;
    function insertHash(string ipfsHash) {
       list.push(hashes{hash: ipfsHash); //costs around 110,000 gas. 
    }
}
38

Your example shows storing an IPFS identity using it's alphanumeric encoding (Qm...), which is the same Base58 encoding that Bitcoin uses. However, what it's representing at its core is a number (the hash). Storing the identifier in the Base58 format needs to be a String because it includes letters (and what actually gets saved is the ASCII code for each alphanumeric character in the identifier). That means you need 46 bytes to store QmWmyoMoctfbAaiEs2G46gpeUmhqFRDW6KWo64y5r581Vz from your example.

However, that identifier can also be expressed in hexadecimal as 12207D5A99F603F231D53A4F39D1521F98D2E8BB279CF29BEBFD0687DC98458E7F89, which is only 34 bytes long (takes 68 characters to write out in hexadecimal, since every two characters in hex is a byte of data).

But, both of those are greater than 32 bytes, which is the max fixed-size byte array, so they're going to need to use a dynamically-sized byte array to store (bytes or string, both of which are expensive, as you noted).

BUT, that IPFS hash is actually two concatenated pieces. It's a multihash identifier, so the first two bytes indicate the hash function being used and the size. 0x12 is sha2, 0x20 is 256-bits long. Currently, that's the only format IPFS uses, so you could just chop off the first two bytes, which leaves you with a 32-byte value, which is small enough to fit in a bytes32 fixed-size byte array, and you save some space there (and when retrieving either your contract can re-attach 0x1220 to the front of it, or your clients need to be smart enough to do that after retrieving the value).

If you want to make sure your code is future-proof, though, you probably want to save that hash function code and size, which you could combine with the hash as a struct:

struct Multihash {
  bytes32 hash
  uint8 hash_function
  uint8 size
}

That will work with any multihash format, as long as size is less than or equal to 32 (any bigger and the actual payload won't fit in the hash property). This struct will take two storage slots (two 32-byte chunks) to store, since the two uint8 pieces can be put in one slot. You could also add up to 30 bytes of additional data to this struct without taking any more storage cost.

| improve this answer | |
  • Sorry I did not get how you obtained: 12207D5A99F603F231D53A4F39D1521F98D2E8BB279CF29BEBFD0687DC98458E7F89. Here(codebeautify.org/string-hex-converter) when I convert QmWmyoMoctfbAaiEs2G46gpeUmhqFRDW6KWo64y5r581Vz into hex I got much larger string as 516d576d796f4d6f63746662416169457332473436677065556d687146524457364b576f3634793572353831567a. @MidnightLightning – alper Jun 2 '17 at 20:35
  • 3
    The conversion you did (string-hex-converter) is taking the string "QmWmy..." and showing you how the string would be stored (the ASCII value for "Q" is 0x51, "m" is 0x6d, etc.). What I did is use a tool that does Base58-decoding and used that to get the actual number being represented by that Base58 string. – MidnightLightning Jun 2 '17 at 20:55
  • 1
    uint8 function is storing an unsigned integer value as the name "function" in that struct object. "Function" is probably not the best name for that since it's a special word in Solidity and other programming languages; I picked it because in the multihash standard, that's what they call that variable; the variable that tells you what hashing function was used for this particular record (e.g. 0x12 for "sha2"). I'll update my answer to not use that special word "function" to be more clear. – MidnightLightning Jun 2 '17 at 20:58
  • 1
    Thank you so much for this. You really saved me a lot of time and effort here. This is exactly what I needed. – rhlsthrm Feb 15 '18 at 0:17
  • 2
    Thank you for saving me a ton of time. I put your approach in an end-to-end example and hope it saves some time for other people: github.com/saurfang/ipfs-multihash-on-solidity – Saurfang Feb 19 '18 at 0:01
5

Here are some js functions for stripping and re-adding the first two bytes containing the hash function and size, suitable for web3.

import bs58 from 'bs58'

// Return bytes32 hex string from base58 encoded ipfs hash,
// stripping leading 2 bytes from 34 byte IPFS hash
// Assume IPFS defaults: function:0x12=sha2, size:0x20=256 bits
// E.g. "QmNSUYVKDSvPUnRLKmuxk9diJ6yS96r1TrAXzjTiBcCLAL" -->
// "0x017dfd85d4f6cb4dcd715a88101f7b1f06cd1e009b2327a0809d01eb9c91f231"

getBytes32FromIpfsHash(ipfsListing) {
  return "0x"+bs58.decode(ipfsListing).slice(2).toString('hex')
}

// Return base58 encoded ipfs hash from bytes32 hex string,
// E.g. "0x017dfd85d4f6cb4dcd715a88101f7b1f06cd1e009b2327a0809d01eb9c91f231"
// --> "QmNSUYVKDSvPUnRLKmuxk9diJ6yS96r1TrAXzjTiBcCLAL"

getIpfsHashFromBytes32(bytes32Hex) {
  // Add our default ipfs values for first 2 bytes:
  // function:0x12=sha2, size:0x20=256 bits
  // and cut off leading "0x"
  const hashHex = "1220" + bytes32Hex.slice(2)
  const hashBytes = Buffer.from(hashHex, 'hex');
  const hashStr = bs58.encode(hashBytes)
  return hashStr
}

Here is the function used in context, calling a Listing contract deployed with Truffle.

submitListing(ipfsListing, ethPrice, units) {
  return new Promise((resolve, reject) => {
    this.listingContract.setProvider(window.web3.currentProvider)
    window.web3.eth.getAccounts((error, accounts) => {
      this.listingContract.deployed().then((instance) => {
        let weiToGive = window.web3.toWei(ethPrice, 'ether')
        return instance.create(
          this.getBytes32FromIpfsHash(ipfsListing), /*** IPFS here ***/
          weiToGive,
          units,
          {from: accounts[0]})
      }).then((result) => {
        resolve(result)
      }).catch((error) => {
        console.error("Error submitting to the Ethereum blockchain: " + error)
        reject(error)
      })
    })
  })

Taken from my work on the Origin Demo Dapp here: https://github.com/OriginProtocol/origin-js/blob/1cfc84d4693974bbf18e345e6c0def843321130c/src/services/contract-service.js#L102-L128

| improve this answer | |
3

I have handled similiar situation with this util function in web3.py:

import base58

def convertIpfsBytes32(hash_string):           
  bytes_array = base58.b58decode(hash_string) 
  return bytes_array[2:]

You need base58 module. Concept is same as accepted answer.

| improve this answer | |
1

This answer is just Python implementation of @MidnightLightning's accepted answer above. I have used Web3.py.

from web3.auto import w3

def _ipfs_to_bytes32(hash_str: str):
    """Ipfs hash is converted into bytes32 format."""
    bytes_array = base58.b58decode(hash_str)
    b = bytes_array[2:]
    return binascii.hexlify(b).decode("utf-8")

def ipfs_to_bytes32(ipfs_hash: str) -> str:
    """bytes32 is converted back into Ipfs hash format."""
    ipfs_hash_bytes32 = _ipfs_to_bytes32(ipfs_hash)
    return w3.toBytes(hexstr=ipfs_hash_bytes32)

def bytes32_to_ipfs(bytes_array):
    """Convert bytes_array into IPFS hash format."""
    merge = Qm + bytes_array
    return base58.b58encode(merge).decode("utf-8")

if __name__ == "__main__":
    ipfs_hash = "QmWmyoMoctfbAaiEs2G46gpeUmhqFRDW6KWo64y5r581Vd"
    ipfs_bytes32 = ipfs_to_bytes32(ipfs_hash)
    _ipfs_hash = bytes32_to_ipfs(ipfs_bytes32)
    assert ipfs_hash == _ipfs_hash  # They should be equal to each other

| improve this answer | |
0

Here's a more complete example using the js-multihash library:

MyContract.sol

pragma solidity ^0.4.24;

contract MyContract {

  event AddFile(address indexed owner, bytes32 digest, bytes2 hashFunction, uint8 size, bytes4 storageEngine);

  function addFile(bytes32 _digest, bytes2 _hashFunction, uint8 _size, bytes4 _storageEnginge) public {
    emit AddFile(msg.sender, _digest, _hashFunction, _size, _storageEngine);
  }
}

Javascript

import Web3 from 'web3'
import multihashes from 'multihashes'
import ipfsAPI from 'ipfs-api'

var web3 = window.web3
web3 = new Web3(web3.currentProvider)

// Utility functions:
const utils = {
  ipfs2multihash (hash) {
    let mh = multihashes.fromB58String(Buffer.from(hash))
    return {
      hashFunction: '0x' + mh.slice(0, 2).toString('hex'),
      digest: '0x' + mh.slice(2).toString('hex'),
      size: mh.length - 2
    }
  },

  multihash2hash (hashFunction, digest, size, storageEngine) {
    storageEngine = web3.toAscii(storageEngine)

    if (storageEngine === 'ipfs') {
      hashFunction = hashFunction.substr(2)
      digest = digest.substr(2)
      return {
        hash: multihashes.toB58String(multihashes.fromHexString(hashFunction + digest)),
        engine: storageEngine
      }
    }

    throw new Error('Unknown storage engine:', storageEngine)
  }
}

// ... code to instantiate contract
// ... code to get the file buffer

ipfs.add(buffer)
  .then((response) => {
    console.log('ipfs hash:', response[0].hash)

    // Prepare data
    let mh = utils.ipfs2multihash(response[0].hash)
    let storageEnginge = web3.fromAscii('ipfs')

    // Call contract
    myContractInstance.addFile.sendTransaction(mh.digest, mh.hashFunction, mh.size, storageEnginge, {from: myAccount, gas: 1000000}, (error, result) => {
      if (error) throw error
      console.log(result)
    })
  })

Reading the data

let fileFilter = myContract.AddFile({
    owner: myAccount
  }, {
       fromBlock: 0,
       toBlock: 'latest'
     }).watch((error, log) => {
      if (error) reject(error)

      console.log('file log:', log)

      let hash = utils.multihash2hash(log.args.hashFunction, log.args.digest, log.args.size, log.args.storageEngine)
      console.log('Hash:', hash)

      fileFilter.stopWatching()
    })
| improve this answer | |
-1

All of the answers here are needlessly complicated. If, in the end, you're going to store 64 bytes anyway, why not simply store the first 32 bytes of the IPFS hash in one string32 and the remainder of the hash in another string32? No need to convert when storing, therefore less gas to process. Simply concatenate to reconstitute the hash, therefore cheaper to process. Same gas cost to store (64 bytes). And, more future proof. Unlike the selected solution, the length of the IPFS hash change grow (or shrink) with no change needed to the code.

| improve this answer | |
  • Memory cost is very expensive in Ethereum, since if you can store in a single bytes32, why would you store it in 2 bytes32? In your contract you've to convert everything for 2 slot of bytes32, if you push those two into a list there will be additinal length so unnessary memory is consumed. "Same gas cost to store (64 bytes)" => this is wrong , storing 32 bytes is cheaper than storing 64 bytes in cases of tight variable packing please see fravoll.github.io/solidity-patterns/tight_variable_packing.html . also when you store it in one slot, you can use it as a key in map – alper Aug 5 at 13:09
  • 1
    "simply concatenate" is actually not that simple in Solidity (there's no native support for that, leading to needing to do some complex manipulations that use up a bunch of gas to do). Your contract could export the two halves as separate strings, but then that's less intuitive to the end user (they need to post-process). – MidnightLightning Aug 6 at 22:02
  • Im not suggesting the reconstitution takes place on-chain, so gas costs to reconstitute would be zero. There's no use for an hash IPFS "on-chain" -- smart contracts can't read the document pointed to by the hash. I'm suggesting concatenating off-chain and that the contract should expose the hash in two parts. Mildly confusing to the user, but zero cost for retrieval. On input, all of the above solutions require splitting the string (plus the other solutions require a conversion from Base58) so that cost is also lower. Split the hash prior to insertion, and you can even eliminate that cost. – Thomas Jay Rush Aug 8 at 13:21
  • 1
    My struct solution also assumes the splitting/joining happens off-chain: inputting the three parts (not as strings) and saving is the same cost as inputting two strings and saving. So, really depends on the needs of the app. Most IPFS off-chain libraries would have functions to unpack a string ID into it's three numeric/bytes parts, but splitting strings would be a custom action not part of the IPFS standard – MidnightLightning Aug 8 at 16:38
  • I won't continue the conversation after this comment, but two things. Splitting a string is part of every off-chain programming language I've every seen, so not sure why it would need to be part of the IPFS standard, plus I want to point out that your solution is less future proof, as the underlying 32 byte hash can't grow past its 32-byte boundary without requiring it to be split anyway, whereas a purely string-based solution does not have this problem. – Thomas Jay Rush 2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.