0

I have several terabytes of relatively simple data, not much more complicated than the ledger storing ids, timestamps, simple metadata.

I understand ETH storage costs are rather expensive.

What are my options for storing as much data as possible off-chain to minimize costs, but still getting distributed, and fault-tolerant benefits of being on-chain, using on-chain for smart contracting on the data? How do I search for and understand strategies for doing this?

If my metadata is rather sparse, say not much more than integers and a couple other ids, and still terabytes in size, is this even possible, or is a blockchain solution a bad fit?

1

Storing terabytes on the blockchain is not possible: it would cost too much. You have to store that data off-chain and eventually group it in a way that makes sense. What I did in a scenario similar to yours was to store my data on a database (in my case was a NoSql - MongoDB) and then calculate an hash and store it in the blockchain along with the id useful to retrieve it from the database.

In this way you a are "extending" the blockchain (or better, a contract) with some data that you guarantee are immutable (if you change the data in the database the hash would not be the same). Of course that data are not distributed and as such you need to implement a strategy to make them fault tolerant and widely available.

The logic to link the data on-chain with that off-chain should be in the application (client or server, depending on your scenario)

  • Thanks! Can I ask to understand extension better: how many hashes did your project have? More like millions or more like trillions? – Mittenchops Nov 13 '18 at 17:50
  • 1
    In my case I was able to aggregate small records in bigger data structures that were some kilobytes each and hashed them. The bigger the structure that contains the smaller records the bigger the gain in terms of hashes generated. You can also create an hash based on thousands of hashes and just store that on the blockchain. In my project, in the end, we were able to "group" records down to a single day with a result of less than a thousand contracts written on the blockchain. – Marco Ottolini Nov 14 '18 at 3:26
1

A blockchain solution is probably the best for decentralization purposes. If your intent is to just distribute your data for ease of access or workflow needs you can easily implement an oracle distributed Database.

For the case of ethereum, you must know that the smart contract storage costs a lot comparing to regular storage, and the size of transactions on the ethereum platform is limited by its cost of gas and the maximum allowed gas so transacting with massive amount of data will be nearly impossible.

If you want to go with ethereum anyway, for decentralization and specific workflows using smart contracts, you can use external distributed storage services. you can use an oracle distributed database or something similar, or you can try decentralized services like IPFS. IPFS does go well with the data storage that the ethereum platform lacks, and doesn't kill the decentralization efforts this latter brings.

  • 1
    Probably not the best for ... ? – Rob Hitchens - B9lab Nov 13 '18 at 18:28
  • Not best for storing huge amount of data, excessive reading and writing, easy deployment needs, ... – Kaki Master Of Time Nov 14 '18 at 12:41
0

As you were already told, Ethereum cannot store anywhere near this much data.

However, if we allow us to deviate a little bit from the topic of this forum (Ethereum), you can store that data in other blockchains which are meant for data storage. Such blockchains are for example Storj and Siacoin.

IPFS is an often references alternative, but it's not a very decentralized data storage in its current form. If you use IPFS, you have to be the one who runs the nodes where the data resides - nobody else will store the data in their IPFS nodes without incentives to do so. Yes, IPFS has plans to incorporate Filecoin for the incentives part, but this is far from complete. So, if you have to run the nodes for the data, I don't see much point in using IPFS currently - you could just store the data in regular mirrored (cloud) data storage.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.