Generate directory hash without upload using Swarm API

Question

I am using the Swarm API (github.com/ethersphere/swarm/) with go to upload a directory on a private network. The client API(/api/client) provides a UploadDirectory function which returns a manifest hash. Is there a way to calculate the same Merkle Tree hash without uploading the files to the Swarm storage, preferably with the API ? The reason for doing this is to upload the directory at some later point of time and verify the hash with the one calculated before.

Any help with some possible areas I can look at would be really appreciated.

Answer 1

Is there a way to calculate the same Merkle Tree hash without uploading the files to the Swarm storage, preferably with the API?

1. It's not the root of a Merkle Tree
2. And yes, you can calculate it in advance
3. But no, you will not obtain what you want to achieve

=== First one first ===

Under the hood, manifest is a text file that contains a description of the folder structure, with hashes of files, content-types, and so on.

Let's say you have a folder with 2 files: hello.txt, containing the word hello, and world.txt, containing the word world!. The manifest will be something like the following:

{
    "entries": [
        {
            "hash": "7a59da2349f6542e16fddc9399f01327084ed0692b5b110b0d62d9670bb451fd",
            "path": "hello.txt",
            "contentType": "text/plain; charset=utf-8",
            "mode": 436,
            "size": 6,
            "mod_time": "2020-06-11T19:22:50+02:00"
        },
        {
            "hash": "685cf57b9fd0231b111105e79bd9b843ba5083d3f1f4cd97bb71aca62848006a",
            "path": "world.txt",
            "contentType": "text/plain; charset=utf-8",
            "mode": 436,
            "size": 7,
            "mod_time": "2020-06-11T21:11:14+02:00"
        }
    ]
}

The real manifest is not formatted, it is without spaces, it is all in one single line, no new line at the end, but you got the idea.

The hash root you receive is not the root of a Merkle Tree at all, it is the hash of the manifest, that is actually a json.

=== Second topic ===

You can calculate hash in advance, two fast options :

1. use disposable swarm command, better if using a dedicated container: start without network, upload, get the hash, stop, destroy. If you don't use a container, block swarm from talking to other nodes starting it with swarm --maxpeers 0 --disable-auto-connect then upload, get hash, stop, remove swarm datadir
2. create a json file by hand or via script and use swarm hash <manifest_file> to get the hash.

=== The third arrives (cit.) ===

As you can see, there a very weird field: mod_time. This assume different values based on exactly how you upload the manifest (i.e. using the CLI it is set to the date time of the file, based on local machine). Two years ago there was a discussion about removing this field from the manifest or at least set void by default , but at some point in time, in some way, memory of problems about this behavior was lost, and the issue was closed.

So, if later on you will manually (or via script) check all files and hashes, it is ok, you can do it.

But if you want to check via swarm with a single command then:

1. you can never get a false-positive match
2. it is very likely you will get a false-negative, a false mismatch because of mod_time differences. And in this case, the only way to go is the manual check.

=== Bonus track ===

You can use IPFS instead of Swarm ;P at least until they fix this behaviour.

❯ ipfs add --recursive my_folder
added QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN my_folder/hello.txt
added QmTToWbuguq2X4fCUbPXYwkmGd1FKWZ52KdGEzSGBSVs2E my_folder/world.txt
added QmQ3KxegSpqM2sPNM86XeYpYYHvc8JokzyJrKtcSVbbu82 my_folder

And if you want to give Swarm a role even here you can store that multihash into it with:

❯ echo QmQ3KxegSpqM2sPNM86XeYpYYHvc8JokzyJrKtcSVbbu82 > my_folder_hash
❯ swarm --manifest=false up my_folder_hash
79e9f2708e6051583f0e9d47f333b1d3954bcf02ad5284543187889a994d5fca

Remember --manifest=false to avoid the same exact mod_time issue, and remember that without manifest you must use the API with the bzz-raw: namespace.