In Ethereum and Swarm, Difference between Sharding and Chunking

Question

Understand that in Ethereum further releases sharding will be included for scalability purposes. In swarm, "Chunks are pieces of data of limited size (max 4K), the basic unit of storage and retrieval in the Swarm"

Though in different context, one is for ethereum state and another for file, aren't chunking and sharding the same in concept? i.e., splitting up the files

Are there subtle difference between chunking and sharding then?

http://swarm-guide.readthedocs.io/en/latest/architecture.html https://github.com/ethereum/wiki/wiki/Releases

Answer 1

At a very high level, chunking is at the file level, and sharding is at the database level.

Let's say you have a large file that is 20KBs, then you need to chunk that into five pieces.

Sharding is the concept that not everybody needs to know everything. For example, let's say you have a database that is the equivalent to a phonebook. This database is maintained across five computers. So you can have that database replicated across all five computers, but that means you have the same information five times. Instead, you can split the database so that not every computer needs to hold ALL the information. So computer A stores names from A-D, computer B stores D-G, etc. Note that the split does not have to be mutually exclusive, for example, you can have a set of your 100 most commonly queried names and replicate those across all five computers (or "shards") to have better querying times.

Hopefully that helps.

Answer 2

To extend on @wtk219's answer, to maintain security of the network, e.g. in terms of the availability and validity of data, there are other design considerations.

What might a basic design of a sharded blockchain look like?

A simple approach is as follows. For simplicity, this design keeps track of data blobs only; it does not attempt to process a state transition function.

There exist nodes called proposers that accept blobs on shard k (depending on the protocol, proposers either choose which k or are randomly assigned some k) and create collations, thus they also act as a collator, and so agents that act as both a proposer and collator may be referred to as prolators. A collation has a collation header, a short message of the form "This is a collation of blobs on shard k, the parent collation is 0x7f1e74 and the Merkle root of the blobs is 0x3f98ea". Collations of each shard form a chain just like blocks in a traditional blockchain.

There also exist notaries that download (to ensure availability) and verify (by executing data to ensure validity) collations in a shard that they are randomly assigned and where they are shuffled to a new shard every period via a random beacon chain (using some Verifiable Random Function such as a blockhash produced by a BLS aggregate signature or RANDAO, although the latter has been tested to be prone to manipulation), and vote on the availability of the data in a collation (assuming no EVM, with an EVM they may also act as an executor and vote on the validity of data).

A committee can then also check these votes from notaries and decide whether to include a collation header in the main chain, thus establishing a cross-link to the collation in the shard. Other parties may challenge the committee, notaries, proposers, validators (with Casper Proof of Stake), etc., e.g. with an interactive verification game, or by verifying a proof of validity.

A "main chain" processed by everyone still exists, but this main chain's role is limited to storing collation headers for all shards. The "canonical chain" of shard k is the longest chain of valid collations on shard k all of whose headers are inside the canonical main chain.

Note that there are now several "levels" of nodes that can exist in such a system:

• Super-full node - fully downloads every collation of every shard, as well as the main chain, fully verifying everything.
• Top-level node - processes all main chain blocks, giving them "light client" access to all shards.
• Single-shard node - acts as a top-level node, but also fully downloads and verifies every collation on some specific shard that it cares more about.
• Light node - downloads and verifies the block headers of main chain blocks only; does not process any collation headers or transactions unless it needs to read some specific entry in the state of some specific shard, in which case it downloads the Merkle branch to the most recent collation header for that shard and from there downloads the Merkle proof of the desired value in the state.

—https://github.com/ethereum/wiki/wiki/Sharding-FAQs#what-might-a-basic-design-of-a-sharded-blockchain-look-like

For the latest spec as of August 13 2018, see https://notes.ethereum.org/SCIg8AH5SA-O4C1G1LYZHQ#.