Generating cache from block_number and seed using Python

Question

I'm trying to run Ethash algorithm from the Python-code provided over here: https://eth.wiki/concepts/ethash/ethash, and have some troubles comparing the produced cache-file with the actual cache-file from the real network, running by geth.

As the first step, I'm generating a cache file from the given block_number and the seed. The algorithm for cache-generation is described in the corresponding mkcache(cache_size, seed) function from the link above.

After running the function, I'm ending up with 16x995326 list, with 32-bit integer in every element. After the serialization of this list into a binary file (using Python struct module), I'm getting a 61-megabyte file.

Now, the question is: How can I verify that the generated cache-file is actually correct and it is intact with the current Ethereum network? Do you guys have any thoughts on that?

The way I'm doing this verification now is by comparing my cache-file with the cache-file from the actual ethereum-node running by geth.

For example:

• let's take a block number: 10757149
• for this block number, we can get the corresponding seed (please, check if the seed is correct): 0xf7f6e04564e6b37fb9017324dc2577bd24cd25977037cfe8cd95b590009e2391
• from the fully-synced ethereum node running geth, I can find the actual cache file, located in .ethereum/geth/ethash/cache-R23-f7f6e04564e6b37f
• executing md5sum on that cache-file, I'm getting the following checksum: d767fc7d317c78c6ac2e8b0a111d5dc0 (btw, if you're running fully-synced geth, you should get the identical md5hash for the R23-f7f6e04564e6b37f cache)

Now, given the block_number and the seed, I'm executing a Python-script from the example in the link above, and the md5sum on my cache-file is different!

On that point, I'm really stuck and have no idea how to check the cache-file correctness so far. Some other questions are coming up to my head, maybe you could clarify something:

1. Are the algorithms from eth.wiki/concepts/ethash/ethash actually followed by the official go-implementation of Ethereum (geth)?

2. Looking at the source-code of geth, I suspect that the cache-generation algorithm is different. I'm not good at parsing the go-syntax, so maybe some of you could check the Python-version of the mkcache function from here: eth.wiki/concepts/ethash/ethash, and the corresponding function from geth over here: github.com/ethereum/go-ethereum/blob/master/consensus/ethash/algorithm.go#L139. Would be very glad if someone could see the actual differences in the algorithms (if they exist).

3. Could it be due to a difference in cache serialization (when the generated array is being written to a file)?

Would really appreciate any hints on that topic. You may find the Python2.7 script I'm using over here: github.com/dugdmitry/ethash_python/blob/master/run_miner.py

Thank you!

Answer 1

Just solved the problem. There were a couple of mistakes I made when implementing the mkcache function in the example above:

1. The seed 0xf7f6e04564e6b37fb9017324dc2577bd24cd25977037cfe8cd95b590009e2391 for block_number 10757149 is incorrect. The correct way to calculate seed from the given block number is through the following function:

def get_seedhash(block_number):
    s = '\x00' * 32
    for i in range(block_number // EPOCH_LENGTH):
        s = serialize_hash(sha3_256(s))
    return s

2. The sha3.sha3_256() and sha3.sha3_512() hash-functions in the eth-wiki over here eth.wiki/concepts/ethash/ethash are incorrect (or rather outdated). From reading geth source-code, it can be seen that the following hash-functions are used instead: sha3.keccak_256() and sha3.keccak_512(). These functions are provided in pysha3 module for Python. Thus, the corrected code for mkcache function looks like this:

def serialize_hash(h):
    return ''.join([zpad(encode_int(x), 4) for x in h])

def deserialize_hash(h):
    return [decode_int(h[i:i+WORD_BYTES]) for i in range(0, len(h), WORD_BYTES)]

def hash_words(h, sz, x):
    if isinstance(x, list):
        x = serialize_hash(x)
    y = h(x)
    return deserialize_hash(y)

# sha3 hash function, outputs 64 bytes
def sha3_512(x):
    return hash_words(lambda v: sha3.keccak_512(v).digest(), 64, x)

# Generate cache
def mkcache(cache_size, seed):
    n = cache_size // HASH_BYTES

    # Sequentially produce the initial dataset
    o = [sha3_512(seed)]
    for i in range(1, n):
        o.append(sha3_512(o[-1]))

    # Use a low-round version of randmemohash
    for _ in range(CACHE_ROUNDS):
        for i in range(n):
            v = o[i][0] % n
            o[i] = sha3_512(map(xor, o[(i-1+n) % n], o[v]))

    return o

3. Additionally, when the computed cache is serialized into a file, a special 0xfee1deadbaddcafe bytes should be prepended to that file.

After making these changes, I was able to generate a correct cache-file, which md5sum matched with the md5sum of the corresponding cache-file, generated by geth for the same block on a real, fully-synced node. All good.

Hope it would be helpful for someone.

Answer 2

Use the keccak algorithm the seed hash "0xf7f6e04564e6b37fb9017324dc2577bd24cd25977037cfe8cd95b590009e2391 for block_number 10757149 is correct. I have test it with python and goeth.

And thanks for this Q&A, I also use the wrong hash algorithm: sha3_256 to generate two different seed hashes between the python example and goeth. And spent some hours to find out the issue.