Different outputs for Keccak256(), which one is right?

Question

To get a value from contract storage I used this geth code succesfully:

var contractAddress = '0x88e726de6cbadc47159c6ccd4f7868ae7a037730'
var index = '0000000000000000000000000000000000000000000000000000000000000001'
var key =  '00000000000000000000000x8703f7b22fc5613497aee971e80480a46b226d3c'
var newKey =  web3.sha3(key + index, {"encoding":"hex"})
    newKey: "0x749bce6b8d0b50e46cbdb3872aefff8c0e7eee46cc4368971ef94ce14e0c3839"

console.log('TokenBalance: ' + web3.toDecimal(web3.eth.getStorageAt(contractAddress, newKey)))
TokenBalance: 1000000000

But, when trying to implement that same code on GoLang the hash newKey is different and everything fails.

Code at https://play.golang.org/p/6z8EX44A3QX

    package main

    import "fmt"
    import "encoding/hex"
    import "github.com/ethereum/go-ethereum/crypto"

    func main() {
        index := "0000000000000000000000000000000000000000000000000000000000000001"
        key := "00000000000000000000000x8703f7b22fc5613497aee971e80480a46b226d3c"

        // Crunching hash index for contract token balance at address 0x8703f7b22fc5613497aee971e80480a46b226d3c
        // var newKey =  sha3(key + index, {"encoding":"hex"})
        var newKey = crypto.Keccak256([]byte(key + index))
        fmt.Println("newKey: ", newKey)
        fmt.Println(hex.Dump(newKey))
        fmt.Println(hex.EncodeToString(newKey))
    }

Output:
    newKey:  [236 52 49 125 144 82 178 166 201 55 151 68 129 249 89 25 207 168 159 24 108 235 39 14 14 14 169 233 76 45 248 192]
    00000000  ec 34 31 7d 90 52 b2 a6  c9 37 97 44 81 f9 59 19  |.41}.R...7.D..Y.|
    00000010  cf a8 9f 18 6c eb 27 0e  0e 0e a9 e9 4c 2d f8 c0  |....l.'.....L-..|
    ec34317d9052b2a6c937974481f95919cfa89f186ceb270e0e0ea9e94c2df8c0

And when trying the same hash on nodeJs I get the same result as GoLang, but different from geth node:

$node
    var util = require('ethereumjs-util');
    var web3 = require("web3");

    var index = '0000000000000000000000000000000000000000000000000000000000000001';
    var key =  '00000000000000000000000x8703f7b22fc5613497aee971e80480a46b226d3c';
    var newKey1 =  web3.utils.sha3(key + index, {"encoding":"hex"});
    var newKey2 = util.keccak256(key+index);

    console.log("NEWKEY 1", newKey1);
    console.log("NEWKEY 2", "0x"+newKey2.toString('hex'));

Output:
    NEWKEY 1 0xec34317d9052b2a6c937974481f95919cfa89f186ceb270e0e0ea9e94c2df8c0
    NEWKEY 2 0xec34317d9052b2a6c937974481f95919cfa89f186ceb270e0e0ea9e94c2df8c0

And at go-ethereum source code I can confirm sha3 implementation is keccak256:

node\api.go(313):
    // Sha3 applies the ethereum sha3 implementation on the input.
    // It assumes the input is hex encoded.
    func (s *PublicWeb3API) Sha3(input hexutil.Bytes) hexutil.Bytes {
        return crypto.Keccak256(input)
    }

The thing gets worst when trying to compute sha3/keccak256 using online tools:

At https://emn178.github.io/online-tools/keccak_256.html        
    input: 000000000000000000000000000000000000000000000000000000000000000100000000000000000000000x8703f7b22fc5613497aee971e80480a46b226d3c
    output: fcc55d8cd146d5891583a2120b6546cc0e95a73b831ce3d165a6bc3bafe777e7

I guess I need to hash the ASCII string representation so trying first a conversion string to hex:

At http://string-functions.com/string-hex.aspx
    string: 000000000000000000000000000000000000000000000000000000000000000100000000000000000000000x8703f7b22fc5613497aee971e80480a46b226d3c
    hex: 3030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303130303030303030303030303030303030303030303030307838373033663762323266633536313334393761656539373165383034383061343662323236643363

And online hashing:

At https://emn178.github.io/online-tools/keccak_256.html
    input: 3030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303130303030303030303030303030303030303030303030307838373033663762323266633536313334393761656539373165383034383061343662323236643363
    output: 1bf2aa0bc8831bea8e5cfa462d8a825f0ed66b406b37fb63dd10ae6ebbf87ccc

Or padding with 0x:
    input: 0x3030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303130303030303030303030303030303030303030303030307838373033663762323266633536313334393761656539373165383034383061343662323236643363
    output: 6d09c345f835d959e3bcb83f0e6612d92041f901c041daf5448dbbf04c59af65

All three different outputs. How can it be? Thx for any hint!

By the way, I read this questions and answers but didn't found my answer: Which sha3 output is the correct one? Different hash values for keccak256 functions?

Answer 1

This is basically down to the interpretation of your input data as bytes or as a string. A good way to test this is using web3 in a browser (eg just visit the remix site and there's a console window you can use. This shows the following:

> web3.utils.keccak256("0x00000000000000000000000x8703f7b22fc5613497aee971e80480a46b226d3c0000000000000000000000000000000000000000000000000000000000000001")
0x749bce6b8d0b50e46cbdb3872aefff8c0e7eee46cc4368971ef94ce14e0c3839
> web3.utils.keccak256("00000000000000000000000x8703f7b22fc5613497aee971e80480a46b226d3c0000000000000000000000000000000000000000000000000000000000000001")
0xec34317d9052b2a6c937974481f95919cfa89f186ceb270e0e0ea9e94c2df8c0

(the first one is the one you want for your use case of accessing contract storage)

I don't know Go particularly, but it's probably converting the ASCII text of your string to bytes, and coming up with something like your "0x3030303030..." string in a later example, which you definitely don't want.

I would expect your node example would work if you prepended the string with "0x", as it will then interpret it as a number rather than a string:

var newKey2 = util.keccak256("0x"+key+index);

Your github example has the parameters the wrong way around!

Here's a simple test with real world numbers you can try in your various languages to see if you can make it work:

> web3.utils.keccak256("0x00")
0xbc36789e7a1e281436464229828f817d6612f7b477d66591ff96a9e064bcc98a
> web3.utils.keccak256("0x0")
0xbc36789e7a1e281436464229828f817d6612f7b477d66591ff96a9e064bcc98a
> web3.utils.keccak256("00")
0xe3f0ae350ee09657933cd8202a4dd563c5af941f8054e6d7191e3246be378290
> web3.utils.keccak256("0")
0x044852b2a670ade5407e78fb2863c51de9fcb96542a07186fe3aeda6bb8a116d

You'll notice that the first two return the same result, meaning that the input is correctly being decoded into a single (zero) byte, while the other two are being interpreted as strings.

Hope this helps!