Verifying personal.sign signature with pyethereum

Question

I'm trying to verify a signed message from Metamask with pyethereum. I can not seem to recover the correct address from the signature. Lack of documentation isn't helping matters.

On the client side, I have metamask signing with web3.personal.sign().

var signer = web3.eth.defaultAccount || web3.eth.accounts[0];
var original_message = "I am but a stack exchange post";
var message = "0x" + original_message.toHex();
var message_hash = web3.sha3('\u0019Ethereum Signed Message:\n' + message.length.toString() + message);
var signature;
web3.personal.sign(message, signer, function(err, res) {
    if (err) console.error(err);
    signature = res;
    console.log({
        "signer": signer,
        "message": message,
        "message_hash": message_hash,
        "signature": signature,
    })
});

{
    message: "0x4920616d20627574206120737461636b2065786368616e676520706f7374"
    message_hash: "0x1a0126ceafb4579293016a4cc3ca0ec753c7d497cda8b3e6ece095c832d92590"
    signature: "0x0cf7e2e1cbaf249175b8e004118a182eb378a0b78a7a741e72a0a34e970b59194aa4d9419352d181a4d1827abbad279ad4f5a7b60da5751b82fec4dde6f380a51b"
    signer: "0x9283099a29556fcf8fff5b2cea2d4f67cb7a7a8b"
}

Then I send the signature, hash of the message, and the address to the backend, where I have something like this:

>>> from ethereum.utils import ecrecover_to_pub, sha3
>>> from eth_utils.hexidecimal import encode_hex, decode_hex, add_0x_prefix
>>> signer = "0x9283099a29556fcf8fff5b2cea2d4f67cb7a7a8b"
>>> message_hash = "0x1a0126ceafb4579293016a4cc3ca0ec753c7d497cda8b3e6ece095c832d92590"
>>> signature = "0x0cf7e2e1cbaf249175b8e004118a182eb378a0b78a7a741e72a0a34e970b59194aa4d9419352d181a4d1827abbad279ad4f5a7b60da5751b82fec4dde6f380a51b"
>>> 
>>> r = int(signature[0:66], 16)
>>> s = int(add_0x_prefix(signature[66:130]), 16)
>>> v = int(add_0x_prefix(signature[130:132]), 16)
>>> if v not in (27,28):
...     v += 27
... 
>>> pubkey = ecrecover_to_pub(decode_hex(message_hash), v, r, s)
>>> assert(encode_hex(sha3(pubkey)[-20:]) == signer)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError

I've tried a few ways to mess around with encoding, but I think I'm still missing something(hopefully obvious). Any ideas?

Answer 1

Everything in Python looks correct. The message hash in JavaScript is being generated incorrectly.

Preparing the Message Hash in Javascript

The message hash must be generated using the original message, without hex-encoding it first.

In practice, that means replacing this JavaScript:

var message = "0x" + original_message.toHex();
var message_hash = web3.sha3(
  '\u0019Ethereum Signed Message:\n' +
  message.length.toString() +
  message
);

with this:

var message_hash = web3.sha3(
  '\u0019Ethereum Signed Message:\n' +
  original_message.length.toString() +
  original_message
);

That will give you the message hash: 0x6e099d83ea72d1ef62e39a501fe000c1458ba5a511510a0e9348b0dfeb298803

When you use that message hash, you will recover the correct signer: 0x9283099a29556fcf8fff5b2cea2d4f67cb7a7a8b

An even better solution: hashMessage()

With web3.js v1, you can call:
message_hash = web3.eth.accounts.hashMessage("I am but a stack exchange post").

V1 is in Beta, as of October 2017

---

Another Common Mistake

(from an earlier draft of the question)

Leaving out the prefix when hashing the message for recovery.

Because web3.personal.sign() adds the prefix for you, it's easy to forget the prefix when hashing the message for recovery. The best solution is again to use web3.js v1 for web3.eth.accounts.hashMessage().

---

The Simplest Option

As of Web3.py v4, there is built-in support to recover the message signer, like:

from web3.auto import w3

# If you have the original message, you need to hash it first
from eth_account.messages import defunct_hash_message
message_hash = defunct_hash_message(text=original_message)

# If you begin with the message hash, start here:
signer = w3.eth.account.recoverHash(message_hash, signature=signature)

Why is it called defunct_hash_message? -- Unfortunately, the message standard is not well supported. It has slightly different implementations in a number of nodes and hardware clients. There are some new message formats that are under discussion and will hopefully gain broader, more consistent adoption soon. For now, Web3.py only explicitly supports the geth-style message format, using the defunct_hash_message method.